| |
Office of the Assistant Secretary |
Policy Information Center |
In this chapter, the U.S. Department of Health and Human Services (HHS) highlights evaluations of general interest to the public health and human services community and illustrates the diversity of HHS evaluations completed in fiscal year (FY) 1996. Included are summaries of 11 evaluation projects selected by the Senior Editorial Advisors on the basis of criteria identified in appendix C and applied to 34 reports nominated by HHS agencies. These criteria are as follows:
Each summary includes a brief abstract; a description of the study, including its purpose, background, methods, findings, and use of results; the names of any publications that resulted; and the name and phone number of the person to contact for additional information.
This study evaluated the results of eight Child Access Demonstration Projects from two waves of demonstrations testing the effectiveness of mediation, counseling, education, and visitation monitoring programs designed to facilitate noncustodial parents' access to their children following divorce and separation. The incidence of access problems ranged from 31 percent to 4 percent of cases. The nature of access disputes varied widely; such disputes were reported by both the custodial and noncustodial parents. Making both parents attend mediation sessions is seen as critical and difficult. Where both parties attended, mediation visitation increased, relitigation was low, and there was increased child support compliance for the experimental vis-a-vis the control groups. Other interventions (e.g., parenting classes, counseling) which were done for severely conflicted parties or at a distance from the divorce/separation were not seen as making an additional impact in these areas. Also, the timing of access disputes was unaffected by the demonstrations.
As set forth in the Family Support Act of 1988, this evaluation explored the effect of two waves of Child Access Demonstration projects on the amount of time required to resolve access disputes; reductions in litigation related to access disputes; improvements in compliance with court-ordered child support amounts; and promotion of the emotional adjustment of children. It also assessed the extent and nature of child access disputes as well as parental satisfaction with the demonstrations.
Recent research in child psychology shows generally that close, frequent, and positive contact with the father following divorce and separation is beneficial for the child.
Child access is also important for child support enforcement. Recent Census data and research studies have indicated that where noncustodial parents have visitation rights or joint custody they tend to be more compliant with child support orders, although it is difficult to show cause and effect since the parents wanting to see the child may also be the better payers. Desire for increased child contact may follow child support payment rather than vice versa. Moreover, denial of visitation is seen as the major reason for nonpayment of child support for noncustodial parents who have money to pay child support.
There has been considerable pressure for the system to give support to the needs of noncustodial as well as custodial parents. Over 43 States authorize joint custody. There are currently over 200 court-based divorce mediation programs and over 280 fathers' rights groups organized throughout the country to facilitate child access by noncustodial parents.
Congress responded to the continuing public debate about the problem of noninvolvement by noncustodial parents and resulting litigation by directing HHS to conduct State demonstration projects relating to a variety of means of facilitating continuing involvement by the noncustodial parent.
In 1996 a new Federal grant program for child access and visitation programs was established nationwide.
Projects involved control and experimental group testing of different interventions (e.g., mediation, parenting education, community services for the noncustodial parent, group counseling, telephone monitoring of visitation) in a variety of public and private, court and administrative settings in Idaho, Florida, Indiana, Massachusetts, Arizona, and Iowa. Evaluation was provided for randomly selected experimental and control groups on such issues as relitigation, relitigation timing, visitation, and child support compliance. There were over 2,400 cases in the baseline and 1,500 cases in follow-up interviews.
The incidence of child access problems for divorced and separated parents ranged from a low of 4 percent to almost 30 percent of the cases by site. This is consistent with findings of other studies. The nature of child access problems are varied and may be longstanding for both parents. Some frequent problems include insufficient amount of visitation time, being cut out of a child's life, scheduling visitation, fights during pick-up and drop-off, concerns for the child's safety, concerns about the other parent's parenting style and negativity, and continuing problems with the parents' relationship.
There is a big problem of nonattendance by one or the other spouse at mediation or related interventions; and some effort should be made to compel attendance. However, where both parents attend mediation there is a high (65 percent to 70 percent) rate of arriving at parenting plans.
Mediation did resolve conflict in many, but not all, cases, and problems decreased after interventions. Visitation days increased for most sites and child support compliance increased to over 20 percent. A majority of both parents indicate satisfaction with interventions. However, the speed of dispute resolution and incidence of relitigation were not affected. In addition, nonmediation experiments did not seem to have an impact. It was difficult to assess improvements in emotional child development where tested.
The results should assist State, local, and private agencies in establishing programs to improve the continued involvement by the noncustodial parent. Mediation interventions at the time of the divorce--when both parents can be made to attend--are most effective. When disputes are high charged or considerable time has elapsed, it is more difficult to make an impact. Scarce resources can be focused on cases with problems before they become intractable. The results of these demonstrations are more relevant for divorce and formal separation cases than for unwed situations.
Office of Child Support Enforcement
David Arnaudo
202/401-5364
PIC ID: 5972.2
Policy Studies Inc., Denver, CO
This study evaluated the implementation of State child support guidelines mandated by the Family Support Act of 1988. It concluded that States need to increase the consistency of guideline application, including income verification procedures, extended custody and visitation arrangements, and consideration of multiple family situations, health insurance, and day care expenses in the calculation of child support awards. Adoption of a standardized support order would improve documentation of departures from guideline award calculations. State reviews of guidelines could be more thorough if carried out with sufficient resources, case surveys, and analysis of the cost of raising children. The study did not recommend the adoption of a national child support guideline.
This evaluation explored three issues: how State child support guidelines are applied; the extent, amount, direction, and causes of deviations from State guideline award formulas; and how States account for special family circumstances in determining child support awards. Research focused on the process of applying guidelines; the extent, cause, and documentation of deviation from guideline formulas; the consideration of special family circumstances (such as child day care, health expenses, and multiple families) in adjusting awards; and actions taken as a result of the mandated guideline reviews prepared by the States.
In 1984, in order to ensure appropriate and equitable child support awards by courts and other decisionmakers, Congress required every State seeking Federal funding for public welfare programs to establish child support guidelines. These initial guidelines were only advisory. However, the Family Support Act of 1988 requires that the State guidelines be used in determining the proper amount of support. Deviation from the presumptive guideline amount of award requires written justification.
Several Federal requirements help States develop child support guidelines. These include need for uniformity within each State; use of numerical formulas that consider all of a payer's income and provide for health care needs; use of guidelines to determine rebuttable, presumptive award amounts in judicial or administrative proceedings; use of guidelines for subsequent award modification; application of guidelines to all cases; and consideration of the child's best interests in State criteria for deviating from a guideline. The Family Support Act also requires that States reevaluate their guidelines at least once every 4 years to ensure the appropriateness of child support awards. States are to consider the cost of raising children and to sample case data on deviations from the guidelines. States must review child support orders for cases handled by a State child support agency at least once every 3 years.
This evaluation employed various sources of data--child support case records, stakeholder interviews, analysis of State guideline review studies, analysis of the Current Population Survey (CPS), and an expert panel. A sample of case records was drawn from 2 counties in each of 11 States. Although not representative of all States, over 4,000 cases were selected to reflect diverse guideline model types and child support formulas. The study conducted 215 unstructured telephone interviews with State and local officials, parents, and other stakeholders to gain a real-life perspective of the overall operation of State guidelines. Case studies of child support guidelines in 20 States were analyzed to assess the review process and the extent of required documentation for deviation from the guideline award amount. Over 2,000 cases in the March-April Matched CPS file with established child support awards in 1992 were analyzed to determine if presumptive State guidelines affected child support award amounts. Finally, an expert panel was assembled to provide advice on the evaluation and to make recommendations based on study findings.
Analysis of sample case record data indicated that only 17 percent of cases involved deviations from the guideline award amount, although the percentage varied considerably across the counties of 11 States. The four most common reasons for deviation were (1) agreement between the parties; (2) second households or multiple families; (3) extensive or extraordinary visitation or custody expenses; and (4) low income of the payer. The majority involved downward deviations--the average award decreased 34 percent. The ordered amount of support differed from the calculated guideline amount of support in a significant percentage of cases. Differences were found both in cases that followed the guideline calculations and in deviation cases. However, reasons for these differences were not documented in the case records. The consistency of the guideline application varied considerably. While some factors (such as income and health care expenses) were considered in nearly all orders, the consideration of other factors (such as multiple families and child care expenses) was quite variable across counties and within the same State.
Unstructured interviews with decisionmakers and other stakeholders in 21 study counties found a consensus that child support issues involving the impact of multiple families and second marriages, income imputation, and support for postsecondary education required further investigation.
Examination of State guideline reviews indicated that only 20 States collected and analyzed case data. Fewer than half of the States consider economic data on the cost of raising children. The extent to which these States took action as result of the data is unclear. When a particular factor, such as multifamily situations, is mandated by a numerical formula in the guidelines, there generally is a more uniform and frequent consideration of that factor in child support actions by the award decisionmaker. Verification of parental income, which is important in calculating accurate child support, is not frequent across counties; parties are often not prepared with the proper forms at hearings. When health insurance was ordered for one or both of the parents, the cost was rarely included in the child support calculation in States that have a mandatory numerical adjustment for such costs. Child day care expenses were included in the child support calculations for the cases that mentioned these expenses. There was no clear consensus in State reviews about the provision of postsecondary education support. Tax exemptions are commonly allocated in a flexible and equitable manner. The States handle extended custody and visitation arrangements without consistency or equity. These arrangements were commonly used reasons for deviation in child support awards.
Analysis of CPS data indicated that support awards may have increased by a small amount after the shift from voluntary to presumptive guidelines in 1988. However, the demographic composition of custodial parents changed over the period and the income of the noncustodial parent was unavailable, preventing any definitive assessment of the impact of mandatory guidelines.
The panel of experts reached the following conclusions:
The findings and recommendations of this evaluation should benefit those States considering the revision or improved consistency of guidelines for awarding child support payments. Other States should be able to use this study to accelerate the implementation of guidelines, periodic case record reviews, and other related actions required by Federal regulations. Examples of specific State actions that can be taken on the basis of this study include (1) more consistent application of guidelines across cases; (2) improved case documentation of departures from the guideline award amount and reasons for the deviations; (3) improved independent verification of parental income; (4) adoption of a standardized support order to ensure that factors common to the majority of child support actions are addressed consistently; (5) use of a mandated numerical formula to adjust for various factors; and (6) coordination and consistency of policy within and between agencies and courts with regard to procedures and documentation.
Office of Child Support Enforcement
David Arnaudo
202/401-5364
PIC ID: 5983
CSR, Incorporated, Washington, DC
The Elderly Nutrition Program provides grants to State units on aging to subsidize the provision of daily meals to people 60 years of age or older. This evaluation study describes participant characteristics, compares the characteristics of recipients of home-delivered meals to those receiving meals in congregate settings, assesses how well the program reaches disabled and poor elderly, estimates the program's impact on the nutritional intake and social contacts of participants, and examines program costs and other sources of funding. The study showed that participants have higher daily intakes of nutrients and more social contacts per month than a comparable group of nonparticipants. The Elderly Nutrition Program is successful in targeting older people who are poor, live alone, are nutritionally "at risk" because they are overweight or underweight, or are more functionally disabled than their age-group peers. Federal expenditures are highly leveraged with State, local, and private funds.
This study was conducted to inform policymakers about the effectiveness of the Elderly Nutrition Program (ENP), the largest Federal program aimed at meeting the nutrition service needs of the elderly. Four major issues were evaluated: (1) the ENP's effects on participants' nutrition and socialization, (2) who is using the ENP and how effectively the program serves targeted groups most in need of its services, (3) how efficiently and effectively the ENP is administered and delivers services, and (4) the range of ENP funding sources and the allocation of ENP funds among its components.
America will face critical challenges in the coming decades as it attempts to provide long-term care services to the nation's elderly population. As the large group of individuals born after World War II ages, a much higher proportion of Americans will be elderly and will require more health services and long-term care. In this context, it is important to examine whether existing long-term care service programs are producing their intended outcomes and whether their services are directed to those who need them the most.
One very important component of the overall package of home- and community-based services available for elderly people is nutrition services. A critical step toward meeting this need was taken in 1972 with the creation of the Elderly Nutrition Program under Titles III and VI of the Older Americans Act. Through Title III, State Units on Aging and Area Agencies on Aging implement a system of coordinated, community-based services targeted to older individuals. The ENP is administered by the Administration on Aging (AoA) of the U.S. Department of Health and Human Services. The AoA awards funds to State Units on Aging, which in turn distribute the funds to Area Agencies on Aging on the basis of State-determined formulas that reflect the proportion of older people in their areas and other factors. The area agencies provide services directly or award grants to and contract with nutrition projects that provide nutritional and supportive services. In addition, area agencies receive financial support from State and local governments, in-kind contributions, private donations, and voluntary contributions from participants.
Title III of the Older Americans Act authorizes the provision of nutrition and supportive services, such as meals, nutrition education, transportation, personal and homemaker services, and information and referral. Congregate meals and supportive services are provided at nutrition projects' meal sites (e.g., senior centers, religious facilities, schools, public or low-income housing sites, or residential care facilities). Alternatively, home-delivered meals are provided to homebound clients, either by the congregate meal sites and affiliated central kitchens or by nonaffiliated food service organizations. ENP meals are targeted to persons aged 60 or older. While there is no means test for program qualification, services are targeted at older persons with the greatest economic or social need. Similar nutrition and supportive services for American Indians, Alaskan Natives and Native Hawaiians are authorized separately under Title VI. Indian Tribal Organizations and agencies serving Native Hawaiians receive grant awards directly from the AoA.
Mathematica Policy Research, Inc., undertook this study for the AoA. A comprehensive research design was used, including in-person surveys of participants and eligible nonparticipants; in-person surveys of staff of congregate meal sites; in-person observation of sanitation and health practices at congregate meal sites, central kitchens, caterers, and home-delivered meal distribution sites; telephone surveys of staff at State Units on Aging, Area Agencies on Aging, Indian Tribal Organizations, and nutrition projects; review of program records on amounts of funding transfers; review of scientific literature related to nutritional requirements for the elderly population; extraction of relevant census data; and extraction of results from the previous national ENP program evaluation published in 1981.
All State Units on Aging were sampled. The sample of Area Agencies on Aging and nutrition projects was selected as a geographically clustered sample to maximize efficiency in conducting surveys. For Title VI programs, the agency sampling was simplified by the fact that each ENP grant is awarded directly to the Indian Tribal Organization, which in turn consists of a single nutrition project, often operating from a single site. In all, data were collected at 197 randomly selected meal sites. Approximately 2,500 interviews were conducted with elderly ENP participants (both home-delivered meal recipients and congregate site meal recipients) and nonparticipants. The nonparticipant sample used for comparison with the Title III program participant sample was selected from lists of Medicare recipients in the same ZIP Code areas as the participant sample, screening for income and disability status to ensure that the samples were matched as closely as possible.
The evidence collected in this evaluation suggests that the ENP has substantial positive effects on participants. Title III participants have higher daily intakes of key nutrients and more social contacts per day than a comparable group of nonparticipants. The average ENP meal meets the program's requirements to provide at least one-third of the relevant Recommended Daily Allowances. Both congregate and home-delivered participants have about 14 more social contacts per month than the comparison group; this finding and interviews of participants suggest that the program increases socialization opportunities for participants. Over 60 percent of all Title III and Title VI participants are satisfied with program services. Approximately 40 percent of Title III ENP service providers have waiting lists for home-delivered meals, suggesting a significant unmet need for these meals.
The evaluation shows that the program has achieved considerable success in efforts to target ENP services to populations in the greatest need. Compared with the overall population in the United States age 60 and older, ENP participants tend to be older, poorer, more likely to be members of racial or ethnic minorities, and more likely to live alone. About one-third of Title III congregate participants and one-half of Title III home-delivered participants have incomes at or below the poverty threshold. More than one-half of Title VI meal participants have incomes at or below this level. The comparable figure for the overall population age 60 and older is 15 percent. Nearly four times as many Title III participants and nine times as many Title VI participants are low-income minorities, compared with the overall population age 60 and older.
Agencies at the various administrative levels of the program have forged close links with other parts of America's emerging home- and community-based long-term care system, primarily through cross-referrals and coordination of service delivery at all levels within the aging network. The ENP provides a continuum of services, including nutrition screening, assessment, education, and counseling. Title III funds support 37 percent of congregate costs and 23 percent of home-delivered costs and are highly leveraged by other funding sources, including U.S. Department of Agriculture commodities or cash in lieu of commodities, State, local, and private funds. Participant contributions support another 20 percent of the program costs. The leveraging rate for Title VI is considerably lower, with over 60 percent of the costs supported by Title VI grants.
This report will provide policymakers with the information they need to determine the future scope of activities of the ENP, as well as future funding levels. The information in this report will also assist service providers in their efforts to improve the operation of their ENP projects and to obtain additional State, local, and private funding to supplement Federal grants.
Office of Program Operations and Development
Jean Lloyd
202/619-2005
PIC ID: 6175
Mathematica Policy Research, Inc., Plainsboro, NJ
The final report on the Oregon Consumer Scorecard (OCS) Project describes the development, pilot testing, and revision of a user-friendly guidebook, "A Consumer Guide to Selecting a Health Plan," by the Oregon Consumer Scorecard Consortium. The prototype scorecard is intended to help consumers choose a health plan consistent with their individual needs and health care service delivery preferences. The model also may help purchasers and State policymakers develop their own comparative reports about health plan performance.
A contract from the Agency for Health Care Policy and Research was awarded in January 1995 to the Oregon Health Policy Institute through the University of Washington for this Federal-State collaboration in policy making and knowledge-building. There were five project objectives:
The model scorecard was intended to aid consumer decision making in two primary ways. First, it was to provide comparative health plan descriptions based on the expressed information preferences of Oregon consumers. Second, it was designed to educate consumers about health plan function, management, and performance. The project was guided by the presumption that people will make more prudent choices among health plans if they have access to objective, reliable measures of technical quality, health plan performance, and consumer satisfaction.
Oregon has one of the most extensive managed care markets in the United States. After Oregon received approval for a Federal Medicaid waiver, the State legislature enacted a series of reforms known as the Oregon Health Plan (OHP). Since its implementation, OHP has extended coverage to approximately 130,000 individuals, including many previously without health insurance coverage who must now navigate an unfamiliar health care delivery system.
These reforms led to the establishment of the Oregon Office of the Health Plan Administrator (OHPA), with responsibilities that included "to assist OHP consumers in selecting a health care provider or a health care plan" (SB 5530, Oregon Statutes of 1993). The OHPA facilitated the creation of the Oregon Consumer Scorecard Consortium, a public-private partnership committed to developing a high-quality, user-friendly health plan guide.
A series of activities was undertaken to develop and pilot test stimulus materials that would subsequently inform the design of a prototype consumer scorecard to be evaluated on a statewide basis. The first step was the formation of the Oregon Consumer Scorecard Consortium, comprising more than 50 stakeholders from the public and private sectors that may benefit from or be affected by the scorecard.
Second, a work plan was established for reviewing and synthesizing policy-related information, conducting focus groups, and reporting on all other Consortium-sponsored data collection activities. The information review and synthesis included analyzing the Oregon Office of Medical Assistance Programs' annual consumer satisfaction survey, designed during the project year in consultation with the OCS Project staff and the Consortium's Technical Committee; pilot testing measures from Medicaid HEDIS (Health Plan Employer Data and Information Set), a widely used health plan performance measurement tool; and gathering objective, comparative, and descriptive navigational information to help consumers understand how a health plan really works.
Through a subcontract to Oregon Health Decisions, Inc., two rounds of four focus groups were conducted with OHP-Medicaid consumers. Targeted at consumers living in rural areas and persons with chronic health care conditions and disabilities, the first-round focus groups were designed to gain understanding of consumers' expressed information needs and preferences for selecting a health plan. In the second round, consumers who had a choice between two or more competing health plans in their counties were given a model guidebook and a set of alternative formats and media presentations to use and critique.
Third, the first set of prototype scorecard materials was produced. Finally, using real health plan data but fictitious health plan names, the guidebook was presented to Oregon consumers in a series of focus groups for their review and assessment.
Consumers expressed a desire for more accountability from their health plans. They wanted objective information on satisfaction and health plan performanceþinformation that is understandable, reliable, meaningful, and geographically sensitive. There is a need for unambiguous descriptive data, free of marketing bias, that show real differences in how health plans manage care.
Materials developed during this project were well received. Consumers were interested in multimedia presentations of choice information, using interactive computer kiosks, videos, and telephone advice. The availability of a trained, live individual was seen as an important backup to the scorecard.
The report noted that consumers and health professionals think differently about quality. Plan-level information was less important to consumers than provider-level information. Consumers preferred information that was relevant to their individual experience, including how they personally use health care services. Condition-specific indicators in population-based performance measures (such as HEDIS) were found to be less well understood, and therefore not as useful to consumers in selecting a health plan.
Despite the advantages of this voluntary collaboration, producing a reliable and uniform consumer scorecard in a timely fashion proved difficult. The report recommended the establishment of uniform standards and data specifications that all health plans and purchasers agree to abide by, with an independent "audit" to ensure compliance.
The report also noted that the political nature and the potential economic consequences of the comparative health plan information made it critically important to have an objective, disinterested third party responsible for producing a scorecard. In addition, because scorecard development is politically and methodologically complex, the report advised that future efforts should proceed in stages, with more modest goals and an examination of the considerable costs of producing a scorecard.
This project provides a unique model for integrating major types of health plan performance information into a consumer-oriented guidebook and adds to the emerging body of knowledge on consumer information needs and preference modeling. This contribution to the science base for statewide quality assessment and reporting can be used by other States in their efforts to evaluate quality of care.
Major policy and methodological issues to pursue in the future are addressed in the report, including the need to address differences in how professionals and consumers think about quality; the need to balance the focus on health plans as the unit of comparison for scorecard purposes with the interest of most consumers in the attributes and performance of individual clinicians and the facilities in which they practice; the need to reconcile population-based measures and personalized, consumer-relevant information; the need to tailor information to consumers with special concerns; and the need to critically evaluate the costs and benefits of various forms of presentation. The report recommends as a top public policy goal the development of a real and meaningful quality feedback loop that extends from the State to health plans to consumers and back to the State.
Office of Planning and Evaluation
Sandra Robinson
301/594-1349
PIC ID: 5960
Oregon Health Policy Institute, Portland, OR
Between 1986 and 1994, the State of Georgia implemented a program of audits of the immunization practices of all State public health immunization clinics (227 in 1994) in an effort to improve child immunization rates. During this time period, the immunization rate for preschoolers increased from 31 percent to 90 percent. This evaluation found that the performance of these audits was significantly associated with higher immunization rates for preschool children in the clinic catchment areas.
Other management and medical practices were found to be significantly associated with increased immunization rates. Greater accessibility, informed vaccine administration practices (such as following only true contraindications), adequate clinic resources, and staff participation in leadership were also associated with higher immunization rates. The report lists 11 recommendations associated with the findings; in particular, the audits of the clinics should continue, and efforts to improve immunization rates should be comprehensive in scope and continuous in application. Findings from the evaluation of this highly successful program formed the basis of Centers for Disease Control and Prevention (CDC) recommendations to all States.
The purpose of this study was to evaluate how audits conducted by the State of Georgia affected immunization rates among the preschool population, to assess management and organization factors associated with immunization coverage, to identify aspects of immunization clinic practice and organizational or managerial processes associated with changes in immunization coverage levels, and to make recommendations regarding the implementation of similar programs in other projects in the United States.
The Public Health Service (PHS) objectives for the year 2000 include 90 percent immunization coverage levels for children up to the age of 24 months. The diseases against which immunization is sought are diphtheria, tetanus, pertussis, poliomyelitis, measles, mumps, rubella, Hemophilus influenza type B, and hepatitis B. The CDC's Comprehensive Childhood Immunization Initiative was developed to address this objective through an intensified strategy that includes service delivery, assessment, information and education, operational research, and surveillance.
To improve its child immunization rates, the State of Georgia conducted a program of annual audits between 1986 and 1994. The program allowed and encouraged district and clinic immunization staff at the State's public health clinics to implement interventions to increase clinic-specific coverage levels. Various approaches and combinations of interventions to change clinic practices were undertaken and evaluated.
Data for the study were drawn from clinic audit data, 1987-94, and from a self-administered mail survey of key immunization personnel in all public health clinics in the State of Georgia (227 in 1994). Individuals surveyed included district health directors or program managers, district immunization coordinators, and clinic immunization coordinators or clinic lead nurses. The survey response rate, due to intensive follow-up procedures, was 100 percent. Univariate and multivariate methods were used for analyses relating information on clinic practices from the surveys to information on audit rates from medical files.
The study found substantial improvement in coverage rates following the audit program, with median immunization rates at the clinics improving from 31 percent to 90 percent. Staff participation in the audit process resulted in their increased awareness of the performance levels of their offices, and of how their clinic compared with state averages; this, in turn, appeared to prompt improved immunization rates at the clinics.
Accessibility factorsþsuch as short waiting times, fixed fees, and convenient hours of serviceþwere associated with higher immunization rates at the clinics. Clinics with higher immunization rates also used improved vaccine administration practices, suggesting an association between appropriate clinical knowledge by key staff and higher coverage rates. Early outreach, aggressive reminder and recall, and close coordination with Women, Infants, and Children (WIC) and other programs were also associated with higher immunization rates.
Clinics with adequate resources had higher coverage rates. Computer technology, appropriately supported and implemented, appeared to benefit clinic rates. Lower coverage rates were more likely to be found at sites in the process of computerization than in those that were fully automated, although clinics without computers had even lower immunization rates. These findings suggest the importance of support and training to the process of computerization, as well as the importance of computerization in the clinic setting.
The study also found that staff participation and management leadership were associated with higher immunization rates. The investigators conclude that sustained efforts to improve immunization rates can make a difference.
Eleven recommendations based on study findings are included in the report, supporting continuing efforts to improve immunization rates through strategies that are long term, comprehensive in scope, and continuous in application. The report recommends continuing the audits, which were a critical element helping to improve these rates, and furnishing audit results to clinic providers and managers. Recommendations also target ready accessibility to clinics, early outreach, and aggressive reminder and recall. Staff training and development were encouraged to provide accurate information about contraindications and to minimize missed opportunities for immunization. Clear management practices that support and recognize the importance of immunization, that foster staff participation in decisions about improving immunization rates, and that provide adequate staff and financial resources were also recommended. The report recognizes the important role that programs such as WIC can play in improving immunization rates, and it calls for ongoing study of the role of computerization and of improvement efforts in other settings as they relate to coverage rates.
These recommendations have had far-reaching application, providing a national model for immunization practice. Georgia staff report that the evaluation findings have stimulated new staff thinking about immunization practice, as well as changes in policies to carry out the study's recommendations. Georgia's successful strategyþusing routine assessment and feedback to achieve high sustainable coverageþwas adopted and produced successful results in public health clinics in Colorado, Illinois, Iowa, Kansas, and South Carolina. It also worked in private and managed care provider settings in Arizona, Massachusetts, New York, and Washington.
Several reports of early study findings have appeared in the Morbidity and Mortality Weekly Report and CDC information sheets. Workshops and presentations on the project have been conducted throughout the term of the project, resulting in replication of this approach in other States. Furthermore, CDC reports that the recommendations of this study now provide the basis for their national immunization program: Since 1995, all States and localities receiving Federal funds for vaccine programs have been required to conduct annual assessments of vaccine rates in public health clinics.
National Immunization Program
Andrew Baughman
404/639-8239
PIC ID: 5354
Battelle Centers for Public Health Research and Evaluation, Arlington, VA
Chronic ventilator patients experience high mortality, and their care is exceedingly expensive. The purpose of this evaluation was to determine whether specialized units serving chronic ventilator-dependent patients deliver better clinical outcomes at a reasonable cost to Medicare. The evaluation compared chronic ventilator patients treated in ventilator-dependent units (VDU's) at four demonstration sites with similar patients treated in conventional hospital settings, usually in intensive care units. The evaluation found that patients treated in VDU's have improved clinical outcomes and lower hospital daily costs, but higher overall costs to Medicare because of greater longevity. For example, while the patients' median survival from the time of admission was about 5 months longer, and their functional status superior at discharge, the overall cost to Medicare was increased by 35 percent, due to longer life expectancy. The findings across the four demonstration sites varied greatly, however. Because of the variability in clinical outcomes and the difficulty of screening patients for placement in VDU'sþboth of which can lead to greater costs without commensurate clinical benefitsþthe evaluation concluded that nationwide implementation of VDU's is not recommended. However, the evaluation did find a small number of special VDU centers of excellence to be warranted. The analysis of costs and outcomes provides important information to policymakers and health providers about a seriously ill population with intensive resource needs.
The purpose of the evaluation was to assess whether chronic ventilator patients covered by Medicare were better served at a reasonable cost by specialized hospital units called ventilator-dependent units. These units, studied at four Medicare demonstration sites, have a rehabilitation focus and are staffed by a highly trained multidisciplinary team of health professionals. The daily cost of care in a VDU is about $500 to $700 less than that of an intensive care unit, where most hospitals care for chronic ventilator patients. The evaluation compared costs and clinical outcomes of 211 chronic ventilator patients treated in VDU's with 401 comparison cases treated in conventional hospital settings. The evaluation also projected the overall costs of providing care in VDU's to Medicare patients nationwide.
Mechanical ventilation is a life-sustaining technology for patients suffering from acute respiratory failure. The ventilator is an apparatus that delivers air to the lungs of patients unable to breathe on their own, usually because of failure of other organ systems or severe chronic illnesses. Because of the severity of their underlying illnesses, 49 percent of Medicare ventilator patients died in FY 1988. The 133,500 Medicare patients discharged in FY 1994 after an episode of mechanical ventilation had an average length of stay of 23.1 days, with total Medicare expenditures amounting to $3.5 billion. Thus, ventilator patients' high mortality, coupled with the high cost of treatment, warranted further scrutiny by the Health Care Financing Administration (HCFA), the agency that administers Medicare.
The focus of the evaluation was on chronic ventilator patients, that is, the small subset of ventilator patients who need ventilation for extended periods. Chronic ventilator patients are especially important to study because they are even more seriously ill and their care is correspondingly more costly. In this evaluation, a chronic ventilator patient was defined as one receiving at least 20 days of ventilation.
This evaluation was conducted under an exemption to Medicare's prospective payment system to allow demonstration hospitals to be paid for VDU care on a cost-reimbursement basis, with some incentives to control costs. Under Medicare's prospective payment system, Medicare normally pays a flat fee for chronic ventilator patients. Because of earlier reimbursement problems, patients are now grouped under one of three distinct diagnosis-related groups specifically tied to their receipt of chronic ventilation, rather than to their underlying illnesses. Yet problems still exist with the revised groupings, because hospitals and patients continue to have strong incentives to discharge patients to another setting. Understanding costs and effective treatments therefore permits further refinement to the prospective payment system for chronic ventilator patients.
The evaluation of costs and outcomes relied upon a constellation of data bases, including Medicare's enrollment and claims data and special clinical data sets. The four VDU demonstration sites were the Mayo Clinic; Temple University Hospital; RMS Health Providers in Hinsdale, Illinois; and Sinai Hospital of Detroit. New clinical instruments were developed for these sites to obtain a comprehensive portrait of the course of ventilator episodes. Claims data were used to track the VDU and the comparison group through 18 months following hospital admission.
The comparison group was drawn from a HCFA pilot data base known as the Uniform Clinical Data Set System, which is no longer in place. The data on chronic ventilator patients were derived from hospital records in five States. A major difference between VDU and comparison cases was that the former were screened for entry into VDU's, while the latter were not. Screening according to criteria set by HCFA excluded medically unstable patients and those with poor prospects for rehabilitation. Multivariate models were employed to control for differences between the VDU and comparison cases.
Three sets of findings were generated by the study: admissions, clinical and cost outcomes, and national implementation findings. The admissions findings revealed that, in many instances, patients were improperly admitted to VDU's. In other words, it was difficult for VDU sites to meet all of HCFA's complex criteria for entry, which relate to medical stability and reasonable potential for rehabilitation. Across VDU sites, there was great variability in the application of admission criteria.
With respect to outcomes, the evaluation analyzed 15 clinical outcomes and 17 expenditure outcomes. On many of the clinical outcomes, VDU cases fared better than comparison cases. Among the most important findings was that median survival from the point of hospital admission was 258 days for VDU cases and 106 days for comparison cases, a statistically significant increase of about five months. The mortality rate within the hospital was 34 percent for VDU cases and 48 percent for comparison cases. When patients were discharged, VDU cases were in better condition: 34 percent of VDU cases were discharged to their home, compared with 27 percent of comparison cases. Their functional status at discharge was superior, using a special index based on activities like bed mobility, locomotion, toileting, and eating. About 18 percent of VDU cases at discharge scored in the highest functional group, compared with 11 percent of comparisons. At two of the demonstration sites, however, some clinical outcomes were not significantly improved relative to the comparison group.
While daily Medicare expenditures for VDU cases were lower, overall expenditures were about 35 percent higher because patients lived longer. Total hospital expenditures averaged $123,000 per VDU case and $91,000 per comparison case. The mean daily spending within the hospital was $1,468 for VDU cases and $1,740 for comparison cases, because VDU's were generally less costly than alternatives.
In the national implementation analysis, costs were projected for covering VDU's for all eligible Medicare patients. The evaluation found that 67 percent to 80 percent of comparison cases would have been admitted to a VDU if one had been available, yielding an estimate of between 24,000 and 41,000 total admissions nationwide. Total Medicare expenditures for their care were estimated between $0.6 and $2.2 billion, depending on assumptions and duration of coverage. In light of the difficulty of controlling admissions and the variability in clinical outcomes, the evaluation concluded that nationwide implementation of VDU's would be very costly and of questionable benefit.
The evaluation recommends continuation of VDU's at selected centers of excellence to refine rehabilitative treatment for chronic ventilator patients. It also recommends further exploration of financing mechanisms that could encourage integration of care, rather than moving patients from one setting to another. Managed care, particularly through case management, is seen as holding promise for such integration. The evaluation also pointed to research questions that warrant further attention. Finally, by providing detailed information about costs and clinical outcomes for a population at high risk of death, the evaluation raises for policymakers and health care providers the difficult question of whether and how resources should be allocated to a seriously ill population in a manner that optimizes duration and quality of life, without being economically wasteful and prolonging suffering.
Office of Research and Demonstrations
Michael Henesch
410/786-6685
PIC ID: 4590
The Lewin Group, Fairfax, VA
This evaluation reports on Phase II of a three-phase effort to develop a fully integrated computerized data system to facilitate planning and evaluation of the programs administered by the Bureau of Health Professions. Phase I assessed indicators of the Bureau's progress in meeting the goals and objectives defined in its strategic plan. Phase II analyzed the feasibility and appropriateness of the goals, outcomes, and indicators developed by the Bureau and identified strategies the agency can use in refining them with periodic input from grantees in the field. A performance monitoring system was proposed to facilitate ongoing program management. The system described in this study builds on the Bureau of Health Professions Grants Management Application System. The study recommended that the proposed performance monitoring system be piloted with several programs before it is fully implemented.
The overall purpose of this project was to help the Bureau of Health Professions develop a set of outcome-based performance measures and design a performance monitoring system, both to measure whether supported programs are meeting national health workforce objectives and to identify necessary program course corrections. The following issues were addressed in the study:
The Bureau of Health Professions provides both policy leadership and support for enhancement of the health professions workforce and development of its educational infrastructure. This evaluation follows earlier work to assess and refine a set of goals with respect to workforce quality, supply, diversity, and distribution, as well as the outcomes and indicators of performance identified to measure and monitor progress toward those goals. This strategic planning process was undertaken by the Bureau in response to internal and external pressures for more effective targeting of scarce Federal resources toward those programs and activities that support and have a demonstrable effect on national workforce priorities. The development of a more explicit outcome-oriented system that identifies measures of performance related to Bureau-funded efforts will help address such concerns and will be very useful in future planning and program management.
The cross-cutting goals developed by the Bureau of Health Professions address (1) the development of a health care workforce that has the mix of competencies and skills needed to deliver cost-effective, quality care; (2) the need for educational programs that will yield professionals who can meet the needs of vulnerable populations; (3) the need for cultural diversity in the health professions; and (4) the need to stimulate and monitor the education system's ability to respond to the changing demands of the health care marketplace. Expected outcomes and indicators of success were developed and refined for each goal as part of phaseI of this evaluation effort. This evaluation was undertaken to address the next set of questions that needed to be examined by the Bureau in order to develop a performance measurement and management system that could link individual grantee level information in all of the Bureau's programs to one or more of the cross-cutting goals, associated outcomes, indicators, and performance-related functions.
The investigators employed several approaches to achieve the evaluation objectives. Among these were ongoing discussions among evaluators, Bureau of Health Professions program administrators, and other agency officials and committees whose input would inform the design of the performance evaluation system. The evaluators also consulted outside experts representing a range of health workforce training perspectives on feasibility issues related to data collection from grantees for program monitoring and evaluation purposes. An initial survey of data sources was then undertaken to explore existing systems that may be applicable to the ongoing data collection needs of the Bureau's performance monitoring system.
This report focuses on the design of a Comprehensive Performance Monitoring System (CPMS). The study calls for a system that will provide information to answer a basic performance question: Can the Bureau of Health Professionsþwith available funding and guiding legislation and through planned and funded grant activitiesþmeet national health workforce objectives for targeted populations? In capturing the information necessary to answer this question, the system would track essential inputs, processes, outputs, and outcomes for Bureau programs. The system would provide that information in regular reports to internal decisionmakers and to the Bureau's external customers, including Congress and grantees. The CPMS should support the following functions: monitoring and measuring performance goals; analyzing and assessing indicator data; identifying successes and problems; reviewing key program processes; and identifying opportunities.
A detailed set of recommended steps was developed for the Bureau's preparation for implementing the CPMS. These include completing final specifications of cross-cutting indicator definitions; specifying performance benchmarks and a process for assessment; identifying external indicators that must be tracked (e.g., market interest, availability); identifying sources of data and pilot testing new data collection; specifying hardware and software requirements of the system; and developing a pilot version of the system for beta testing.
The processes essential for maintaining the functions of the CPMS will require data, hardware, software, and human resources. The analytic and reporting functions of the system would be largely automatic once designed and implemented. Obtaining key information resources will require collecting data that are readily available and developing data collection instruments or surveys to collect data that are not currently reported by all grantees on a systematic basis. Other data collection resources managed outside of the Bureau may also be integrated into the system. The CPMS could exist in a variety of forms, ranging from paper files and reports to computer data base systems. The study recommended that an add-on module linked to the Grants Management Application System may yield the most efficient system. The system could be built as a relational data base, with linkage variables for cross-referencing data. The types of resources that would be needed for a computer-based performance monitoring system could include computer hardware; system software, including data base system software; and trained software and hardware systems support staff to develop, maintain, and update the capabilities of the system as needed. The data base could be on a stand-alone computer or could reside on a network.
The study concluded that in the final phase of this effort, the Bureau of Health Professions must address the questions necessary to perform key functions of the CPMS for each of the cross-cutting goals, outcomes, and indicators. Furthermore, the Bureau must decide what data elements should be collected from grantees to support the CPMS. Many data elements may already be collected and captured in the Grants Management Application System; thus, the CPMS should be integrated with this system and the staff with these responsibilities should coordinate their efforts.
This Bureau of Health Professions study on performance monitoring to support program management and Bureau-wide planning also fits into the context of broader efforts within the Health Resources and Services Administration and HHS to conduct strategic planning for resource investment and compliance with the GPRA. Policymakers and program staff in other Federal agencies within and outside HHS can benefit from the efforts described in this evaluation.
Bureau of Health Professions
Jennifer Burke
301/443-1590
PIC ID: 5497
The Lewin Group, Fairfax, VA
This study assessed the effects of State board and care home regulations on the quality of care provided by those institutions. The investigators also attempted to identify the characteristics of the board and care environments in the States selected for the study and to determine whether licensed and unlicensed homes differed in terms of those characteristics or in the quality of care provided. The evaluation concluded that extensive State regulation and licensure can improve the quality of care provided by board and care homes.
This investigation examined the quality of care in board and care homes in 10 Statesþ5 with extensive regulatory systems and 5 with more limited regulatory systems. For the purposes of this study, board and care refers to nonmedical, community-based residential settings that house two or more unrelated adults and provide such services as meals, medication supervision or reminders, organized activities, transportation, or help with activities of daily living.
The principal purpose of the study was to analyze and compare State board and care regulations and their effects on the quality of care received by board and care residents. Investigators attempted to do the following:
To accomplish these objectives, the study analyzed data on State regulatory systems, constructed a sample frame of licensed and unlicensed homes, and implemented a complex multistage sample design. In addition, the investigation compared data with the findings of earlier studies of board and care and with current data on the characteristics of other residential long-term care settings to clarify the role played by board and care homes.
Changes in population demographics and a number of policy initiatives have increased demand for residential facilities that offer support services for the aged and disabled. Chief among these are a rapidly growing elderly population with significant levels of physical disability and mental impairment, an almost universal rejection of nursing home care by younger persons with disabilities and their advocates, and a strong preference among the elderly for in-home and community-based services as opposed to traditional nursing home care. While family efforts continue to be the primary source of long-term care for elderly and disabled loved ones, different types of residential settings with support services have emerged to supplement the efforts put forth by families.
There are approximately 34,000 licensed board and care homes in the United States, with more than 613,000 beds. These homes fall into one of three basic types of licensed facilities: those serving mentally retarded or developmentally disabled persons, those serving mentally ill persons, and those serving a mixed population of physically frail elderly, cognitively impaired elderly, and persons with mental health problems. Not all board and care homes are licensed, however, and by some estimates, unlicensed homes are as numerous as licensed facilities. Thus, the total number of persons living and receiving care in all types of board and care homes may be as high as one million.
While the Federal Government has traditionally played only a limited role in monitoring or regulating the quality of services provided by board and care homes (leaving primary oversight to the States), the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in HHS has maintained a long-standing interest in the potential of board and care homes to meet the needs of elderly and disabled persons for residential services. This becomes particularly important in light of the strengthened Federal oversight of nursing home quality required by the Omnibus Budget Reconciliation Act of 1987.
The present analysis stems from Federal concerns regarding the lack of systematic information on board and care residents, changing levels of disability among the population served, adequacy of care, protection from health and safety risks, and the significant numbers of unlicensed and unregulated homes in the United States.
To accomplish the goals of the investigation, the study design incorporated several activities, including a major collection of new data. However, all activities focused on the main study goals of facilitating cross-sectional comparisons among facilities and residents based on the licensure status and regulatory environment under which the homes operated.
After reviewing current State regulatory approaches, the investigators selected 10 States for inclusion in the study; these States represented the extremes on a continuum of regulatory systems, ranging from very extensive regulation (California, Florida, New Jersey, Oklahoma, and Oregon) to very limited regulation (Arkansas, Georgia, Illinois, Kentucky, and Texas). A stratified, multistage, cluster design was used to select probability-based samples of homes, staff, and residents in each State. A sampling frame of eligible unlicensed homes was created using the Social Security Administration's State Data Exchange (SDX) tapes and network sampling of State and local agencies.
Primary data collection was then undertaken in 385 licensed and 129 unlicensed board and care homes, including interviews with 490 operators, 1,138 staff, and 3,257 residents. Site visits were conducted at each facility and extensive information was collected on characteristics of the home, including patient and payer mix; characteristics of the staff, including knowledge of aging and caregiving; resident demographics, health status, and satisfaction; and quality of care, including the physical environment, adequacy of staff, unmet health needs, and patient satisfaction. Data were analyzed using descriptive statistics and logistic and linear regression models.
This inquiry found substantial and widespread positive effects of both regulation and licensure on the quality of care in board and care homes. States with extensive regulatory systems had a significantly smaller proportion of unlicensed facilities than States with limited regulation (7 percent versus 25 percent). Extensive regulation also had a positive effect on several quality-of-care and quality-of-life indicators, such as lower use of psychotropic drugs and medications contraindicated for the elderly, more operator training, and greater availability of social aids (e.g., reading materials and community rooms) and supportive devices (e.g., grab bars in showers and call buttons in bathrooms).
Licensure also had a positive effect on many of the quality-of-care measures explored. For example, licensed homes were more likely to have operators with training and to make more social aids and supportive devices available to residents. Licensure also enhanced the availability of key services and the prevalence of safety features. A significant finding is that licensure was effective in raising homes above a minimum level of acceptable performance, or "preventing" the worst performance. Thus, licensed homes were less likely to have the lowest scores on such measures as availability of social aids, physical amenities, safety features, and an environment with little diversity and a very institutional atmosphere.
The study also found that neither extensive regulation nor licensure had a positive effect on some aspects of quality, including a requirement for preservice training of staff and staff knowledge of care, monitoring, and medication management. Regulation also had no significant effect on the cleanliness of homes and the availability of amenities or the likelihood that a home would have any licensed nurses (registered nurses or licensed practical nurses) on staff. Furthermore, there was little variation among homes on such issues as unmet health care needs, residents' rights, and indicators of resident satisfaction.
Finally, the report noted that the niche for board and care homes appears to be different in States with extensive regulatory systems compared with States with limited systems. States with extensive regulatory systems have higher-than-average board and care bed supply and lower nursing home bed supply. Homes in States with extensive regulation have residents with higher levels of disability than homes in States with limited regulatory systems. Thus, it appears that States with extensive regulatory systems were using board and care beds as substitutes for nursing home beds, particularly in comparison with States with limited regulatory systems.
The study findings have several important implications for key participants in the board and care sector. Findings point to a board and care population that is considerably more frail and disabled than it was 10 years ago. Furthermore, the mix of physically frail elderly, cognitively impaired elderly, and residents with mental illness and developmental disabilities (some of whom are nonelderly) presents a complex caregiving challenge. These factors should prompt a reexamination of the health and safety issues that confront board and care providers and the States' systems for regulating the industry. Of primary importance are the range of services, staffing patterns, and staff training and knowledge needed to meet the needs of today's residents.
This study confirms that there is a well-defined role for board and care homes in the provision of long-term care. The findings specifically suggest the following:
Office of Disability, Aging, and Long-Term Care Policy
Floyd Brown
202/690-6613
PIC ID: 4720
Research Triangle Institute, Research Triangle Park, NC
Medical technology embraces innovations in medicine, such as new drugs, biologics, medical devices, and procedures, as well as existing therapeutic and diagnostic capabilities. Medical technology has been identified as a major factor driving the increase in national expenditures for health care services. Thus, the evaluation of the clinical and cost-effectiveness of medical technology is of substantial interest to many parties. Technology assessment is the term most often applied to such evaluations. A strong national system of technology assessment has not emerged in the wake of a series of failed efforts by the Federal Government. With rapid changes in health care, particularly the rise of managed care, the demand for technology assessment has increased in the private sector. It is not clear if technology assessment activities in the private sector are replacing the assessment functions once expected of Government.
This report briefly describes a number of private-sector health organizations that are engaged in technology assessment. Technology assessment is one of a family of evaluation activities occurring within these health care organizations. Private-sector technology assessment is highly evidence based. Rigorous evaluation of clinical effectiveness based on a systematic review of scientific and clinical evidence has become the norm among private organizations conducting technology assessment. Technology assessment in this sector has increasingly become "full service," encompassing drugs, medical devices, and clinical procedures.
This report considers the use of technology assessment by managed care organizations, which constitute the most significant change in the financing, organization, and delivery of health care. Managed care organizations have strong incentives to hold down the costs of care, and one way to do so is to avoid providing unnecessary, inappropriate, or inadequately tested medical technologies and procedures. This study aims to answer the following questions: What is the demand for technology assessment? Who are the performers of technology assessment? What characterizes the conduct of technology assessment? How are the results of technology assessment used? What unmet needs for technology assessment might be met by cooperation between the public and private sectors of health care?
Much attention has been given in the past two decades to a strong governmental role in technology assessment, a term used to describe the evaluation of the clinical and cost-effectiveness of medical technology. Yet the Government bodies established to conduct such evaluations have not survived the budget cuts and political environment of the 1990's. At the same time, large corporate purchasers of health care have become more active in seeking to rein in the growth of health care expenditures. Managed care organizations have responded to market opportunities by enrolling an increasing proportion of patients, reducing excess hospital capacity, and shifting care away from physician specialists to primary caregivers. In this context, the evaluation of clinical practice for cost-effectiveness has assumed greater market value, and the demand for technology assessment has increased. There are two main sources of increased demand for technology assessment: the press for cost containment and performance evaluation and the need to be responsive to a changing health care marketplace.
With the failure of health care reform efforts in the early 1990's, and with the emergence of managed care, new questions have emerged about the extent of technology assessment activity in the private sector and whether these activities fulfill the functions that were once expected of the Federal Government. This study briefly describes a number of private-sector health organizations engaged in technology assessment.
The scope of this study was narrowed to focus on managed care. Data collection was accomplished through site visits, interviews, attendance at meetings, document review, and follow-up interviews. Since the Blue Cross/Blue Shield Association and The Emergency Care Research Institute had long been involved in technology assessment, they were the primary contacts for data collection. In total, 46 interviews were conducted with 56 individuals in 29 different organizations, including performers of technology assessment, third-party insurers, managed care organizations, professional organizations, and other settings. This report is not a comprehensive enumeration of all the organizations engaged in technology assessment, but is a selective picture of the major players; thus, the extent to which these practices are diffused is unknown.
In the past decade, a strong, decentralized technology assessment capability has developed in the private health care sector of the United States, consisting of a small number of national organizations and a growing number of larger health plans. Private organizations are conducting technology assessment in response to private-sector demands. These assessments are often designed to serve special clientele and require a significant number of internal staff resources.
Some national insurers and managed care organizations conduct centralized technology assessment efforts that serve member plans. Several membership organizations have constituency-oriented technology assessment programs, and some individual managed care organizations have significant technology assessment capabilities. In short, the private-sector technology assessment system is a mix of distributed centralized (national subscription, plan, and constituency efforts) and decentralized (regional and local health plans) efforts. Managed care has strong incentives to support and conduct technology assessment.
Several issues in technology assessment appear to have been settled in the past decade, including the organizational decisions supported by such assessments, the evaluative criteria for assessing a technology, the means of setting priorities for conducting assessments, and the stage of technology that is the object of the assessment. However, issues of how technology assessment applies to clinical trials and the development and maintenance of data bases are yet to be resolved.
In general, the evaluative context within which technology assessment is conducted has changed, focusing more on effectiveness, outcomes, appropriateness, and clinical practice guidelines. There has also been a shift toward evidence-based assessment, that is, a deemphasis on intuition and more emphasis on systematic clinical experience as sufficient grounds for clinical decision making. Although consensus processes are used to interpret and apply the results of technology assessments, the evidence-based movement discounts very heavily the use of consensus to assess the scientific bases of clinical practice.
Driven by cost-containment and quality-assurance objectives, health plans are increasingly examining the clinical effectiveness of interventions in relation to their financial implications. The use of technology assessment in the managed care sector is primarily to support coverage decision making. However, a number of innovative developments go beyond this narrow range of issues. These include benchmarking for member organizations of evaluative activities believed to be needed for survival in the current marketplace and providing a forum for reviewing the evidence related to competing technologies. They also include educational efforts in some plans, directed at changing physician behavior toward evidence-based medicine. Finally, the working limits of technology assessment have been broadened to include the evaluation of drugs after Food and Drug Administration approval for marketing.
An examination of private-sector technology assessment leads back to questions about the appropriate role of the Federal Government in technology assessment and the appropriate division of labor between the public and private sectors. One strong implication is that the Federal Government, as purchaser of health care services, should be no less competent and effective than the private sector in supporting and conducting technology assessment. However, this does not necessarily imply a centralized Federal Government leadership role, which does not appear feasible at present. Although the Federal Government has not shown a leadership role in technology assessment, it can play a significant role as sponsor of technology assessment-related research.
Office of the Assistant Secretary for Planning and Evaluation
Cheryl Austein-Casnoff
202/690-6102
PIC ID: 6180
RAND Corporation, Santa Monica, CA
This investigation examined policy-relevant issues pertaining to the development of more effective and efficient health care delivery systems. The evaluation report provides an overview of and historical perspective on subacute care, which generally refers to skilled care for patients with complex needs outside of the acute care, or hospital, setting. In addition to describing the current state of the art in subacute care and how it varies across selected market areas, the evaluation examines the evidence about its cost and effectiveness. Implications of the study's findings are discussed, and key policy issues related to the future of subacute care are presented. The study concludes that the emerging concept of subacute care holds promise in terms of better care and lower cost, but that it also poses many challenges for the public and private sectors.
This study was initiated by ASPE to examine policy-relevant issues pertaining to the development of more effective and efficient health care delivery systems and involving substantial public and private expenditures. These issues encompass the definition of subacute care; variations in subacute care patients, providers, and settings; incentives and barriers to the growth of subacute care; and the current and potential size of the market. The primary purpose of the study was to provide useful information to both the public and the Department, including a synthesis of subacute care research, practices, and trends and an assessment of public policies related to subacute care. The way in which the term "subacute care" has evolved and the extent to which the ideal vision is currently implemented in practice are key subjects of the study.
Public spending for post-acute care services increased in the past decade from $3.4 billion to over $12.1 billion, an average annual growth rate of nearly 25 percent. In contrast, Medicare spending for acute care services grew by only 6 percent per year during the same period. If the current system is unchanged, Medicare expenditures for post-acute care and acute care are expected to rise even more rapidly as the number and proportion of the older adult population increases. Escalating costs and shifting demographics have prompted some providers to advance the concept of "subacute care" as a cost-effective alternative to inpatient acute hospital care. Proponents believe that subacute care has the potential to generate savings and improve patient outcomes.
At the time of this study, there were many definitions of subacute care. Historically, "subacute" described care provided for hospitalized patients who did not meet established criteria for a medically necessary acute stay. Currently, the term refers almost exclusively to patients treated somewhere other than in acute-care beds. The concept now focuses on shorter stays that result in improved rehabilitation for lower cost, using specific procedures designed to achieve better outcomes for patients whose service needs fall somewhere between those traditionally provided by acute care hospitals and nursing facilities.
Little reliable information is available about the likely impact of subacute care on costs, quality, and access. This lack of data may be related both to differing definitions of "subacute care" and to the recent and rapid expansion of the industry.
Initiated in August 1994, this study had several methodologies: a comprehensive literature search and review of published and unpublished documents; the establishment of an advisory group comprising public- and private-sector experts; interviews conducted with additional national experts from a variety of fields and settings; site visits to 19 institutional providers in four market areas (Los Angeles, Miami, Boston, and Columbus, Ohio); interviews with seven home health care providers; telephone interviews with eight home care firms; and a series of stakeholder interviews.
The study team concluded that insight into the different ways in which "subacute care" is defined is critical to understanding the subacute care phenomenon. The term commonly refers to skilled care for patients with complex needs that some nursing facilities, home care providers, and others have been providing for years under a variety of names. There is, in addition, a growing movement to create a new type of subacute care. The core elements of this idealized, prototypical form of subacute care include a program organized around particular disease categories, specific interventions, or homogeneous patient characteristics; a focus on achieving measurable outcomes in a more efficient and lower cost manner; special resources, such as certain physical plant characteristics and more and better trained staff; and a set of techniquesþfor example, the use of interdisciplinary teams, case managers, care maps, or critical pathway protocols; evaluation of outcomes; and an emphasis on continuous quality improvement.
Estimates of the number of days of subacute care provided each year in this country range from 1.2 million to 8.1 million. It was not possible to develop a reliable estimate of the current amount of subacute care provided. Distinguishing characteristics of subacute care are not captured in any large national data base. In addition, the study team encountered difficulties in identifying and comparing subacute care providers, in finding subacute care providers in reportedly better developed markets, and in obtaining data on subacute patients.
Factors shaping the development of subacute care nationally include the growth of managed care; efforts of managed care providers to find more cost-effective services; the implementation of new Medicare payment policies applicable to acute and post-acute care providers; changes in patient preferences, such as a desire for a homelike environment for patients with minimal nursing needs; and the growth of publicly owned, for-profit, post-acute care companies. Industry growth is also being driven by facility-based subacute care providersþskilled nursing facilities, freestanding and hospital-based nursing facilities, rehabilitation hospitals, distinct-part rehabilitation units, and long-term hospitals. The establishment of minimum standards and broadly defined quality guidelines by the Joint Commission on Accreditation of Health Care Organizations and the Rehabilitation Accreditation Commission has also influenced the subacute care industry.
The study team found that home infusion therapy and full-service home health agencies were providing a product with many of the elements of the prototypical subacute care facility. In response to the growing potential for substituting home health care for facility-based subacute care, many interviewees expressed concern about the quality of care, staffing qualifications, the role of managed care, access issues for lower-income patients, the perception of a "no-care zone" for patients with lower acuity, and the Medicare reimbursement policy for home infusion therapy.
Little evidence was discovered to support the premise that shifting patients earlier from hospitals to subacute care will save money. It was noted that a national study of potential subacute care savings for Medicare was based on several questionable assumptions. There are only a few empirical studies on quality in subacute care facilities; these have generally found poorer outcomes for patients in skilled nursing facilities than for those in rehabilitation hospitals. The four available studies comparing functional status outcomes for rehabilitation patients in hospitals and skilled nursing facilities involved a limited number of facilities; none was nationally representative.
In conclusion, the study team determined that the emerging concept of subacute care is appealing because of the new attention it has focused on some types of patients and the programs envisioned in the ideal. While site visits identified some subacute care providers that are successfully applying elements of the concept, many other services labeled "subacute care" fell short of the ideal. Despite aggressive marketing, the study team found that much that is called "subacute care" is little more than a new name for care provided to higher acuity, medically complex patients or to those requiring more intensive therapies. The idealized approach to subacute care promises similar or better care for lower cost but poses major challenges to both the public and private sectors.
Significant proposed legislative changes to Medicare and Medicaid have heightened the need for accurate and reliable information about subacute care, including a rigorous examination of the potential savings. This kind of information can assist Federal and State Government policymakers in determining their role in the evolution of subacute care and in understanding the implications of their actions on public and private expenditures, quality of care, patient access, and provider equity. This study underscores the importance of finding replicable technologies for producing better value, not just cheaper care, and of developing appropriate payment policies that achieve both savings and value.
Office of Disability, Aging, and Long-Term Care Policy
Jennie Harvell
202/690-6613
PIC ID: 5951
The Lewin Group, Fairfax, VA
Cost-effectiveness analysis (CEA) is one tool available to decisionmakers to help determine the relative value of different approaches to improving health or life expectancy. For a variety of reasons, however, CEA has not been widely used in health policy settings. This study represents a review of the methodological and theoretical aspects of CEA, its effectiveness, and its expected outcomes and uses. The resulting report is a consensus among many of the top researchers in the field and provides guidance to the conduct of CEA. The report sets forth several overlapping recommendations: some intended to improve the overall quality of CEA and some intended to set a research agenda for further improvement of CEA. Recommendations are made in the areas of framing and designing the study, identifying and valuing outcomes, assessing effectiveness, estimating costs, discounting, reflecting analytical uncertainty, and reporting analyses. It also discusses practical constraints to CEA's policy uses, carefully taking into account the needs of its varied audiences.
The purpose of this study was to assess the current state of the art of the use of CEA in health care and to recommend steps to improve its quality and comparability. The Panel on Cost-Effectiveness in Health and Medicine, working closely with methodologists and liaisons from the Public Health Service and the Health Care Financing Administration attempted to identify a number of methodological problems in existing CEA's that limit the analyses' policy relevance and usefulness.
Society is confronting many difficult choices regarding the provision of health care services and the funding of public health programs to improve the Nation's well-being. The aging of the population and the costs of expanding medical technologies have placed growing pressure on health care resources. The private sector, which has long paid for health insurance for a significant proportion of the population, is confronted by competitive challenges and shifting nationwide economic trends. In the public sector, there are competing demands for tax dollars to fund health programs that range from the clinical care supported by Medicare and Medicaid to the population-based approaches employed by public health programs. The Nation is faced with an apparent series of tradeoffs among priorities, as we seek to make wise investment decisions to improve health.
There are many ways to resolve these choices, none of them straightforward and none without compromise. To make the most informed decisions, information is needed about the impact of programs and interventions, their costs, and what is given up when one program is implemented at the cost of another. This is true for Government, managed care organizations, providers, employers, and consumers. As the pressures to control U.S. health care spending have accelerated, the term "cost-effective" has found an increasingly broad and interested audience. But this term often has different meanings to groups as disparate as the U.S. Congress, business leaders, managed care organizations, the pharmaceutical industry, and the public. A more systematic and reliable approach to determining relative values through CEA is required.
This report summarizes the work of the Panel on Cost-Effectiveness in Health and Medicine, a non-Federal expert panel appointed in 1993 by PHS. The panel consisted of 13 scientists and scholars with expertise in CEA and related methods. The panel was charged with assessing the current state of the science of CEA and with providing recommendations for the conduct of studies to improve its policy relevance and utility. The panel met from 1993 through 1995 in collaboration with methodologists from Public Health Service agencies and the Health Care Financing Administration.
The panel organized itself into nine working groups focusing on specific topics: (1) the role of CEA in decision making; (2) CEA's theoretical foundations; (3) the appropriate framing of a study; (4) measuring effectiveness; (5) valuing outcomes; (6) measuring costs; (7) discounting considerations; (8) handling uncertainty within a study; and (9) guidelines for reporting analyses. Panel members and staff drafted papers on topics addressing the major controversies in each area. The papers contained recommendations that were debated by panel members in successive meetings until consensus was reached and formal recommendations could be issued or, in a few cases, until it became clear that consensus could not be reached. In areas where consensus was elusive, the report describes the full range of arguments presented.
CEA furnishes a framework for evaluating the economic and health impacts of different types of investments and can inform a wide range of policy decisions. The results of a CEA are typically summarized as a cost-effective ratio, for example, $138,000 per quality-adjusted life-year, in which the denominator reflects the gain in health from a specific intervention (e.g., years of healthy life gained, premature births averted, or sight-years gained) and the numerator reflects the cost in dollars of obtaining that gain.
A well-constructed CEA provides information that helps decisionmakers identify which strategies will best serve their programmatic and financial objectives. Depending on the type of analysis conducted, a CEA can compare similar or very different approaches to a given problem, such as strategies for decreasing cardiovascular disease. For example, State and Federal decisionmakers with responsibility for Medicaid programs, as well as for public health systems, may wish to know where investments of public dollars will have the largest impact in decreasing premature death from heart disease. In the private sector, a managed care organization might wish to know how extending its coverage to include cardiovascular disease prevention programs would affect the organization's expenditures.
Some decisionmakers may also wish to make comparisons among health conditions and programs; for example, they may wish to know the efficiencies of investments in interventions to improve neonatal outcomes relative to investments targeted at decreasing cardiovascular disease in adults. Ideally, CEA should allow its users to assess the relative value of dissimilar health-producing programs or treatments.
Unfortunately many CEA's are inadequate to these tasks. Reviews of CEA's chronicle a troublesome failure of CEA practitioners to adhere to basic analytic principles; there also is wide variation in sources and the quality of information on costs and effectiveness. Concerns about the discretionary nature of the application of analytic methods and the economic incentives of commercial sponsors of some analyses have resulted in a decision by at least one medical journal to restrict publication of any CEA where authors may have a financial conflict of interest.
Methodological inconsistency has often stymied the application of CEA in important public policy decisions. For example, in the Oregon Medicaid experiment, in which the State legislature attempted to expand coverage to uninsured Oregonians by basing inclusion of services on the cost-effectiveness of different treatment-condition pairs, deficiencies in CEA technique and knowledge were partly responsible for the fact that the initial list of included services lacked credibility. In a more narrow application to policy needs, a review by the National Cancer Institute of an extensive literature on the cost-effectiveness of screening mammography found that the study results ranged from a finding that mammography would be cost saving to a finding that it would cost nearly $84,000 per year of life saved. Confusing results can lead to confusing policy.
The imprecision attached to the term "cost-effective" stems in part from the variety of masters the concept serves: purchasers of health care, who use the term to convey a careful assessment of the value of different health care services; producers of health care technologies and programs, who use the idea to support marketing claims; and advocates for particular constituencies or illnesses, who use the term to garner resource investments. But the imprecision also comes from the manner in which methods of performing CEA have evolved over the past three decades. CEA is an analytic tool whose fundamental purpose is to provide information to decisionmakers about the relative value of different approaches to improving health, life expectancy, or both. Architects of the field and analysts who apply CEA methods come from a number of academic disciplines, including economics, medicine, operations research, medical sociology, psychology, public health, and ethics. Each discipline brings a particular set of concepts and a unique language that have been melded in the building of the CEA technique.
The panel has produced a book, Cost-Effectiveness in Health and Medicine, summarizing CEA methodology with the objective of improving its usefulness to policymakers. The report will be distributed to Federal decisionmakers and analysts to inform them about the interpretation of existing CEA and ways to improve studies conducted by and for the Government. The interdepartmental group is planning broader dissemination and discussion of the panel's work through an international conference. The conference will explore the implications of the panel's recommendations and will identify next steps in moving toward the production and use of high-quality comparable CEA.
Office of Disease Prevention and Health Promotion
Kristine McCoy
202/205-8180
PIC ID: 5619
Office of Disease Prevention and Health Promotion, Washington, DC