| |
Office of the Assistant Secretary |
Policy Information Center |
In this chapter, the U.S. Department of Health and Human Services (HHS) highlights evaluations of general interest to the public health and human services community and illustrates the diversity of HHS evaluations completed in fiscal 1995. Included are summaries of 11 evaluation projects selected by the Senior Editorial Advisors on the basis of criteria identified in appendix C and applied to 38 reports nominated by HHS agencies. These criteria are as follows:
Each summary includes a brief abstract; a description of the study, including its purpose, background, methods, findings, and use of results; the names of any publications that resulted; and the name and phone number of the person to contact for additional information.
This final report describes the design, implementation, and evaluation of the Iowa Patch project, which was undertaken to improve the quality, accessibility, and accountability of social services to children, youth, and families in a small "patch," or geographic area, of Cedar Rapids. By the end of the project, services had been integrated and improved, a community-based interagency service team had been developed, and agency turfism had been reduced. The patch team, including line workers, were more likely to use assessment tools as a team, to share tasks, and to support one another with greater role flexibility. The focus of services shifted from a crisis orientation to a more preventive, proactive approach.
A 3-year demonstration grant was provided to the University of Iowa by HHS's Administration for Children and Families. Its purpose was to overcome categorical barriers that prevent the pooling and use of resources to deliver flexible services that build on the strengths of individuals, families, and neighborhoods. As defined by its advisory committee, the Iowa Patch project's goal was "to improve the quality of life of children and families by providing accessible services that are responsive to family and individual needs in the context of the consumer's local community, developing the existing resources and building on the strengths of the neighborhood."
Funded from October 1991 through September 1994, the project replicated elements of the patch system of community-centered service delivery developed in the United Kingdom during the 1970's. Teams of human services workers with different levels and types of skills were placed in neighborhood offices that served about 10,000 people in Linn County, Iowa. Using a community-oriented approach to social work practice, these teams were able to provide more accessible, flexible, and holistic services based on their knowledge of the local cultural and physical environment and on the formal and informal relationships they established.
In Iowa, patch was viewed as a concept that, if appropriately replicated, could complement other approaches to system restructuring, including the Child Welfare Decategorization project that was developed to reshape the service system. Selected Iowa counties, including Linn County, were permitted to merge their share of various funding streams into one child welfare fund. From the outset, the Iowa Patch project was adopted as an initiative of the Linn County Decategorization project. The demographics of the 9,921 residents of the patch area included higher rates of poverty, unemployment, child placement, single-parent homes, and poor housing than those in the rest of Cedar Rapids or in the State. Many children and families had multiple needs, and human services agencies were heavily involved in the communities at the project's outset.
To achieve project goals, a patch process model was designed and implemented using five primary methods: (1) integrated governance involving agencies, community organizations, neighborhood residents, and service users; (2) localized service delivery; (3) a core interagency team generating new roles and methods; (4) an extended team coordinating service delivery across a wide range of other public and private organizations; and (5) a single point of entry to the service network so consumers could access help with fewer barriers.
A logic model was developed for the project evaluation that included problems endemic to service delivery in the United States; substitutionism (formal services substitute for, rather than support, informal systems of care); turfism (agencies protect their service domains); fragmentation; unresponsiveness to local conditions and consumer needs; deficit orientation (a focus on weaknesses rather than strengths of families and neighborhoods); crisis orientation; and bureaucratic rigidity.
The evaluation focused on three aspects of the project:
The project demonstrated the feasibility of integrating services through a neighborhood-based, collocated interagency team of human service workers representing both public and private nonprofit agencies and was accomplished without a large input of new resources.
The patch approach to practice and service delivery reduced problems of fragmentation, deficit- and crisis-orientation, bureaucratic rigidity, turfism, and substitution of formal for informal systems. Researchers observed the interweaving of formal and informal supports, innovative resource mixes, and consumer and citizen involvement. More emphasis was given to prevention, early intervention, and increasing collaboration among service workers. Putting a local team in the neighborhood made human service workers more accessible, more approachable, more able to develop informal relationships with consumers, and more able to establish and work with informal helping networks.
After the grant ended, the Patch project was integrated into an initiative to develop family resource centers. The project established a strong desire to maintain and further develop the patch approach and a commitment to support it financially and administratively.
For State or county agencies with statutory responsibility for the protection and welfare of children, the patch process offers an innovative approach to building new partnerships. It may be appropriate for replication in other geographic areas, in other populations, and in other types of service delivery systems. The findings also have implications for institutions that educate human service workers in terms of developing curricula, field placements, and research opportunities that prepare faculty and students for integrated practice models such as patch.
Administration on Children, Youth and Families
James Dolson
202/401-0133
PIC ID: 5977
University of Iowa, Iowa City, IA
This report describes the most detailed and generalizable study undertaken to date on runaway, throwaway, and homeless youth in the United States, and their associations with high-risk behaviors such as substance abuse, suicide attempts, unsafe sexual behavior, and criminal activities, both inside and outside the home. A high correlation was found between runaway, throwaway, and homeless experience and high-risk behaviors. Researchers concluded that community-based youth shelter services, including prevention, intervention, and treatment components, are critical to this vulnerable population.
This study was undertaken in response to a congressional mandate "to support research on illicit drug use by runaway and homeless youth, the effects on such youth of drug abuse by family members, and any correlation between such use and attempts at suicide" (Public Law 100-690, Section 3511[a][6]). It is the first national study of substance use, suicide, and other at-risk behaviors among youth with runaway, throwaway, and homeless experiences. The study is also the first nationally representative survey on substance use and other at-risk behaviors among youth residing in both federally and nonfederally funded shelters. In addition, it is the first study of its kind with a multicity sample of street youth.
Runaway, throwaway, and homeless youth comprise a vulnerable population at risk for substance abuse, suicide, sexual exploitation, and physical violence. Because of the need for valid, national statistics about these young people, researchers sought to determine the prevalence of such high-risk behaviors in this population, the extent of exposure to drug prevention efforts and treatment services, needs for and access to health care, the relationship between substance use by youth and their families and youth suicide attempts and other risky behaviors, and the number of available youth shelter beds and the proportion of beds occupied.
The study was conducted by the Research Triangle Institute under a contract with the Research, Demonstration, and Evaluation Branch of the Administration on Children, Youth and Families (ACYF). Four data collection components were developed. The first component was a survey conducted in the fall of 1992 of 640 youth between the ages of 12 and 21 who were residing in 23 federally and nonfederally funded youth shelters located in 17 urban areas. The sample was nationally representative.
The second component was a purposive survey of 600 youth who reported a runaway or homeless episode in the previous 12 months. These young people were found on streets or in other public places in 10 urban areas. A small substudy designed to validate self-reports of substance use also was undertaken.
The third component was a probability sample of youth in households surveyed through an interagency agreement between ACYF and the National Center for Health Statistics. Nine questions concerning runaway experiences were added to the Youth Risk Behavior Supplement of the National Health Interview Survey. In 1992 and 1993, 6,496 youth between the ages of 12 and 17 responded to in-home audiocassette interviews.
The fourth component was a mailed survey to directors of a nationally representative sample of youth shelters in 1992 and 1993. Surveys were returned by 160 shelter directors, an overall response rate of 97 percent from directors of eligible shelters.
Researchers found that many youth in the shelter and street study components reported runaway, throwaway, and homeless experiences. Single parents were raising 52 percent of the youth in the shelter component (48 percent in the street component). In the household component, 34 percent of youth did not live with both parents. In the 30 days prior to leaving home, 32 percent of youth in the shelter component (46 percent in the street component) had at least one family member who got drunk or used marijuana or other drugs.
Fifty-six percent of youths in the street component had never used a shelter. In the household component, 75 percent of youth with a runaway experience had not used either a youth or an adult shelter. An average of 10 beds were available in shelters serving only youth aged 12 to 17, while an average of 28 beds were available in shelters serving youth aged 12 to 21. On any given night, an average of 55 percent of the beds in youth shelters were occupied. Only 10 percent of youth in the shelter component and 14 percent of youth in the street component had ever been turned away from a shelter.
In the household component, lifetime substance use among youth with runaway and homeless experiences was high compared with other youth. Eighteen percent of substance users in the shelter component had been in treatment for an alcohol- or drug-related problem (24 percent in the street component). Of the young people who had ever received treatment, 44 percent of those in the shelter component (23 percent in the street component) were in treatment when data were collected.
Suicide attempts were reported by 26 percent of youth in the shelter component and 32 percent in the street component. More youth reported attempting suicide at home than away from home.
Many young people had been robbed (12 percent in the shelter component, 30 percent in the street component), assaulted (15 percent, shelter; 29 percent, street), or sexually victimized (8 percent, shelter; 14 percent, street). Sex was traded for money, food, shelter, or drugs by 7 percent of the shelter component and 21 percent of the street component. Reported crimes committed by youth in the study included theft-related activity (66 percent, shelter; 81 percent, street); assault-related activity (30 percent, shelter; 42 percent, street); and drug-related activities (32 percent, shelter; 50 percent, street). In the household component, youth with any runaway experience were three times as likely as other youth to have carried a weapon in the past 30 days and to have been in a physical fight in the past year.
In the household component, youth with any runaway experience were four times as likely as other youth to ever have had sexual intercourse. At the time of the interview, 12 percent of the females in the shelter and street components were pregnant; 41 percent of females in the shelter component (50 percent in the street component) reported having ever been pregnant.
A strong relationship was found between youth substance use and suicide attempts. Having at least one family member who got drunk or used marijuana or other drugs in the 30 days before a youth left home substantially increased the likelihood that the youth had ever attempted suicide.
This study should be of interest to Federal and State policymakers, community leaders, residential and outreach service providers, law enforcement officials, and other individuals concerned for the well-being of this population. Findings may be useful in shaping public policy, directing resources to this population and their families, improving the design and delivery of prevention and treatment services, strengthening community-based systems and linkages, and undertaking additional research.
Administration on Children, Youth and Families
James Griffin, Ph.D.
202/205-8138
PIC ID: 3871
Research Triangle Institute, Research Triangle Park, NC
This report describes a 1-year project to evaluate long-term care ombudsman programs, which investigate complaints made by or on behalf of residents of nursing facilities and board and care (B&C) homes. The report sets forth recommendations related to program availability, adequacy, and effectiveness for residents of long-term care facilities; compliance with Federal program mandates; adequacy of current Federal, State, and local program resources; and the feasibility of extending ombudsman services to older persons who do not reside in long-term care facilities. Overall, the report strongly supports long-term care ombudsman programs and their continuation and urges a stronger Federal Government role in providing program leadership, guidelines, and oversight.
The purpose of this study is to assess the effectiveness of State-based long-term care ombudsman programs in providing advocacy for the health, safety, welfare, and rights of those who reside in long-term care facilities and B&C homes. Under contract with the Administration on Aging (AoA), the Institute of Medicine convened a multidisciplinary expert committee to evaluate long-term care ombudsman programs and to consider the potential impact of these programs on other target populations, in other settings, and in other circumstances. This evaluation project was mandated by the 1992 amendments to the Older Americans Act.
Long-term care ombudsman programs were created more than two decades ago as a response to serious concerns about the quality of nursing home care. Currently operating in all 50 States, the District of Columbia, and Puerto Rico, these programs are funded from a combination of Federal, State, and local sources.
Over the years, the role of the ombudsman has evolved to include both individual and systemic advocacy. Individual advocacy activities are designed to ensure that ombudsman services are accessible to all residents, systematically receive and respond to resident concerns about their long-term care services, and work cooperatively with a variety of agencies to resolve problems.
Systemic advocacy activities include evaluation of laws and regulations related to long-term care services for older adults, education of the public and facility staff, dissemination of data about the ombudsman programs, and promotion of the development of citizen organizations and resident and family councils. Both roles are important, but the ombudsman programs primarily identify, investigate, and resolve individual resident complaints.
The committee developed a formative evaluation strategy to identify program issues, strengths, and weaknesses. Both qualitative and quantitative data were collected and analyzed using a triangulated approach and a form of recta-analysis, which incorporates multiple dimensions of program effectiveness, individual interpretations, and expert opinions. A technical panel was convened twice and consulted as needed.
Although there is no national database or source of standardized information, numerous sources of primary and secondary data were used. Two- to three-day site visits were made to programs in California, Colorado, Florida, Massachusetts, Minnesota, and Virginia. Seven papers were commissioned, and a thorough literature review was undertaken.
Structured, systematic contacts were made with directors of State units on aging, State and local long-term care ombudsman program offices, long-term care physicians, and consumer advocacy groups. A 1-day invitational symposium, a public hearing, and two "open-mike" sessions were conducted. Discussions were also initiated with four national associations of long-term care facility providers.
The report underscored the vital public purpose served by long-term care ombudsman programs and endorsed the continuation of their mandate, offering several examples of ways in which the programs have provided valuable assistance to individuals and improved the service system. The committee could not offer a definitive assessment of program effectiveness, however, because of the lack of uniform process and outcome data for all States and the lack of theoretical consensus regarding the use of appropriate effectiveness measures. Instead, the committee reported selected findings and encouraged the development of a process to facilitate uniform process and outcome evaluation in the future.
In some areas, the committee found that meaningful access to ombudsman services does not exist for all eligible nursing facility residents because of infrequent site visitation and insufficient information outreach efforts. In general, complaints receive a timely response. Program implementation in B&C homes is much more limited than in nursing facilities.
Considerable variation among States in the extent of program implementation in nursing facilities is noted. At the systemic level, the committee observed that too many States operate fragmented programs that only respond to individual complaints rather than functioning as unified, integrated programs. Many States lack access to adequate, independent legal advice and backup for ombudsmen and for residents who need assistance in courts and regulatory hearings.
The committee also found that current resources are inadequate to support full implementation of program mandates. It recommends that Federal funds for the program be increased from the current $25 million to $39.5 million by fiscal 1998. This increase would provide funding to achieve the committee's recommended staffing level of one ombudsman full-time equivalent for every 2,000 long-term care beds. (The current ratio is one to every 2,700 beds.) The committee also recommended that resources be apportioned among the States on the basis of the number of beds rather than the number of persons aged 60 years and over, as is currently done.
The committee recommended a strengthened leadership and oversight role for the Federal Government, including the implementation of an objective compliance review method for States. An improved data system is needed that includes information about program awareness, as well as the extent of complaint resolution, the extent of ombudsman input into systemic improvement activities, and the extent of ombudsman impact on the overall system of long-term care and services.
Concern was expressed that expanding the program to cover persons who do not reside in long-term care facilities may hinder the ability of ombudsmen to serve their current constituencies in nursing facilities and B&C homes. The committee recommended that expansion capacity, target populations, organizational placement, staffing needs, and funding receive careful consideration before any expansion efforts to non-institutional care settings are undertaken. The importance of developing and supporting research and demonstration projects in this arena was also noted, with attention to demographic and health care system trends affecting the long-term care system.
The report indicates that it was premature to evaluate the impact on residents of B&C homes because long-term care ombudsman programs have been only partially implemented there.
The development of a nationwide database of key structure, process, and outcome program measures is proposed. This study can be used by local and State ombudsman programs to assess their own strengths and weaknesses and to improve their performance. The Federal Government can consider recommendations to strengthen its role as leader, researcher, funder, guide, and monitor. Findings can also be useful to agencies that consider developing, adapting, or expanding ombudsman programs for other populations and services. It is also suggested that program expansion be delayed until existing mandates are fully implemented.
Office of Program Operations and Development
Nancy Wartow
202/619-1058
PIC ID: 5819
NTIS Accession
Number: PB95-174223
National Academy of Sciences, Institute of Medicine, Washington, DC
Increasingly, private purchasers of health care, consumer groups, Federal and State agencies, and health care plans are searching for methods to compare clinical performance among providers to guide choices, ensure accountability, provide data for quality improvement, and track change. Unfortunately, currently used clinical performance measures do not provide meaningful comparisons of clinical performance or, at worst, are actually misleading because they are limited in scope, insufficiently detailed, methodologically flawed, or not standardized across providers. This project is a first step toward identifying gaps between the clinical performance measures that exist and those sought by potential users. Knowledge of gaps permits prioritizing needs for clinical performance measurement and assessing the feasibility of addressing those needs through future research and development.
The objectives of this 6-month study were to (1) collect information on the range of clinical performance measures currently in use; (2) summarize the resulting information; (3) assess the feasibility of deriving various clinical performance measures from existing databases; (4) explore the cost of different strategies for clinical performance measurement; and (5) explore the sampling issues associated with the application of selected clinical performance measures that would be useful in measuring quality of health care plans.
Clinical performance measures are instruments that estimate the extent to which a health care provider delivers clinical services that are appropriate for each patient's condition; provides the services safely, competently, and in an appropriate timeframe; and achieves desired outcomes in terms of those aspects of patient health and patient satisfaction that can be affected by clinical services. Clinical performance measures concern the technical content of health care and assess health care in terms of individual patients. Clinical performance measurement requires summing data about the health care given to many patients to create a rate or score for average performance. Performance can be measured by identifying a representative sample of similar patients and collecting data about the care received by those patients within a given time period. By applying criteria for quality of performance to these data for each patient, good and poor quality care can be determined. Results are then aggregated to form a performance rate or score.
Current measurement techniques are plagued by a variety of flaws. A significant obstacle is that many types of data currently used as indicators of quality are not directly usable for comparisons of clinical performance. For example, utilization statistics (e.g., hospital admissions or rates of surgery) are not helpful unless applied to the individual patient. Measures of health status or patient outcomes are not useful without allowing for the probability that each case would experience a good outcome if good clinical care were provided. Measures of patient satisfaction are flawed because of subjectivity, although patient surveys are useful if patients are asked about the facts of their care.
Although many existing indicators of quality provide inaccurate comparisons of clinical performance, they can serve as an intermediary step toward better measures. For instance, measures of patient outcomes that cannot currently be used to compare clinical performance may become useful for that purpose as methods are developed to allow for patient differences in the likelihood of achieving a good outcome. Similarly, users who currently have inadequate data sources to construct precise clinical performance measures may find it essential to use crude indicators while working to improve data sources.
To be serviceable, measures must be useful internally within health care organizations and must have adequate levels of sensitivity, specificity, and predictive value. Measures must be reliable and valid for their intended purposes, as well as affordable. A reliable typology of performance measurements would allow potential users to select an approach--quality control, choice of health care plan, or accountability--that is appropriate for their own purpose.
A typology of measurements was created, as was a data set that included the relationship of measures to each other, the aspects of clinical performance that the measures addressed, the properties of measures that determine appropriateness for specific uses, and the data needed to create the measures.
Existing and evolving clinical performance measures were identified by two approaches: (1) a literature search conducted by using the Medical Analysis and Retrieval System® database of the National Library of Medicine; and (2) direct personal contact by phone with 112 individuals or agencies known to be involved with performance measurement research, use, or evaluation. The contacts yielded 40 sets of measures consisting of 1,287 clinical performance measures. Data concerning measure attributes were extracted, coded, and entered into six relational databases.
The project developed a classification scheme to assist users in identifying and evaluating clinical performance measures. It developed and defined key attributes of clinical performance measures and applied this framework to 40 measure sets used by public and private organizations to measure and improve clinical quality. The 40 measure sets were classified on 7 dimensions:
In addition to describing general characteristics about the type of measure set, the Typology framework classified measures with respect to their structure (factors such as data requirements, sampling, time window, scoring, risk adjustment, and interpretation) as well as their clinical content (e.g., whether a measure addresses health promotion, early detection, or treatment of a disease; whether it is a process or outcome measure, etc.).
The objective was to develop and test a prototype framework sufficiently flexible to encompass the structural and clinical characteristics of the wide variety of clinical performance measures currently used. The result was a series of interlinked databases containing information on measure sets, batches of measures (measures with similar structure or content), and clinical conditions and events that are associated with the measures.
The interrelated nature of the databases enables users to access data by measure or clinical event. Frequencies or percentages of measures in the various categories were computed and the results presented in graphs. A broad range of performance measures were included in the database (i.e., process and outcome measures, health care setting, demographics of patient population, data derived from clinician judgment and patient perception, and mental and physical health measures). The development of the prototype raised other issues to be considered (e.g., cost and sampling) when developing future databases for clinical performance measurement.
Performance measures were constructed from administrative data (e.g., enrollee files, claims data files, disease registers, pharmacy records, medical records), and from special data collections (e.g., patient or provider surveys).
The typology is proposed as a starting point for a data system that would permit users to find out what measures are available for given conditions and associated clinical events, what data resources are required, and which measures are suitable for the users' specific purpose. The classification framework and its definitions form the basis of a common or uniform language to describe and compare the thousands of clinical performance measures under development and in use today. The framework also serves as a teaching tool to help those interested in learning about how to construct, compare, and evaluate the utility of measures.
The study also concluded that future work is needed to test the framework and prototype databases against the needs of users. To accomplish this objective, the Agency for Health Care Policy and Research (AHCPR) is using the typology framework as the basis for a follow-on project, CONQUEST 1.0, the COmputerized Needs-Oriented QUality Measurement Evaluation SysTem. This project builds on the typology framework in three ways. First, it evaluates the typology by verifying the content of the measure database with measure developers. The verification effort has resulted in enhancements to the typology structure and content. Second, CONQUEST 1.0 builds on the typology by creating a database of information on clinical conditions that can be used to steer the search for appropriate measures. The condition database contains information from AHCPR-supported clinical practice guidelines, clinical practice guidelines produced by other organizations, and medical effectiveness research findings. Third, CONQUEST 1.0 translates the typology into a more useful system by making it available on computer. The project develops a computerized system with a user-friendly interface to link measures to clinical information and guide the selection of measures.
Follow-on efforts are currently under way at AHCPR. One such related project involves evaluating this product by convening users to pilot test CONQUEST 1.0 and participate in focus groups about its usefulness. Also, AHCPR has issued a Request for Contract Proposals for a project called QM-Net, to use the typology and CONQUEST 1.0 as the basis for a national data source for information on clinical quality measures. Information on CONQUEST 1.0 can be obtained through AHCPR's web site (http://www.ahcpr.gov) or through the Agency's clearinghouse at (800) 358-9295.
Office of Planning and Evaluation
Irma Arispe, Ph.D.
301/594-1455
PIC ID: 5630
NTIS
Accession Number: PB 96-144639
Center for Health Policy Studies, Columbia, MD
This study examined the likely effects of managed care reforms on the delivery of childhood lead poisoning prevention (CLPP) programs. Three major service-related components of CLPP programs--screening, laboratory analysis, and case management--were evaluated at selected State and local public health departments (S/L PHDs). Results suggest that, as managed care continues to increase its share of the health care market, there is a possibility that appropriate childhood lead screening will decrease, that the participation of public laboratories in performing blood lead analysis will decline, and that followup services for children with elevated lead levels will be delivered outside the traditional health department setting. Assurance that these activities continue to be performed adequately will require public health monitoring of screening and followup and carefully written Medicaid managed care contracts. This report focuses on CLPP programs, but the underlying question it asks--how can we ensure that important public health services to children (and others in need) will continue as managed care takes over health care systems, including Medicaid--has broad implications for many other programs.
This study was undertaken to (1) gain a thorough perspective on how various State reform efforts using different types of managed care arrangements have influenced, either positively or negatively, the three major service components of CLPP programs; (2) gather detailed information about how selected programs are handling the challenges presented by managed care and, where possible, identify patterns and themes; and (3) identify strategies that might help lead poison prevention grantees and other providers respond creatively to the challenges of a new health care environment.
State health care reforms are diverse, but current trends suggest that privatization of direct services previously delivered by the public sector is on the rise. Marked increases in the cost of health care are driving this trend. By 1993, Medicaid was the single largest and fastest growing component of most State budgets, accounting for 18.4 percent of the States' total expenditures.
In an effort to stem rising health care costs, many States have turned to managed care models to deliver personal health care services to low-income persons and to other vulnerable populations, including those eligible for Medicaid. At this time, all but eight States offer some type of managed care to Medicaid enrollees; approximately 25 percent of all Medicaid enrollees are part of a managed care arrangement, and most of these individuals are in full-risk capitation programs, which generally are responsible for all care rendered. The shift from the traditional indemnity insurance system to a managed care system is expected to alter markedly the delivery of direct and nondirect services. The type of managed care organization (MCO) models that predominate within a State will likely further affect the role of public health agencies.
At both the State and local levels, CLPP programs serve primarily the most vulnerable of populations--children of low-income families. Several mechanisms provide ongoing support for these programs. For example, the Centers for Disease Control and Prevention (CDC), which launched its CLPP initiative in 1989, funds 37 S/L PHDs in an effort to develop comprehensive programs to screen, identify, and provide adequate medical and environmental followup to children with elevated blood lead levels (EBLLs). The future of this and other programs is uncertain, however, because of changes in our health care delivery system. In October 1993, the Lead Poisoning Prevention Branch (LPPB) of CDC commissioned a study to examine the effects of managed care reforms on CLPP programs, with a focus on CDC-funded activities. The results of the CDC-initiated study are presented in this report.
The five sites chosen include three states (Indiana, Rhode Island, and Tennessee), one county (Pinellas County, Florida), and one city (Minneapolis, Minnesota). They serve as models for how S/L PHDs can adapt to a managed care environment, and were selected on the basis of their involvement in a range of managed care environments and their varied responses to managed care. Teams of two contractor staff members and at least one CDC/LPPB staff member conducted the site visits. Initial site visits took place between July and September 1994; followup interviews with key personnel from all sites were conducted by phone in February 1995. Team members interviewed CLPP program staff responsible for each of the major service components of the program. They also interviewed maternal and child health staff, Medicaid staff, and key administrators. A mix of State and local respondents was sought. Where possible, the contractor/CDC team interviewed MCO staff. The report's authors acknowledge that the study's findings are based on data from a limited number of case study sites.
This report recognizes the extent to which managed care has been incorporated into our health care system. For the three components studied (screening, lab analysis, case management), data indicate that services, and therefore the individuals who receive these services, will be adversely affected in the absence of legislated mandates or highly specific contracts written between the State Medicaid agency and the MCOs operating within the State.
Data on the first component--screening--indicated that the number of lead screenings in the surveyed States decreased between 1993 and 1994. In Tennessee, for example, approximately 40 percent fewer screens were reported in 1994 than were projected for that year, but decreases in lead screenings occurred studywide. Possible explanations for the decrease include inadequate reporting and communication between private and State laboratories; poor provider compliance (e.g., with Early and Periodic Screening, Diagnosis, and Treatment program [EPSDT]); and difficulty in obtaining approval for screening.
The report recommends the following strategies to help States ensure that lead screenings are conducted: increase provider-based education and advocacy; legislate a screening mandate; expand reporting requirements; add lead screening to the State's quality assurance indicators; enforce EPSDT; retain some public sector service delivery role, including joining MCO networks as primary care providers; and try to obtain blanket authorization for lead screening within the private sector.
Data on the second component studied--lab analysis--indicated that data collection in general is likely to be severely affected by managed care, particularly in States without a strong and comprehensive reporting law. Historically, CLPP programs have received the data on blood lead tests primarily from public laboratories, whether or not a State had a reporting mandate. Surveyed sites reported success using the following strategies: establish a reporting law; mandate the use of public laboratories for selected tests; establish price (and other) incentives; and focus on the importance that surveillance serves in maintaining and improving public health.
Case management services, including necessary followup treatment and retesting of children with EBLLs, are currently performed by CLPP staff either directly or through links with other public agencies. Data from site visits revealed that private providers and MCOs had a number of concerns about CLPP case management services and admitted to a lack of knowledge of comprehensive care and followup, a lack of ability, and a lack of incentive. The report suggests the following strategies to ensure appropriate and adequate case management: establish reporting laws; identify and separately fund core public health functions; require data collection at the individual person level; require that case management responsibilities be detailed in the MCO contract; and encourage mandated or contractual relationships between MCOs and S/L PHDs.
Finally, the report recommends that S/L PHDs seek to maintain public sector service delivery, build a legal infrastructure to influence the behavior of private providers, and fund core public health functions with monies derived from non-Medicaid sources.
In the evaluation of the activities of the lead programs funded by CDC, MACRO International, Inc., studied the effect on lead programs of the shift to managed care of formerly public-sector patients and assessed three lead poisoning prevention program functions: screening, lab analysis, and case management.
As a result, LPPB was able to improve its guidance to State and local childhood lead poisoning prevention offices about the appropriate language for State Medicaid managed care contracts requiring lead poisoning prevention services. LPPB was also able to provide guidance on improving data systems to monitor managed care performance. Thus, some programs have quickened the pace of their efforts to complete monitoring systems, and others have formed ties with Medicaid agencies.
In addition, investigators studied the community perspective on the childhood lead poisoning prevention efforts of public health agencies. Representatives of two types of community groups were consulted: umbrella organizations, which generally have national or State-level structure and support, and community-based organizations, which are generally local in origin and support. Representatives participated in focus groups to discuss several topics, including their relations with public health agencies and their visions of how to improve these relationships. As a result, some programs have been able to change their tactics for collaborating with community-based organizations, and some have broadened the focus of their collaborative efforts.
National Center for Environmental Health
Nancy Tips
770/488-7277
PIC ID: 4743.1
NTIS Accession
Number: PB 96-144670
MACRO International, Inc., Silver Spring, MD
This negotiated bundled payment demonstration was implemented in May 1991 by the Health Care Financing Administration (HCFA) as one of many cost-containment demonstrations launched in response to increases in Federal spending under Medicare. Its goal was to assess the benefits of a negotiated package pricing arrangement for heart bypass surgery with hospitals and associated physician groups that have capabilities and incentives to manage and coordinate the complex services needed for high-quality care. Participating hospitals and physicians accepted the negotiated global package price as payment in full. The evaluation of the demonstration focused on several factors including quality and appropriateness of care, savings to Medicare, savings to hospitals, changes in patient care and hospital management, and hospital competition and marketing. The study found that both Medicare and hospitals could benefit from this type of payment arrangement through reduced costs and better coordination of services while maintaining high quality of care.
This study examined the impact of a Medicare demonstration project to test the overall feasibility and cost-effectiveness of paying hospitals and physicians a single negotiated amount for all hospital and inpatient professional services associated with coronary artery bypass graft (CABG) surgery. Among the key questions assessed were the following:
Throughout the 1980's, Federal spending under Medicare increased at an average rate of 11 percent annually. Expenditures in hospital care increased threefold and spending on physician services increased fourfold during the same period. Every year, the government spends several billion dollars on inpatient care for coronary artery bypass procedures provided to Medicare beneficiaries. A major concern of both hospital managers and policymakers in controlling inpatient costs for high-technology procedures is the asymmetry of financial incentives faced by hospital staff versus physicians. Currently, hospitals are paid for bypass surgery on a per case basis, primarily within the Diagnosis Related Groups (DRGs 106 and 107). Except for extraordinary outlier costs, hospitals are paid a fixed amount regardless of the intensity of care provided each patient. Although surgeons, like hospitals, receive a bundled fee for inpatient services, other physicians, by contrast, are paid for every additional service they provide, including routine daily hospital visits and consultations. In addition, surgeons receive higher compensation for more complicated bypass surgeries. Moreover, physicians do not bear the hospital's financial risks of keeping patients in the intensive care unit longer or using more expensive pharmaceuticals. The rationale for the demonstration was that a global fee that includes physician services would align hospital and physician incentives and encourage physicians to use institutional resources in a more cost-effective manner.
The research demonstration was conducted from May 1991 through June 1996. HCFA ultimately selected seven hospitals to serve as demonstration sites during the term of the study. Selections were based on the completeness of the package of services each hospital was willing to provide Medicare beneficiaries under the demonstration, the quality of care provided by the hospital and its physicians, and the size of the discount the hospital and physicians were willing to accept for their services. The demonstration was implemented at four sites in May 1991: Saint Joseph Hospital of Atlanta, St. Joseph Mercy Hospital in Ann Arbor, the Ohio State University Hospital in Columbus, and Boston University Medical Center Hospital. In the spring of 1993, three more sites were added: St. Luke's Episcopal Hospital in Houston; St. Vincent Hospital and Medical Center in Portland, Oregon; and Methodist Hospital in Indianapolis.
Under the demonstration, participating hospitals and physicians received a global payment covering hospital and related physician services, including outliers, for each CABG procedure (DRGs 106 and 107). In each case, the negotiated rate represented a discount from what Medicare paid, on average, for these procedures. Depending on the package proposed by the individual hospital, the package of services also included pre- and postdischarge physician services and some cardiac rehabilitation services as well as readmissions within a specified time after discharge.
Selected sites received the designation of Medicare Participating Heart Bypass Centers and were encouraged to market their services to referring physicians and beneficiaries and to offer incentives to attract patients. Quality assurance was carefully monitored by HCFA as well as by project evaluators. Hospitals not participating in the demonstration continued to provide services under the traditional Medicare fee-for-service program, and Medicare beneficiaries were free to choose between demonstration and nondemonstration hospitals.
A two-phase evaluation strategy evolved. Phase I covers implementing the demonstration and evaluating the original four sites for a 2-year period while focusing on marketing, volume increases, quality of care, and developing and implementing the appropriateness model. In addition, the phase I evaluation gathered baseline data and first-year operational statistics for the three sites that began the demonstration in May 1993.
A 3-year evaluation contract was awarded to The Lewin Group and its subcontractor, Health Economics Research, Inc. After the first evaluation contract ended, a new evaluation contract was awarded in 1994 to Health Economics Research, Inc., which then continued the evaluation for the remaining 2 years of the demonstration.
An interdisciplinary group of economists, physicians, and marketing experts assembled a variety of databases and conducted numerous onsite interviews with participants as part of an extensive quantitative and qualitative evaluation of the demonstration. Methodologies included two quasi-experimental designs, time-series studies, case studies, surveys, and interviews. Several major literature reviews also were undertaken, including one on efficacy and risks of CABG surgery. A separate appropriateness study was conducted, in which an expert technical advisory panel was convened to help formulate a model of appropriateness indicators for CABG surgery. Information from the literature reviews and the appropriateness study was used to draft materials to assess quality of care at the demonstration sites.
Medicare claims were used to document national trends in Medicare bypass volumes, patient demographics, lengths of stay, mortality rates, and costs. Physician costs were broken into three categories: 30 days prior to bypass surgery, inpatient, and 90 days post-discharge. Prices negotiated with each demonstration site were compared with predicted Medicare prospective payment rates and physician inpatient outlays to determine the immediate savings from the demonstration. Market share savings also were calculated.
Every demonstration hospital provided a set of clinical information on each patient, including discharge status, risk indicators, admission priority, age, gender, and previous bypass operations. Additional information was provided on disease anatomy. The seven demonstration hospitals were compared extensively by using logistic analyses.
HCFA received over 200 letters of interest and preapplications from the initial request for participants in the demonstration, indicating that many hospitals are willing to work jointly with their medical staffs to develop the data necessary to submit a single proposed price. Of the four initial demonstration participants selected, the two nonacademic medical centers experienced statistically significant increases in Medicare bypass market shares. The third had a significant increase in market share, and the fourth had no increase.
In the first 2.5 years of the demonstration, total Medicare savings (Medicare program and beneficiaries and their insurers) were estimated at $17.6 million. Three of the four original hospitals were able to make major changes in physician practice patterns and in hospital operations that generated significant cost savings. Physicians in the three hospitals were able to quickly and dramatically reduce the length of inpatients' stay, substitute generic for brand drugs, and reduce unnecessary testing and other services. In the one hospital where surgeons resisted attempts to change practice patterns, costs continued to rise. The study found that alignment of physician and hospital incentives facilitated a closer working relationship between physicians and hospital staff.
One of the first four demonstration sites significantly reduced its patient mortality rate during the course of the demonstration. Most participating hospitals reduced intensive care unit stays by 1 full day and total hospital stays by 2 days. For the first 3 years of the demonstration, there was some evidence of a growing severity in case mix, including a higher percentage of patients over the age of 80 with comorbid conditions. Complication rates appeared to increase commensurately during this period. Nevertheless, these factors did not produce an upward trend in mortality.
The demonstration involved major changes in Medicare payment arrangements. According to providers, patients were satisfied with the single copayment amount. Hospitals, in general, were pleased with HCFA's prompt payment, which was received within 30 days by wire. Supplemental insurers responsible for paying patient deductibles and coinsurance amounts were uniformly displeased with the flat actuarial payment calculated by the government because it was incompatible with their computer systems, and patient policies differed in their deductibles, coinsurance amounts, and so forth. However, HCFA intentionally had calculated an artificial copayment amount that was comparatively low, intending to share the savings with the beneficiary--although in most cases, ironically, their supplemental insurers reaped the benefit.
By January 1996, more than 9,900 cases had been performed under the demonstration, with an estimated savings of nearly $38 million to the Medicare program. The demonstration has shown that it is feasible for Medicare to negotiate a bundled payment for an episode of care that represents an appreciable savings to Medicare while maintaining quality of care for the beneficiary. The demonstration also has shown that an all-inclusive global fee for both hospital and physician services can align incentives to encourage hospital managers and physicians to cooperate in using institutional resources in a more cost-effective manner.
The success of the demonstration has led to the inclusion of language in the President's 1997 proposed budget package calling for new legislation to implement negotiated bundled payment arrangements under the regular Medicare program. The experience gained from this demonstration has led HCFA to develop a new bundled payment demonstration that expands the concept to a group of orthopedic and a group of cardiovascular procedures. The Participating Centers of Excellence Demonstration for Orthopedic and Cardiovascular Services is expected to be implemented at selected hospitals beginning in mid-1997.
Office of Research and Demonstrations
Armen H. Thoumaian, Ph.D.
410/966-6672
PIC ID: 5958 and
5958.1
The Lewin Group, Fairfax, VA
This report to Congress examined the impact of legislation that reformed Medicare payment methodology to physicians. Through analysis of Medicare databases and national surveys, the report examined the use of, and access to, Medicare services before and after the new payment system was launched. Among the major findings was that legislative reforms in physician payments led to the outcome that Congress intended: the legislation shifted utilization away from surgical procedures and toward medical evaluation and management. According to the report, the legislation did not create new barriers to care for vulnerable patient groups. The report also found strong racial and income disparities in utilization. Finally, the report provided directions for future study of the relationship among income, race, and access to care.
The purpose of this report was to keep Congress abreast of changes in Medicare patients' access and utilization following the passage of legislation in 1989 that introduced changes in Medicare physician payment policies. The report sought to provide detailed analyses of whether the legislation resulted in a shift to greater reliance on primary care and medical specialties and less reliance on surgical specialties. The report also sought to analyze barriers to, and satisfaction with, physicians' care and the relationships among race, income, health status, and utilization.
This report represented the fifth in a series of annual reports required by Congress under the Omnibus Budget Reconciliation Act of 1989 (Public Law 101-239). One key purpose of this legislation was to create a more rational and equitable payment system for physicians' services under Medicare. The law mandated a Medicare Fee Schedule (MFS) that fundamentally changed the way physicians were paid. Instead of doctors' payments being based on what they charged for each service, payments are now based on a relative value scale, which reflects the relative costs of the resources needed to provide various physician services. The MFS shifted physician payments from surgical procedures to evaluation and management services. The law also placed limits on billing for amounts above the MFS fee and instituted target rates of growth in Medicare physician expenditures.
Researchers from the Health Care Financing Administration prepared the main body of the report, which summarized and integrated all analyses conducted by HCFA and by the Center for Health Economics Research. The analyses, contained in 12 separate appendixes, were based on several Medicare databases and ongoing national surveys. For example, utilization information was derived primarily from Medicare's national claims history files and its Medicare provider analysis and review (MEDPAR) file. Other utilization information came from the National Health Interview Survey conducted by the National Center for Health Statistics and from the Medicare Current Beneficiary Survey conducted by HCFA. Income information was estimated from U.S. census data based on beneficiary zip codes from the claims files. Information on patient satisfaction was derived from the Medicare Current Beneficiary Survey. To determine the impact of instituting MFS, the analyses examined trends during the years after the new system was established (1992-94) and made comparisons with data from the year before (1991).
The report's major finding was that the legislation achieved its intended objective to enhance the use of medical services and lower the use of surgical services. For example, from 1992 to 1993, claims for medical visits and consultations increased by 9.3 percent, while claims for doctors' services related to surgery, radiation therapy, and anesthesiology declined by 2.4 percent (see table II-1). These trends were consistent with those found during the first year of operation (1992) relative to the final year of the conventional fee-for-service system (1991).
Year |
Total |
Medical Visits and Consultations |
Surgery, Rad. Ther., Anesthesia, & Assistants |
X-ray and Lab Tests |
All Other |
|---|---|---|---|---|---|
| Allowed Charges (in millions) | |||||
| 1993 | $46,124 | $16,311 | $12,984 | $8,962 | $7,868 |
| 1992 | 43,942 | 14,926 | 13,301 | 8,785 | 6,930 |
| 1991 | 42,915 | 13,885 | 14,116 | 8,727 | 6,186 |
| Percent Changes | |||||
| 1992-93 | 5.0 | 9.3 | -2.4 | 2.0 | 13.5 |
| 1991-92 | 2.4 | 7.5 | -5.8 | 0.7 | 12.0 |
| Adjusted for Changed Population | |||||
| 1992-93 | 3.5 | 7.8 | -3.8 | 0.6 | 12.0 |
| 1991-92 | 0.7 | 5.7 | -7.3 | -1.0 | 10.2 |
Source: Medicare Part B Monitoring System: Allowed charges derived from Medicare National Claims History File. Population information from June 30 Medicare Part B enrollment files.
The report amplified these general trends with specific information about utilization rates for numerous procedures. From 1992 to 1993, there was a decline in rates of many procedures. For example, the rate of sigmoidoscopies (per 1,000 beneficiaries) declined by 8.9 percent; the rate of hip replacements declined by 1.8 percent; and the rate of hysterectomies declined by 5.4 percent.
Overall, the passage of the legislation was associated with improved patient access. There were general increases in the number of visits per beneficiary for outpatient, nursing home visits, and consultations. But disparities emerged when these data were analyzed for vulnerable populations within the overall Medicare population. Vulnerable populations are defined as beneficiaries who live in rural areas, poor areas, or areas with a shortage of health providers; who are African-American, Medicaid-eligible, disabled, or older than 85 years. These vulnerable populations still were found to face the barriers to care that they faced before MFS was in place. Yet MFS did not create any new barriers. The report concluded that greater understanding of barriers to care for vulnerable Medicare populations is needed.
Physicians' caseloads--the total number of different Medicare patients treated in a year--remained stable or increased during the period studied. Caseload is viewed as an indicator of physician willingness to deliver services to Medicare patients. From 1992 to 1993, the mean caseload for primary care and medical specialties tended to increase more (+4-5 percent) than that for surgical specialties (+2 percent). The trends in the mean-allowed physician charges revealed a somewhat similar pattern. Physicians in primary care and medical specialties tended to experience the greatest increases in caseload, while surgeons, radiologists, and anesthesiologists tended to experience little or no growth. Some groups of surgical specialists--ophthalmologists and otolaryngologists--may have witnessed decreases in mean-allowed charges.
According to a survey analyzed in the report, Medicare beneficiaries reported greater satisfaction with their care in the years after the introduction of MFS than before. They were more satisfied with the quality of care, its availability, the ease of getting to a doctor, and the cost. In addition, fewer beneficiaries reported a health problem for which they did not receive care.
The report also showed dramatic differences in income between White and African-American Medicare beneficiaries. Based on inferences from median household income by zip code, 73 percent of African-American, but only 19 percent of White, beneficiaries were classified in the lowest income quartile. In comparison with Whites, African-Americans experienced higher mortality rates, lower rates of ambulatory visits (see figure II-1), lower use of referral-sensitive procedures, and higher rates of hospitalization. These patterns suggest that African-Americans encountered greater barriers to comprehensive and continuous care.
Figure II-1. Ambulatory Visits per Person by Race and Income: Persons Age 65 Years and Older, 1993
Some of the racial
differences in utilization were attributable to income. For example, the rate
of ambulatory visits declined with income, almost irrespective of race. Whites
in the lowest income quartile not only had far fewer visits than wealthier
Whites, but their rate also was just as low as that for the poorest
African-Americans (see figure II-1). Other analyses revealed that if incomes
for Whites and African-Americans were equal, some racial differences in
utilization would diminish. Race, however, continues to play a role in access
and utilization.
This report is intended to help Congress monitor the impact of legislative changes to physicians' payments under Medicare. It is expected to build on findings presented in previous reports to Congress under the same legislation. Future studies by HCFA are to concentrate on developing new utilization measures. These measures will be used to provide Congress with greater information for policy purposes.
Office of Research and Demonstrations
Marion Gornick
410/786-6696
PIC ID: 5493
Health Care Financing Administration, Baltimore, MD
There are increasing expectations across the Federal Government that agency programs, with a given set of resources and through a series of actions and decisions, will produce the desired outcomes and products for the intended audiences or beneficiaries. This study provides an assessment of the Health Resources and Services Administration's (HRSA) current capacity to respond to such expectations in compliance with the Government Performance and Results Act (GPRA) of 1993. The findings and recommendations in this study address the development of the kind of performance management system needed to respond to the governmentwide search for greater effectiveness.
HRSA commissioned this study to assess its ability to develop and implement a performance measurement and management system in response to the GPRA. Under GPRA, each Federal agency is required to develop comprehensive strategic plans as well as annual performance plans that set specific performance goals for each program activity. Moreover, each agency must report annually on actual performance as it compares with goals. The specific objectives of the study were to provide information about the current status (as of summer 1995) of the development of performance indicators in the four HRSA bureaus and in the Office of Rural Health Policy (ORHP); the adequacy of data and data sources for applying the indicators; and the potential for the agency to use the performance measures in strategic planning or other efforts requiring indicators and data that can be aggregated across its organizational units.
A variety of economic and political forces have increased emphasis on performance in both the public and private sectors. This larger societal phenomenon has produced converging political, budgetary, and management environments that collectively result in a systematic search for effective Federal programs. One of the basic underlying purposes of GPRA is to provide information about the comparative performance of the many Federal programs and agencies competing for budgetary resources, so that available resources may be allocated to those with the best return on the Nation's investment.
Under GPRA, each Federal agency is required to report annually (beginning in March 2000) on actual program performance as it compares with program goals. A preliminary set of performance measures for program and spending is to be developed for fiscal 1997 budget submissions, with actual performance data obtained from fiscal 1995 to fiscal 1997. The intent of GPRA is to motivate Federal agencies to identify programs that are effective in achieving goals and to reallocate resources away from those that are ineffective. GPRA focuses on the quality of performance, quantity of products, timeliness of products, and program efficiency.
The Lewin Group undertook this study for HRSA, organizing the approach around a central question: Can this organization, with a given set of resources, through a series of actions and decisions, produce outputs that have the desired effects and outcomes for the intended audiences or beneficiaries? Preliminary design activities included meetings with HRSA staff, a literature review, and other background research.
An interview protocol was developed and administered to each of the four bureaus and ORHP. Information was solicited about each program's objectives, inputs (e.g., dollars, staff), processes (e.g., training approaches), outputs (e.g., number of people trained), impacts (e.g., number of trainees who would not have found comparable work), outcomes (e.g., number of trainees who find and retain work), and current use of performance measures and indicators. In addition, a wide variety of program materials was collected and reviewed. Draft assessments were developed and shared with staff from the relevant bureaus and offices to ensure that the information was complete and accurate. Findings from the interviews and background research were synthesized, analyzed, and then compiled into assessments of the program clusters and individual programs or budget line items.
The study used an overall assessment approach and analytical framework to document the linkages and commonalities among various HRSA organizational components with respect to resources, processes, products, outcomes, and populations served. Application of a program logic model that illustrates the rationale, structure, and division of labor of each program was critical to the assessments. The program logic model uses a matrix to represent the interactions and relationships among a program's objectives, illustrating the flow of inputs, activities, outputs, outcomes, and impacts required to achieve them.
HRSA's current program structures and measurement efforts form a base for a HRSA-wide measurement system. As part of this system, significant indicators that are common to many programs could be used to measure achievement of agencywide goals. The study identified three principles that should underlie the performance measurement strategy:
The study found that it is both feasible and desirable to use a standard approach to measuring program performance across HRSA programs. In addition, progress in measuring and collecting performance data is quite varied across HRSA programs and program clusters. Within most programs, there is inadequate linkage along the continuum of performance measurement, with a general emphasis on producing outputs rather than evaluating program performance or outcomes. The study also concludes that data and data sources are not treated as strategic HRSA resources and that approaches to specification, collection, and use of data vary widely. Furthermore, the budget does not provide resources specifically for the data collection and evaluation necessary to build and sustain the kind of performance measurement system recommended in the study.
Recommendations are made for both short- and long-term actions. Short-term recommendations include the appointment of a central coordinator responsible for performance management efforts; the establishment of intra-agency working groups to specify performance measures, data collection, support strategies, and involvement of service beneficiaries; the provision of resources; and the provision of technical assistance tailored to specific HRSA components for developing and implementing systems of performance measurement.
Long-term recommendations address the development of a performance management system infrastructure. First, the HRSA strategic planning process should be continued and refined. Second, the performance management process needs to be linked to the budget process, to evaluation, and to grants and contracts management. Third, mechanisms need to be developed to incorporate feedback and data from grantees and populations served by HRSA. Fourth, some further restructuring of HRSA programs will be needed to facilitate performance measurement in light of HRSA-wide strategic priorities. Finally, the commitment and ongoing support of senior management is needed if staff members are to undertake the changes necessary to implement and sustain the performance system.
The study produced a framework for assessing HRSA functions and related performance measures and for how individual program activities and outcomes relate to the Agency's overall strategic goals. Subsequent technical assistance to HRSA on performance indicator development is proceeding, based on needs identified in this initial status report. The logic models developed for the four bureaus and ORHP are serving as an internal guide to the assessment of program inputs, outputs, and outcomes. Assistance to individual bureaus to expand staff capacity in performance measurement began at the end of fiscal 1995. The results of this study have broader applications as well. Many Federal agencies could use the performance measurement framework to assess the linkages between their strategic goals and individual program components, activities, and performance. The identified performance measures, the recommended questions on assessing program effectiveness, and the information on how to develop performance indicators are all valuable tools for any Federal agency developing planning and performance measurement strategies in compliance with GPRA.
Office of Planning, Evaluation, and Legislation
Karen Thiel Raykovich, Ph.D.
301/443-7718
PIC ID: 5954
The Lewin Group, Fairfax, VA
This study evaluated the utility of a model diabetes demonstration program created by the Indian Health Service (IHS) in 1979. Through analysis of 634 medical records, focus groups, and interviews, the model program was found to improve patients' blood sugar control and to reduce their hospitalizations. A critical element of the model program was a team of health professionals who augmented primary care by providing a combination of education, outreach, service delivery, and referrals. Recommendations were made to patients, providers, administrators, and policymakers that would refine and expand this demonstration program.
The purpose of this study was to evaluate a model diabetes demonstration program at two of the five sites at which the program was launched by IHS in 1979. The hallmark of the program is a team of three to five midlevel health professionals, including a physician's assistant or nurse practitioner, nurse, and dietitian/nutritionist. The team educated patients in preventive care, referred patients to specialists when needed, coordinated services, performed community screening, and maintained patient registries. The study measured two key patient outcomes--blood sugar control and hospitalizations--in relation to a control group of patients who received usual IHS diabetes services.
The Indian Health Care Improvement Act (Public Law 94-437) of 1992 mandated an evaluation of a model diabetes demonstration program established by IHS in 1979. Since its inception, this model program of intensive primary and preventive care has been extended to at least one site within each of the 12 regional areas of IHS. The impetus for the formation of the model program was to improve patient management and to reduce hospitalizations for a costly condition that disproportionately affects American Indians.
The prevalence of diabetes in this population (6.9 percent) is almost three times higher than that in the U.S. population as a whole (2.4 percent). American Indians also suffer from disproportionately higher rates of diabetic complications. Complications include loss of vision or blindness, lower extremity amputations, and end-stage renal disease. Mortality rates among diabetic American Indians also are higher compared with those of other ethnic groups. Moreover, diabetics are intensive consumers of medical services, and the costs per patient are among the highest of any medical condition.
The underlying philosophy behind the creation of the model program was that a team of primary care professionals could effectively deliver patient education, outreach, and treatment. Patient education was designed to address diet, exercise, foot care, administration of medications, and monitoring of blood sugar. Equipped with more knowledge and skills, patients were thought to be in a better position to control their blood sugar levels. Achieving better control over blood sugar levels was expected to lower the rate of diabetic complications and other conditions that lead to hospitalization.
The study methodology consisted of quantitative and qualitative components. Abstracted medical records from a total of 634 patients who entered treatment between 1983 and 1992 were analyzed. Information from medical records was supplemented by data on inpatient and outpatient contacts from IHS health services utilization databases. Additional analyses were conducted to determine the generalizability of findings to other IHS sites. Using 1993 IHS audit data, patient demographics, severity of illness, and extent of services at model sites were compared with similar data at other model sites. The analyses also compared the control site with other IHS sites that provided usual treatment.
Focus groups with patients and interviews with IHS and tribal staff supplemented quantitative analyses. During four focus group meetings--two at model sites and two at control sites--patients were asked about problems they encountered in complying with medical recommendations and their views of the services they received. Interviews with medical personnel yielded information about the content and the perceived quality of diabetes services.
The study found the model program to be successful at controlling patients' blood sugar levels and at reducing their hospitalizations. Fewer patients (11-35 percent) at model versus control sites experienced poor blood sugar control, as defined by fasting levels of more than 220 milligrams per deciliter of blood or by random levels of more than 250 milligrams per deciliter of blood. Patients at model sites were at lower risk of being hospitalized, operationally defined as the time from diagnosis to the first diabetes-related hospitalization.
The study identified several elements of the program that were associated with good blood sugar control: consultations with a dietitian and a podiatrist in the first year after diagnosis and receipt of a hemoglobin A1c test one or more times in 2 years. This test for blood sugar levels is more reliable than others because it reflects average levels over the previous 3- to 4-month period. Similar programmatic elements helped prevent hospitalizations. As to the generalizability of all study findings to other IHS sites, the analysis of audit data could not provide a definitive answer because of undersampling at the control site.
Observations from patients and providers at focus groups and interviews, respectively, offered important insights about the model program. Patients and providers both claimed changes in diet to be the most difficult for patients to implement. Patients stressed the importance of good communication with their providers, and those receiving the usual care reported more communication problems. This finding was considered noteworthy because earlier published research revealed that the quality of the patient-physician relationship is predictive of diabetic patients' compliance with lifestyle changes and medication dosing.
The study offered a battery of recommendations to refine the model program and to enhance care at IHS sites that do not have the program. The study is being prepared for publication. Knowledgeable clinical staff who are current with diabetes management were considered vital to the delivery of primary care to diabetes patients. Those IHS sites that do not have the model program can greatly benefit from creating one new staff position that is devoted to diabetes education, coordination, and outreach.
The finding that patient education in the first year after diagnosis had important and long-lasting impacts led to a recommendation for early patient education that focused on difficult lifestyle changes, particularly dietary changes. Blood sugar evaluation by both standard tests and by the hemoglobin A1c test were requirements for good clinical management and were correlated with better sugar control and lower hospitalizations. Only a small percentage of patients at model sites, and even fewer at the control site, received both tests once a year.
The study recommended more time for direct patient contact with providers even in lieu of some educational services. A strong bond between patients and providers was viewed as essential for improving patient compliance. Finally, the study recommended training for providers in counseling patients who are especially stressed by the diagnosis of diabetes.
Office of Planning, Evaluation, and Legislation
Frank E. Marion
PIC ID: 6005
Native American Consultants, Inc., Washington, DC
A committee of experts was convened by the National Academy of Sciences, Institute of Medicine, to evaluate published studies on fetal alcohol syndrome (FAS) and related disorders. The committee offered new diagnostic categories to resolve confusion and thus improve diagnosis. Through recommendations for policy, research, active and passive surveillance, and enhanced prevention and treatment, the committee sought to intensify the public health response to a completely preventable group of disorders with serious developmental consequences. Recognizing that the problems of FAS and related disorders cross medical disciplines, the committee recommends establishing an interagency task force, led by the National Institute on Alcohol Abuse and Alcoholism (NIAAA), to coordinate national efforts.
The FAS committee was responsible for enhancing the understanding of available research and clinical knowledge by reviewing and evaluating studies on FAS and related disorders with respect to diagnosis; prevalence; and effectiveness of surveillance, prevention, and treatment programs. Because of uncertainty on fetal outcome associated with moderate or low levels of maternal alcohol consumption, the committee restricted its analysis to the effects on the fetus of maternal consumption of large amounts of alcohol.
As part of the Alcohol, Drug Abuse, and Mental Health Administration Reorganization Act (Public Law 102-32 1), Congress mandated a study by the National Academy of Sciences' Institute of Medicine of FAS and related disorders that result from heavy alcohol use during pregnancy. FAS is a serious disorder with physical and mental deficiencies that are costly to treat and rehabilitate and that often require long-term care. The costs of FAS and related conditions can be quite high for the individual, for the family, and for society. Alcohol's effects on the fetus are considered more severe than cocaine, heroin, and marijuana, yet this serious public health condition is rarely diagnosed and treated.
FAS and related disorders are characterized by a constellation of physical malformations, growth deficiencies, and neurodevelopmental abnormalities. The most obvious malformations are found on the face, often in the form of a flat upper lip and midface, but other malformations can be present in other parts of the body. Neuro-developmental defects include impairments in intelligence, motor skills, gait, hearing, and eye-hand coordination. Although not all FAS victims experience mental retardation, FAS is considered the most common totally preventable cause of mental retardation. Variability in the expression and permanence of these defects is dependent on numerous factors, including the amount and pattern of maternal drinking, the timing of drinking during pregnancy, and a host of biological and environmental factors before and after birth.
The incidence of FAS and related disorders appears to be 0.5 to 3 cases per 1,000 births, although the data are far from ideal, in part because of inaccuracies in diagnosis. While up to 14 percent of women drink heavily when they drink, only a small percentage of women, about 4 percent, are dependent on alcohol according to accepted clinical criteria. The rates of heavy drinking during pregnancy vary, but most studies find that less than 1 percent of pregnant women drink heavily. Although these percentages appear small in relative terms, the absolute number of women engaging in hazardous drinking is large. The annual costs of FAS and related disorders are estimated at $75 million to $9.7 billion. The great range of costs reflects uncertainty about the long-term consequences, such as the need for medical and educational interventions, as well as uncertainty about the actual incidence of FAS and related disorders.
The 14-member committee of experts reviewed and evaluated published studies on FAS and related disorders. Committee expertise included pediatrics, developmental psychology and neurology, obstetrics, nosology, teratology, epidemiology, psychiatry, and substance abuse prevention and treatment.
The committee formulated new diagnostic criteria for FAS and related disorders. The purpose of the new criteria was to capture the range of defects, both obvious and subtle, and to improve clarity, rigor, and consistency of diagnosis for clinicians and researchers. Symptoms were presented in five major diagnostic categories, with three categories describing partial characteristics of full FAS. The two categories for subtler related disorders, collectively referred to as "alcohol-related effects," are for either physical malformations or neurodevelopmental abnormalities. Alcohol-related effects are thought to occur more frequently than full-blown FAS.
Passive and active surveillance measures were compared. With passive surveillance, data are collected retrospectively through reviews of medical charts, although more expensive active surveillance in which data collection takes place prospectively through experimental protocols could identify more cases and provide superior data. The committee recognized that passive surveillance measures could be heightened but were inadequate to capture the incidences of FAS and related disorders.
Identified risk factors for maternal drinking were a major research gap. Understanding risk factors that predict drinking behavior during pregnancy was deemed vital for developing targeted prevention programs. The committee concluded that more study was needed of the psychological and social-environmental factors that lead to heavy drinking during pregnancy, including dependence on alcohol, depression, low self-esteem, family history of alcoholism, sexual abuse, other violent victimization, and dietary habits.
The committee constructed a conceptual framework for prevention that takes into account cultural, sociological, behavioral, public health, and medical disciplines. This new framework was seen as an important guide to the field, given the scarcity of prevention-related research on FAS and related disorders. The proposed prevention model is a continuum of interventions designed for targeting the general risks of all women who drink, to the specific risks incurred by some women who drink heavily during pregnancy. The committee endorsed the universal prevention message to all women that alcohol consumption should be avoided during pregnancy, although it noted a paucity of studies to determine the behavioral impact of this message.
The committee stressed that an important aspect of preventing FAS is treating the alcohol-dependent woman. There is a compelling need for studies that assess the effectiveness of specialized programs that treat pregnant women who are alcohol dependent. The committee recommended a high priority be placed on efforts to design, implement, and evaluate prevention programs that effectively guide pregnant women who drink heavily into alcohol treatment. The committee further recommended continued research on biological methods of detecting alcohol consumption by pregnant women and continued basic research in animals to elucidate mechanisms of alcohol-induced fetal abnormalities so that medications can be developed that ease the symptoms of FAS.
The availability of treatment for individuals affected by FAS and related disorders was viewed by the committee as inadequate. The committee was unable to identify specific treatment programs. Because studies have clearly shown the value of early intervention in ameliorating symptoms of other neurodevelopmental disorders, the committee stressed the importance of identifying children with FAS and related disorders as early as possible. The committee also concluded that there was inadequate and insufficient attention given to preventing secondary disabilities such as deteriorating mental health, criminal behavior, and substance abuse.
The committee recommended research to assess the utility, reliability, and validity of its new diagnostic criteria for FAS and related disorders. Other research recommendations included active surveillance through national surveys that are repeated at periodic intervals; identification of psychological and social-environmental risk factors to guide the development of more targeted prevention programs; and studies to determine the effectiveness of prevention and treatment programs geared toward pregnant women, families, and individuals with FAS and related disorders.
In terms of major policy recommendations, the committee embraced the Surgeon General's 1981 position of total abstinence from alcohol prior to conception and throughout pregnancy and recommended the establishment of clinical practice guidelines for treatment of children with FAS and related disorders. While recognizing that NIAAA has historically led research efforts on FAS, the committee noted that NIAAA is a small institute and FAS is a complicated and diverse problem. In the absence of a single organized discipline of health professionals with responsibility for FAS prevention and treatment, the committee recommended forming an interagency task force to coordinate national efforts. The task force, which would be spearheaded by NIAAA, would include appropriate Federal research, surveillance, and service agencies.
National Institute on Alcohol Abuse and Alcoholism
Lura Abbott, M.S.
301/443-2369
PIC ID: 6006
National Academy of Sciences, Institute of Medicine, Washington, DC
In this study, a contractor for the Office of Research Integrity (ORI) contacted and interviewed by mail individuals who had made allegations of scientific misconduct to ORI. In this first examination of its kind, ORI sought to better understand the impacts of whistleblowing and develop an empirical base by which to examine the consequences of whistleblowing on whistleblowers. More than two-thirds of the whistleblowers surveyed reported experiencing at least one negative outcome as a result of their action, with the most serious negative consequences being actions taken by institutional officials. Adverse consequences were even more likely to be experienced if the alleged misconduct developed into a high-profile case that gained notoriety outside the institution.
Data were collected on whistleblowers from closed cases to provide an empirical base to consider the consequences of whistleblowing. These data were used to inform the Commission on Research Integrity and ORI staff as they sought to deal seriously and effectively with whistleblowers' allegation of misconduct and to protect them from retaliation or negative consequences resulting from their action. Data were collected on the types of actions experienced by whistleblowers before and after their allegation and the effects these actions had on their personal and professional lives. In addition, more detailed information was gathered on the particular circumstances of the allegation (i.e., the relationship of the accused, the type of allegation, the outcome of the allegation, and the amount of publicity the allegation received).
Uncovering misconduct in science often depends on the willingness of those who are aware of or suspect the misconduct to report it. Before reporting misconduct, potential whistleblowers must consider whether administrators will take the allegation seriously, treat it confidentially, and protect them from retaliation.
Current Federal Regulations (42 CFR 50 103) require that institutions develop policies and procedures to handle allegations of misconduct. These policies must include provisions for "undertaking diligent efforts to protect the positions and reputations of those persons who, in good faith, make allegations" (42 CFR 50 103 [D][13]). If there is an immediate need to protect the interests of the person(s) making the allegations, Federal policy requires institutions to notify HHS.
The National Institutes of Health Revitalization Act of 1993 created the Commission on Research Integrity, which was established in March 1994 to make recommendations to the Secretary of HHS and Congress on how the Public Health Service (PHS) should deal with research misconduct in research funded by the PHS Act. In an interim report released in January 1995, the Commission identified "retaliation against whistleblowers" as one of three problem areas affecting the responsible conduct of scientific research.
Congressional hearings revealed that some whistleblowers may have suffered retaliation by the accused or others in the institution, which led to mandated additional regulations to protect whistleblowers. Despite legislative and executive efforts, however, empirical evidence and anecdotal reports continue to suggest that some students and faculty engaged in scientific research are, or feel they are, vulnerable to retaliation if they report misconduct.
The Research Triangle Institute conducted the study for ORI. Individuals listed in ORI files (closed cases only) as having made allegations of scientific misconduct were identified. There were two phases of data collection. In the first phase, whistleblowers were contacted by mail; and a followup telephone call was made to create an up-to-date mailing list, which resulted in current addresses for 105 of the 127 individuals (82 percent) listed in the files.
A second-phase questionnaire was mailed to individuals, and a followup telephone call was made to remind them to return the completed survey. Eighty-nine individuals responded to the questionnaire; of these, 68 whistleblowers were considered eligible and willing to participate. Ninety-one percent of respondents reported that they held doctoral degrees; 78 percent worked in an academic setting; 9 percent were postdoctoral or graduate students.
Whistleblowers in scientific misconduct cases are highly likely to experience one or more negative consequences as a result of their whistleblowing, but most perceive these consequences as having had a neutral impact on their careers, professional activities, and personal lives. Sixty-nine percent of whistleblowers reported experiencing at least one negative outcome. Twenty-five percent reported serious consequences such as loss of position or denial of tenure, promotions, or salary increases. Other negative consequences included reduction in research support or travel funds, counterallegations, delays in reviewing manuscripts or processing grant applications, and ostracism.
The majority of the negative consequences experienced by whistleblowers were due, they said, to the actions of institution officials, respondents, colleagues, and professional societies; the most serious consequences felt by whistleblowers were attributed to the actions of institutional officials while the institution was responding to their allegations (i.e., while the case was still open) and after the inquiry or investigation was completed. Consequences were most likely to involve pressure on the whistleblower to drop the allegations of misconduct.
Data revealed that whistleblowing was most likely to have adverse outcomes in situations in which fabrication of data was alleged, the case received publicity, the allegations were made to a senior administrative official or misconduct official of the institution or to the funding agency, the allegations were made both within and outside the institution, the allegations were made to many different types of individuals, and the allegations were subjected to an investigation without recourse to an initial inquiry.
About 62 percent of whistleblowers perceived their whistleblowing to have had a neutral impact on their careers, professional activities, and personal lives; 28 percent perceived a negative impact; and 10 percent reported a mixed impact. Although few whistleblowers perceived positive consequences of their actions, 68 percent reported a willingness to make another allegation; 12 percent said they probably would make another allegation; 10 percent were uncertain; and 10 percent said they would not.
Not every whistleblower suffers substantial negative consequences as a result of reporting misconduct, but most individuals who report allegations of misconduct frequently face the prospect of significant hardship because of their efforts.
The study results will be used by ORI to develop the mandated regulation on the protection of whistleblowers, educate institutional officials and scientists about current abuses, and create a system of monitoring the treatment of whistleblowers in scientific misconduct cases.
These data suggest that ORI focus first on limiting adverse actions while the case is still active. To prevent the most serious consequences of whistleblowing, regulations and enforcement approaches will need to primarily target institutional officials. Finally, potential whistleblowers should be counseled about the likely harm they will suffer if they make their case a cause celebre by taking their concerns outside their institution or getting their case publicized by the media.
Office of Research Integrity
Lawrence J. Rhoades, Ph.D.
301/443-5300
PIC ID: 5659
NTIS Accession Number: PB 96-200449
Research Triangle Institute, Research Triangle Park, NC