Advanced Models and Methods, Part 2

Presenters: Christopher Forrest, M.D., M.P.H. Assistant Professor, Johns Hopkins School of Hygiene and Public Health Lawrence C. Kleinman, M.D., M.H.A. Chief of Health Studies, Lehigh Valley Hospital Andrea K. Biddle, M.P.H., Ph.D. Associate Professor, University of North Carolina at Chapel Hill Uma Kotagal, M.B., B.S., M.S.C. Director, Health Policy and Clinical Effectiveness, Cincinnati Children's Hospital Euni Lee, Pharm.D., Ph.D. Center on Drugs and Public Policy Maryam Navaie-Waliser, Dr.P.H. Research Associate, Center for Home Care Policy and Research, Visiting Nurse Service of New York Ilene Zuckerman, Pharm.D., BCPS Associate Professor, Center on Drugs and Public Policy, University of Maryland, Baltimore

Contents

Introduction
Assessment of Pharmacologic Management of Asthma in Children Using Medicaid Databases
Methodological Issues in Studying the Impact and Cost-Effectiveness of Care Coordination Services
Assessing Guideline Use Linking Outcomes to Performance
Conceptual Framework for HSR and the Uncertainty Index

Introduction

Dr. Lawrence C. Kleinman introduced the objectives of the session:

• To use current research to illustrate the application of advanced methodological approaches to answer important questions about children's health services.
• To discuss a conceptual framework that provides context for these methods in child health services research.

In achieving these objectives, it is important to begin by clarifying definitions of assessment, program evaluation, and key factors such as quality, structural/system components, process of care, and outcomes:

• An assessment is a measurement or series of measurements placed into a conceptual context. Therefore, a useful assessment answers the questions that you care about in a way that provides you value in your context and is the "reliable, valid and meaningful characterization of something you care about."
• With respect to evaluation there are different perspectives to consider. With respect to public policy/public health perspective, one can ask, "How well does this program compare to all possible interventions?" Alternatively one can evaluate a program with reference to itself, i.e., "How well does this program do compared to its potential as defined in its conceptual model?"
• Dr. Kleinman reminded participants of Donabedian's constituents of quality: structure/system of care, process, and outcomes. Structural and systems characteristics included: who delivers care, facilities involved, office and clinical systems, quality improvement systems, financial mechanisms and incentives, and the organization of health care delivery. The process of care asks the questions, "What services are delivered to whom, and how well are they delivered?"
• Finally, the variety of outcomes includes clinical outcomes (morbidity, functional status, wellness, mortality, and patient satisfaction), as well as financial outcomes.

Dr. Kleinman's orientation was followed by three presentations on CHSR research that emphasized the process and steps of research as well as the importance of different research contexts. The various research settings included:

• A Medicaid drug utilization review program in Pennsylvania.
• A Medicaid care coordination program in North Carolina.
• A research and care improvement collaborative involving eight hospitals.

Return to Contents

Assessment of Pharmacologic Management of Asthma in Children Using Medicaid Databases

Child Health Services Research can take advantage of public databases such as medical claims data, and systems such as Medicaid drug utilization review. A study presented by Euni Lee, Ilene Zuckerman and Diane McNally described how they used these resources to look at adherence to guidelines of the National Asthma Education and Prevention Program (NAEPP, 1997) with respect to the use of a quick relief medication.

The purpose of the study was to evaluate the therapeutic appropriateness of asthma drug use among Pennsylvania fee-for-service enrollees. The researchers outlined the following research steps:

1. Select topic and goal.
2. Develop criteria based on the guidelines.
3. Validate the criteria.
4. Collect data.
5. Perform an analysis.
6. Provide necessary intervention.
7. Perform a re-assessment.

The study's topic focus was asthma management with short-acting beta 2-agonist in Medicaid children. Asthma is the most common childhood chronic disease; 4.8 million children were affected in 1996 and there have been increasing rates of mortality and hospitalization.

Criteria for evaluating prescribing patterns were developed from the NAEPP guidelines: the drug of choice for quick relief medication is SA inhaled beta-2 agonist and long-acting (LA) inhaled beta 2-agonists should not be used to treat acute exacerbations. Over-reliance of SA beta 2-agonists and poor asthma management was defined as using greater than 1 MDI canister per month and more than or equal to 2 MDI canisters within 25 days at least twice.

The database included all pharmacy claims of any asthma drug from prescription records for children in Medicaid with asthma medications between July 1, 1998 and January 31, 1999. Data collection required substantial editing and validation of key elements of the Medicaid claims data. For example, algorithms needed to be developed to eliminate duplicate claims; data entry errors or missing information such as dual eligibility (Medicare and Medicaid) status needed to be determined based on other information.

The study identified 14,292 children as asthma drug users during the 7-month period. Of these, 392 children were identified as SA beta 2-agonist inhaler overusers, 42 children received non-recommended SA beta 2-agonist inhalers, and 37 children used salmeterol as a LA beta2 inhaler without SA beta2-agonist inhalers. This information was used to assist the Pennsylvania Department of Public Welfare as groundwork for educational interventions with physicians and pharmacies.

The advantage of Medicaid data use includes:

• Size of the database.
• Time and cost of generating data.
• Lack of recall bias that would be associated with patient/provider survey data.

At the same time, some potential limitations were recognized, including information about patient utilization and patient history which could demonstrate contraindications to specific medications for individual patients.

Return to Contents

Methodological Issues in Studying the Impact and Cost-Effectiveness of Care Coordination Services

Estimating the impact and cost-effectiveness of North Carolina's care coordination program relative to infant survival through the first year of life was the objective of a study presented by Drs. Maryam Navaie-Waliser, Andrea Biddle, and Sandra Martin. Their presentation focused on methodology and the 12 key steps involved in the study, listed below:

• Define the audience. The critical audiences for this study were state policymakers, program planners as well as the scientific community.
• Define the problem and study objective. While enhanced Medicaid services, including care coordination are offered to combat infant mortality, to date, few studies have examined care coordination's effect on infant mortality and none have examined its cost-effectiveness relative to infant survival. The objective of this study was to estimate the impact and cost-effectiveness of North Carolina's program relative to infant survival through the first year of life.
• Identify the prevention strategy and its comparator. There were 23,235 individuals that received Medicaid plus care coordination, and 19,066 who received "Medicaid only."
• Specify the analytical perspective. In this case, the point of view represented was State government, within which decisions were being considered about whether to expand or reduce the program.
• Define the relevant time frame and analytic horizon: in this case, 1994 calendar year data, reported in 1998 U.S. dollars.
• Determine the analytic approach. Three approaches were considered, including cost-benefit analysis (CBA), cost-effectiveness analysis (CEA) and cost utility analysis (CUA). Cost-effectiveness was selected. CBA was rejected as inappropriate when looking at infant survival, and CUA was rejected because there were no good quality adjusters to use with the administrative data.
• Determine if analysis is to be marginal or incremental. Marginal cost-effectiveness would be the effect of additional investment in the program, and incremental cost effectiveness would look at the relationship between making an investment in a different program and the health outcomes expected to be produced by that strategy. For the project, both marginal and incremental cost-effectiveness ratios (ICERs) were used.
• Identify relevant costs and health outcomes. Medical costs included inpatient, outpatient physician and drug charges. Program costs included direct and indirect costs of services provided from pregnancy to 60 days postpartum, and for eligible infants from 60 days to one year. Costs were adjusted for age, race, marital status, education, medical risk, previous infant death and tobacco use. Health outcomes were essentially infant survival through the first year of life.
• Specify a discount rate. None were used for this study.
• Identify sources of uncertainty and plan sensitivity analyses. One example of this was the variation in the number of care coordination visits actually received by participants.
• Determine the summary measure for results dissemination. Compared to participants receiving no care coordination, survival was increased among infants weighing less than 1,500g for care coordination participants. The incremental cost-effectiveness ratios revealed that the cost for each additional life saved through the supplemental provision of care coordination was $62,480 for infants weighing less than 1,500g and $19,498 for infants weighing 1,500g to 2,499g.
• Identify distributional effects. For State policymakers, program planners and the scientific community, this study showed that North Carolina's program appeared to improve infant survival, most notably among the highest risk infants. The study suggests that program planners should prioritize the delivery of care coordination and services to families with multiple risk factors for having very low birth weight infants. Finally, the authors note that death may not be the most appropriate outcome to evaluate, although it is often the most important to legislators and their constituencies.

The presenters concluded by emphasizing that the benefits of economic analyses of intervention programs include the potential for improved targeting of programs to populations likely to receive the greatest benefit.

Return to Contents

Assessing Guideline Use Linking Outcomes to Performance

Research often takes place in the context of delivery systems that offer special opportunities for research but also present particular challenges. Dr. Uma Kotagal presented an assessment of compliance with evidence-based guidelines among eight hospitals participating in the Child Health Accountability Initiative (CHAI).

CHAI was developed to design, evaluate and implement collaborative projects and research to promote clinical practices that lead to improved health outcomes for children. This hospital network provided a basis for research collaboration and opportunities to achieve an adequate sample size. At the same time, it presented challenges related to satisfying institutional (as well as research) objectives, and also coordinating research across sites.

The study looked at process and outcome measures associated with implementation, during the winter of 1999, of a guideline for Bronchiolitis at the eight CHAI hospitals. Process measures that were looked at included:

• Implementation methods (e.g. presence of physician order entry).
• The role of respiratory services.
• Resource use.

Data related to these measures was gathered via:

• Administrative databases (length of stay, charges).
• Chart review (frequency of bronchodilators, respiratory score use, and discharge practices).
• Analysis of administrative data and chart agreement.

Outcome measures included:

• Resource use.
• Satisfaction with care.
• Adverse outcomes.
• Recovery from illness.

The latter involved looking at such things as lingering symptoms, economic impact and child care use. A telephone survey of parents was employed to gather outcomes-related data.

The study found that the NAEPP guideline was followed in 79 percent of cases, although compliance was not associated with illness duration, or post-discharge burden or resource use. Satisfaction with care was associated with resolution of symptoms. Finally, there was a significant economic impact of the acute illness on families involved. A third of the families took significant time off, in part because lingering symptoms were a significant factor in the study sample.

With respect to the use of the hospital network as the basis for research, the study found that demonstrating an effect, or no effect, may require multiple sites even for common illnesses. Also, measuring process and outcomes requires multiple data sources, and studying practice in situ provides opportunities for linking process indicators to outcomes.

Return to Contents

Conceptual Framework for HSR and the Uncertainty Index

In concluding the session, Dr. Kleinman reiterated the importance of establishing a conceptual framework for health services research, and also for translating practice into research (or TRIP as referred to by AHRQ). He also identified the need to think of ways to evaluate where research is most needed, and where research dollars should be devoted.

In this regard he discussed the Uncertainty Index, or UNI, which indicates the proportion of non-refuted current practice that is also not corroborated by the research base. UNI will be higher for less-researched areas, and lower for well-researched areas. This allows in some respects flexibility for standards of evidence, with lower standards for evidence of areas with high UNI, and higher standards for evidence of areas with low UNI. UNI can be used to address the fact that there is a disproportionate amount of research devoted to adults as compared with children in the United States.

Return to Contents

Current as of June 2000

---

Internet Citation:

Advanced Models and Methods, Part 2. Role of Partnerships: Second Annual Meeting of Child Health Services Researchers. June 27, 2000. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/chsr2mm2.htm

---

Return to Overview
Proceed to Next Section