In this comprehensive analysis of outcomes and effectiveness research at the Agency for Health Care Policy and Research, the Lewin Group describes accomplishments and lessons of the past decade and recommends how to increase the measurable impact of that research.
The full text of this report (AHCPR Publication No. 99-R044) is available online; you may order a print copy from the AHCPR Publications Clearinghouse. Call toll-free 800-358-9295.
Intended for use in ongoing discussions of strategic directions for the outcomes research program of the Agency for Health Care Policy and Research (AHCPR), this report covers three main topics:
The primary questions examined by this analysis are, "How can the outcomes and effectiveness program at AHCPR most effectively advance the field of health services research (HSR), fulfill its unique role in HSR, contribute to public health, and address the expectations of policymakers and stakeholders?" "The Outcome of Outcomes Research at AHCPR" is a first step to redefining the goals of OER, as well as an honest appraisal of prior successes and opportunities for improvement.
Work on geographic variations in medical practice, appropriateness of care, and the poor quality of medical evidence set the stage for the "Effectiveness Initiative" announced in 1988. The primary responsibility for carrying this initiative forward found an institutional home when AHCPR was established in 1989. The effectiveness initiative itself represented an important hypothesis: guidance for optimal medical practice could be gleaned from analysis of data routinely gathered in the process of delivering and paying for patient care. The output from AHCPR's program for outcomes research over the past decade offers some empirical evidence with which to assess its validity.
OER evaluates the impact of health care (including discrete interventions such as particular drugs, medical devices, and procedures as well as broader programmatic or system interventions) on the health outcomes of patients and populations. Part of the challenge for AHCPR is the fact that the nature of OER is complex, and often apparently resistant to easy translation for policymakers and clinicians. Despite this complexity, expectations continue to be very high that research done by the Agency will have a clear, measurable impact on health care quality and costs.
A framework was developed that outlines an idealized process by which basic findings in OER are linked over time to increasingly concrete impacts on the health of patients. The four levels of impact are:
Based on the premise that the researchers should have a clear understanding of the most important findings, a survey was mailed to all principal investigators (PIs) funded by AHCPR's Center for Outcomes and Effectiveness Research (COER) between 1989 and 1997, asking them to describe their most important work. Of the 95 PIs contacted, responses were received from 61 (64 percent). Survey results suggest that PIs have been most successful in providing increasingly accurate and detailed descriptions of what actually occurs in health care, developing tools for measuring costs of care and patient-reported outcomes, and identifying topics for future research. Few PIs reported findings that provide definitive information about the relative superiority of one treatment strategy over another. There are also few examples of findings that have been incorporated into policy (level 2 impacts) or clinical decisions (level 3), or interventions that have measurably improved quality or decreased costs of care (level 4). One of the main challenges for the next generation of outcomes studies is to move from description and methods development to problem solving and quality improvement.
The field of OER has clearly progressed over the past decade. At least three conceptual developments have been strongly influenced by AHCPR-sponsored work:
OER studies have often provided descriptive data that challenged prevailing clinical ideas about how best to manage specific clinical problems. Documenting patterns of care or outcomes that are inconsistent with existing understandings of disease processes and management creates a tension that can lead to further study or directly to changes in practice. The most concrete accomplishments of the 10 to 15 years of OER are the tools and analytic methods that have been developed. These include strategies for conducting systematic reviews and meta-analysis, instruments for measuring health outcomes important to patients, and sophisticated techniques for analyzing observational data to adjust for disease severity and minimize bias. Some findings from OER may not be definitive enough to influence practice, but may help direct future research efforts. Public funding for OER has produced a network of institutions and trained investigators capable of carrying out OER. Beginning with a relatively small core of academic centers and professionals, there are now large numbers of individuals and institutions pursuing outcomes studies in the public and private sectors.
The past decade has also been a learning experience for the field of OER. In its simplest form, the framers of the effectiveness initiative realized that existing data and studies might represent an inexpensive source of knowledge about effective care. The debate over the pros and cons of randomized controlled trials (RCTs) and observational studies partially obscures a basic observation that is less controversial. Different research designs are associated with different susceptibility to systematic bias. The two critical questions to ask when considering the adequacy of a particular study design then are: "How likely is it that bias is affecting the results, and how certain of the results is it necessary to be in order to change policy or practice?"
It will be valuable to do a more systematic exploration of how to associate the features of a particular clinical problem with the most appropriate tools and methods to study that problem (given that the goal is to promote decisions that will improve outcomes of care).
It is increasingly clear that limitations of observational designs prevent these studies from providing definitive answers to many questions of comparative clinical effectiveness. More experience has also confirmed that changing practice and realizing savings in health care are not easily achieved. Research and experience have demonstrated that development and dissemination of even high quality, highly credible information is often insufficient to alter practices. Enhanced knowledge must be linked with supportive practice environments and incentives for change. And while it had been hoped that reduction of inappropriate, high-cost care would represent savings of hundreds of billions of dollars through outcomes studies, actual success has been more modest.
AHCPR can take steps to maintain its strength in methods and tools development while increasing support for studies with greater potential for impact. The Agency could also play a more active role in the transfer of knowledge and documenting change when it occurs. Many studies supported by COER in the past decade have served as hypothesis-generating studies, but there has not been a systematic effort directed to orchestrate the hypothesis-testing follow-on. COER could take on greater responsibility to make sure that once these critical knowledge gaps are identified, they are addressed in follow-on studies. While unchanneled intellectual exploration is a time-tested and vital approach to research, it may not be entirely satisfactory for AHCPR, given the expectations for accelerating the pace of translating research into improved patient outcomes.
COER should also leverage resources by seeking new partnerships in addition to maintaining collaborative efforts with the Health Care Financing Administration (HCFA) and other payers, managed care organizations, medical groups, medical professional organizations, peer review organizations, and medical product manufacturers. Collaboration with these organizations would ensure that potential studies are crafted to meet the applied needs of these organizations. This next stage of OER would increase attention to creating designs and blueprints for improving care and selectively applying the OER tools when they are needed. It would represent a move from a "tool-using" culture to a "problem solving" culture, for which the primary objective is developing an empirical science of clinical improvement. The COER could be refashioned into the "CHOIR" (the Center for Health Outcomes Improvement Research).
There is a need for more attention to developing innovative methods and strategies for efficiently addressing the large number of unanswered questions about effectiveness and cost-effectiveness that incorporate relevant environmental characteristics. There has been no concerted effort to craft a new methodological, organizational, and ethical framework for these studies. The "conceptual infrastructure" for conducting effectiveness trials needs further development. COER could engage in a major initiative to create and refine new approaches to conducting prospective effectiveness trials, similar in visibility and scope to the "effectiveness initiative" of the late 1980s. The creative energy and ingenuity of health services research would be directed to crafting new approaches to studying effectiveness across the spectrum of health care settings using prospective, experimental methods. As with AHCPR's Patient Outcomes Research Teams (PORTs), the new effectiveness initiative would support methods development, enhance specific technical skill in the research community, promote conceptual discussion of effectiveness trials, and create networks of researchers closely linked with institutions and practitioners. Prospective effectiveness studies can best be done as partnerships between researchers, providers of clinical care, delivery systems, and funding organizations. The operational and organizational challenges to this are quite significant.
Improvements in outcomes measures and development of strategies to encourage their routine use are an essential future direction for COER. Additional work on the incorporation into practice, feasibility and credibility of disease-specific and general outcomes measures, and development of tools and strategies to make this possible, are critical requisites to getting to outcomes that people experience and care about.
The Agency should consider developing the capacity to identify important research findings (generally level 1 impacts), and to move findings to actions. This capacity should be proactive, intensive, coordinated, and consistent, and not left solely to the ingenuity of each research team. A dedicated knowledge transfer function would systematically consider findings from OER studies and carefully assess them to determine how the findings can have more impact. An internal or external center would develop a strategy for collecting and documenting examples of OER impact stories. In order for COER to be seen as accomplishing its goals, those goals must be agreed upon and clearly stated. It is also important to clearly convey the complementary nature of OER studies with traditional basic science and clinical research. Policymakers need to have a firm understanding that OER provides the link between basic research and clinical practice.
A high level of interaction with stakeholders in the health care system will ensure that the basic studies reflect real problems faced by those involved in health care delivery. Ideally, managed care organization (MCO) managers and medical directors, hospital administrators, insurance company business managers, medical product industry marketing directors, pharmaceutical product managers, private sector outcomes researchers, and others could directly participate in a range of COER activities.
The effectiveness initiative, launched a decade ago, has yielded both important findings and opportunities for improvement. A new emphasis on evidence-based decisionmaking and interest in assessing and improving quality of care, along with interest in OER among a broad array of private sector organizations, set the stage for this analysis. The attempt to systematically review research output from an organization has few clear precedents, and necessarily reflects the limited perspective of the reviewers themselves, as well as the difficulty of determining the value of generated knowledge. A key conclusion from this analysis is that the accomplishments of the first decade provide a foundation to build on. However, it is clear that interests and incentives of the research enterprise and the "change agents" within the health care system must be more effectively aligned to achieve the promise envisioned by the establishment of AHCPR.
Current as of September 1999
Internet Citation:
The Outcome of Outcomes Research at AHCPR:Final Report. Summary. Agency for Health Care Policy and Research, Rockville, MD. http://www.ahrq.gov/clinic/outcosum.htm
Return to Outcomes & Effectiveness
Clinical Information
AHRQ Home Page
Department of Health and Human Services