"What we really want to get at is not how many reports have been done, but how many people's lives are being bettered by what has been accomplished. In other words, is it being used, is it being followed, is it actually being given to patients? [W]hat effect is it having on people?" —Congressman John Porter, Chairman, House Appropriations Subcommittee on Labor, HHS, and Education, 1998 |
The purpose of research is to produce information that can inform decisionmaking. There is increasing awareness among those who provide and receive health care services, those who pay for those services and those who are making policy decisions that health care should be research led and that the services which are provided should be evidence-based. As a result, research agencies must find a way to demonstrate the benefits of the research produced, not only in terms of how many research findings are published in professional journals but how the investment in research results in practical everyday applications that can be used by people who need information to make decisions about health care.
Demonstrating that research has led to tangible effects in the care provided to individual patients is difficult. Impact is not always immediate. For example, it may be several years for a health care organization, that has adopted a policy based on research funded by AHRQ, to learn what effect it has had on overall patient care. Knowing that some clinicians or health systems are changing their practice is different from knowing how overall practice patterns are being influenced and what the effect is on clinical outcomes.
To address the need to demonstrate the impact of research on people's health, AHRQ staff developed a "pyramid of outcomes" model that includes four levels of impact. At the base level is the impact on knowledge and further research development, at the top is the impact on patient outcomes.
This model of assessing impact of AHRQ-sponsored research forms the basis for the development of performance measures. Similarly, AHRQ must ensure that performance measures are developed to assess the impact of the research investment at all levels of the pyramid.
Performance measures aimed at the base of the pyramid focus on research that contributes to the health care knowledge base, leads to future research, or both. Research at this level includes the development of tools and research methods, instruments and techniques to assist clinical decisionmaking, and identify areas that do not have a sufficient evidence base. The process indicators that are developed to measure performance at this level of the pyramid assess the quantity or quality of activities that have the potential to contribute, at least indirectly, to helping AHRQ meet its strategic goals or to monitor the establishment of major new initiatives or implementation of improvements in core activities where significant resources are involved or the potential for significance of the ultimate impact is high.
Output indicators are developed aimed at capturing the impact at the second and third level of the pyramid. Research at the second level of impact is research that results in the creation of a policy or program by a professional organization, health plan, hospital, legislative body, regulator or accrediting organization. Similarly, impact at level three of the pyramid is defined as research that results in a change in what clinicians or patients do, or changes in a pattern of care.
AHRQ has developed outcome indicators to measure impact at the fourth level of the pyramid, that is impact on the quality of care, actual health outcomes, cost of treatment or access to health care. Often, however, the connection between a particular research project and health outcome is indirect and can take years to emerge. As a result, AHRQ has developed measures that utilize a "convergence of evidence" approach to establish a connection between research and outcomes. This involves identifying bodies of research which, when considered together, establish a connection between research and outcomes.
Phase of Initiative |
Year 1—research initiative starts |
Years 3-5—results received |
Years 3-10—results used in health care system |
---|---|---|---|
Indicator type |
Process indicators |
Output indicators |
Outcome indicators |
Indicator examples |
Grants funded, creation of reports, partnerships formed |
Publications, Web site, dissemination, research findings, reports, products available for use in health care system |
Results of evaluation studies, users stories, analysis of trend/other data |
The AHRQ GPRA annual performance report and plans are aligned with the Agency's three budget lines:
The first two budget lines are where Agency programs are funded. The goals associated with each of the budget lines represent core activities funded in each. The following two tables illustrate how the GPRA goals are aligned with the AHRQ budget lines.
Table 1, representing the GPRA goals for FY 2002, uses the cycle of research as a basic framework underpinning the development of goals and measures for AHRQ's budget line: Research on Health Care Costs, Quality and Outcomes.
What the Indicators Address |
GPRA Goal |
---|---|
Budget Line 1: Research on Health Care Costs, Quality, and Outcomes |
|
Cycle of Research Phase 1: Needs Assessment |
GPRA Goal 1: Establish Future Research Agenda Based on User's Needs. |
Cycle of Research Phase 2: Knowledge Creation |
GPRA Goal 2: Make significant contributions to the effective functioning of the U.S. health care system through the creation of new knowledge. |
Cycle of Research Phase 3: Translation and Dissemination |
GPRA Goal 3: Foster translation of new knowledge into practice by developing and providing information, products, and tools on outcomes, quality, access, cost and use of care. |
Cycle of Research Phase 4: Evaluation |
GPRA Goal 4: Evaluate the effectiveness and impact of AHRQ research and associated activities. |
Lead role for quality initiative |
GPRA Goal 5: Support Department-wide initiatives to improve health care quality through leadership and research. |
Budget Line 2: Medical Panel Expenditure Survey |
|
Core MEPS activities |
GPRA Goal 6: Collect current data and create data tapes and associated products on health care use and expenditures for use by public and private-sector decisionmakers and researchers. |
Budget Line 3: Program Support |
|
Agency management activities: contracts management and the AHRQ Intranet. |
Goal 7: Support the overall direction and management of AHRQ. |
Beginning in FY 2003, AHRQ has redesigned its strategic management system and revised its GPRA goals to align more closely with the Agency's strategic plan. Table 2 shows this revision and realignment and the strategies AHRQ will use to accomplish these goals.
What the Indicators Address |
GPRA Goal and Strategies for Meeting the Goal |
---|---|
Budget Line 1: Research on Health Care Costs, Quality, and Outcomes |
|
Strategic Goal 1: Support Improvements in the quality, safety and outcome of healthcare |
To have measurable improvement in the quality, safety and outcome of healthcare for Americans.
|
Strategic Goal 2: Identify Strategies to Improve Access, Foster Appropriate Use and Reduce Unnecessary Expenditures |
To develop the evidence base for policymakers and health systems to use in making decisions about what services to pay for, how to structure those services, and how those services are accessed.
|
Strategic Goal 3: Build capacity to improve the quality of health care |
To build the capacity for improving the Quality of Health Care Delivery through Research and Training.
|
Budget Line 2: Medical Panel Expenditure Surveys |
|
Core MEPS activities |
To provide comprehensive, relevant and timely data on health care use and expenditures for use by public and private sector decisionmakers and researchers.
|
Budget Line 3: Program Support |
|
Enhance the value of AHRQ as the leader in Healthcare Outcomes, Quality, Cost, Use and Access |
Maximize the value of AHRQ by developing efficient and responsive business processes, aligning human capital policies and practices with AHRQ's mission, building an integrated and reliable information technology infrastructure.
|
This realignment will help AHRQ determine how well the basic knowledge which forms the core of AHRQ's work provides information that can be turned into actions by clinical decisionmakers, purchasers and providers who make decisions about what services to use and pay for and how to structure those services, as well as by policymakers.