Appendix 1. Approach to Performance Measurement
The goals and objectives of the AHRQ performance plan are aligned with the Agency's three budget lines.
Performance Indicators
Accurately measuring the outcomes of research programs continues to be a challenge. By its very nature, research is unpredictable. Research activities may or may not yield conclusions that are immediately amenable to application. There can be a considerable time lag between research activities and the outcomes of those activities. Whether and how research findings get used in the health care system is dependent on countless variables over which AHRQ has no control. Another important limitation is the nature of extramural research. AHRQ cannot control what types of applications it will receive, nor what the results will be of the funded research. Thus, AHRQ, like other research agencies, continues to face the challenge of showing the impact of research activities within these constraints.
In order to mitigate these factors, the Agency sets research priorities based on its strategic plan and input from the end users of the research. Program announcements and requests for applications are used to communicate the research priorities to the field. The portfolio of research is managed to contain a mix of short and long term projects. Partnerships are integral to the conduct of AHRQ's work to promote timely application. More recently, added emphasis has been placed on efforts to translate research into practice to assure impact.
To understand and report on the impact of Agency programs on health care, additional emphasis is being placed on evaluation activities. As a result, the Agency will be able to report on process, output, and interim outcome goals through out the course of its major initiatives. Goal 4—Evaluate the effectiveness and impact of AHRQ research and associated activities—is designed to capture the results of the emphasis on evaluation of impact. In each performance plan, the Agency includes a number of evaluations that illustrate the impact of research products when used to inform consumers, measure quality, and make policy decisions.
Alignment with Committee on Science, Engineering, and Public Policy (COSEPUP) Report on Evaluating Federal Research Programs
In reviewing the COSEPUP report (1), AHRQ was pleased to find that many steps taken in recent years to improve Agency evaluation processes and connections to users of Agency research are supported by the report's recommendations.
On page 6, the reports points to three types of expert review that are the most effective ways of evaluating federally funded research programs. These are quality review, relevance review, and international benchmarking.
Agency staff regularly make presentations to the AHRQ National Advisory Council on major research initiatives. The members provide direct feedback on the quality and relevance of the work. National researchers who make presentations to AHRQ staff provide another form of direct feedback on Agency programs.
Finally, as part of AHRQ's commitment to have its research begin and end with the user, expert meetings are held to gather input when planning new initiatives. These meetings provide another venue for national experts to provide feedback on the quality and relevance of work-to-date, as well as advice on directions for the future. While AHRQ does not have a formal benchmarking program, increased involvement of Agency staff with health care improvement efforts in Russia, programs at the World Health Organization, and other international activities are providing valuable input on Agency programs.
On page 38, the report states "In addition, agencies should conduct periodic reviews of the overall practical outcomes of an agency's overall past support of applied and basic research." AHRQ previously has conducted this type of review when developing and implementing next steps for major initiatives. With the advent of annual performance plans, evaluations of the outcomes of Agency investments has become integrated into the core processes of AHRQ evaluation activities. Recently, the Agency completed an evaluation of the outcomes of the outcomes research program. The results of the evaluation are informing the next phase of the outcomes research and other initiatives under development. Goal 4 of each GPRA performance plan presents the evaluation studies that the Agency is using to determine impact Agency research and products are having on the health care system.
AHRQ will continue to improve its performance measurement activities by strengthening, and making more explicit, the connections between current evaluation strategies and those recommended in the COSEPUP report.
(1) The committee on Science, Engineering, and Public Policy, Evaluating Federal Research Programs, Research and the Government Performance and Results Act, National Academy Press, Washington, D.C., 1999.
Data Collection
AHRQ recognizes that its commitment to accountability will not be achieved easily and that it entails an added investment in measurement as part of all of the Agency's programs. The Agency's approach consists of capitalizing on data collection opportunities as a byproduct of the work we do or sponsor, partnering with public and private organizations, and maximizing the use of information technology applications. AHRQ will use a variety of data collection methods.
Research Applications. Studies which address the translation of research into practice will be required to include in their design appropriate measures of impact.
Grants Management Databases. AHRQ is investing in the development of an Intranet-based integrated information management system through which progress on funded grants and their results will be captured as part of routine reporting. A number of yes/no indicators are included in the plan, and they should present few data collection problems thanks to this system.
Information Technology. Expanded use of technologies (e.g., Intranet, extranets, LISTSERV®) will permit efficient capturing of important qualitative information on the impact of Agency programs. (One example is Objective 3.2, where AHRQ will report on the impact of the User Liaison Program by gathering user stories from attendees on their use of program information in decisionmaking.)
Performance Management System. AHRQ has aligned its employee performance management system with each organizational unit's plan and the Agency plan and incorporated many process and output measures into employee plans. These will be aggregated annually to yield some of the measures in the GPRA plan.
Customer Surveys. These are a critical source of information on the appropriateness, use, and quality of AHRQ products and services. This approach is being expanded in the 2000 plan. In some instances, the mechanisms for collecting customer service data are already in place and the first set of data has been analyzed, such as with the Publications Clearinghouse. In other instances, the customer surveys must be designed, fielded, and responses analyzed. Because of the substantial financial costs involved, often it will be necessary to survey large representative samples to obtain information on the usefulness, relevance, and quality of AHRQ's work and its associated impact. The Agency has started working on the issues, including cost analyses for the surveys required, and expects to have the necessary infrastructure in place by Fiscal Year 2000. We will work closely with HHS both to share our experience and to apply the knowledge and expertise of others. Identifying opportunities for collaborations and/or cost sharing will be a priority. Notations are made within the text of the plan for each measure that will use a survey mechanism.
Partnerships. Many public and private sector organizations collect data on processes of care which AHRQ programs and research are intended to improve. For example, Peer Review Organizations (PROs) have taken AHRQ research findings and recommendations, worked with practitioners and institutions to adopt them, and provided the Agency with feedback regarding improvements in practice. The Health Care Utilization Project (HCUP) database, developed in partnership with 19 States, provides additional insights regarding changes in clinical practice in those states. Another source of information will be the growing number of collaborative research projects that make use of the internal databases of large private sector health care delivery organizations. A rich source of information will be the external organizations that serve as partners for the reports being developed by the Agency's Evidence-based Practice Centers program. They have made a commitment to implement the reports in a variety of ways and will provide the Agency with data on the utilization and impact of their efforts. By working collaboratively we can meet some of our measurement needs more cost-effectively.
Evaluations. Specifically commissioned studies (both intramural and extramural) will be used to evaluate the impact of AHRQ programs more rigorously. These studies are presented in AHRQ's Goal 4. These studies will evaluate such things as:
- The effect a product, e.g., a quality measurement tool, had on improving the quality of health care.
- Whether a product that is effective in one care setting can be generalized to other settings, e.g., a clinical decision support system.
- Whether a product is user friendly and useful.
Because these studies will be complicated, resource-intensive, and expensive, the Agency will propose a limited number each year.
Other Mechanisms. The Agency has developed a variety of other mechanisms that will enable it to collect information on the impact of its work. These include:
- Research Translation Team. This newly formed group is responsible for capturing and distilling qualitative data on the use of Agency sponsored and conducted research and products in the health care system. Through investigating the details of anecdotal evidence, literature searches, tracing the impact of completed research projects and other methods, AHRQ will compile evidence of the impact that it is having in the health care system.
- Partnership liaison. AHRQ has assigned a senior staff person to stimulate and coordinate partnerships and liaisons with other organizations within the Federal Government, State governments, and the private sector. This will help us identify the uses to which existing research and products have been used, stimulate implementation demonstrations, and identify the need for future research. The creation of this position is part of the Agency's increased efforts to create clear and ongoing mechanisms to obtain input from the user community.
- Conferences and expert panels. The agency will convene conferences and expert panels to help identify effective methods of translating research into practice and evaluating those methods through demonstrations and other projects. The information gained through these activities will impact the Agency's future research agendas and its translation and dissemination activities.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850