GPRA Goal 3: Foster translation and dissemination of new knowledge into practice by developing and providing information, products, and tools on outcomes; quality; and access, cost, and use of care. (HCQO)

Strategy

This phase of the cycle of research bridges the gap between the development of new knowledge and its implementation in the health care system. AHRQ has taken its commitment to "ensure that the knowledge gained through health care research is translated into measurable improvements in the American health system" and integrated it in its approach to promoting the adoption and use of research findings.

Through an investment in demonstration projects, public (Federal, State, and local government) and private-sector partnerships, and targeted dissemination activities, AHRQ is focusing on closing the gap between what we know and what we do. We have named this focus "Translating Research Into Practice."

Building on the previous 10 years of research findings, AHRQ will identify ongoing gaps between what we know now and what we do in health care and will begin to close those gaps through research and demonstrations that develop and test implementation strategies in different settings in the health care system. A major focus within this goal is identify ing existing implementation strategies in use in health care settings and demonstrating their applicability to widespread dissemination in other areas of the system.

Types of Indicators

AHRQ uses output indicators, with some process indicators, to assess its progress in the translation and dissemination of research. The indicators regarding number of partnerships, attendees at User Liaison Program meetings, or hits on the AHRQ Web site helps the Agency determine that what it produces is of use to major audience segments. The Agency will evaluate the results of the GPRA plan indicators in combination with other information such as details about what products were released, feedback from attendees at programs, where the hits are on the Web site, and feedback from customers to manage and improve its dissemination efforts.

Frequently, the results of research are not readily implemented in the health care system without an interim step such as the creation of a tool that facilitates use. A major focus for the Goal 3 indicators, therefore, is to look at the creation and use of tools. The indicators for the Agency's investment in training helps the Agency track its success in furthering the field of health services research by fostering new talent. The number of trainees funded are a reflection of Agency commitment and the success of the training programs in attracting successful candidates. This data can be used in combination with other information about individual trainees, their research projects, professional credentials of professors and mentors, etc. to assess the overall success of the program.

Use of Results by AHRQ

The Goal 3 indicators are used to assess AHRQ's ongoing efforts to Translate Research Into Practice. The statistics on such things as usage of the Web sites, number of and attendance at User Liaison Programs, and/or the production of evidence reports by the Evidence-based Practice Centers are used to measure output—Is the Agency developing and disseminating the products needed by users?

AHRQ combines these statistics with evaluations of customer satisfaction and the use and usefulness of the products in order to assure quality as well as quantity. The Agency also uses the information to allocate resources, for example, providing the staff needed to maintain and update the Web site, expand its capacity, and respond to user suggestions for improvements. As previously stated in this report, the indicators of actual use of the research and/or products provides AHRQ with data on its ultimate goal of getting research into use in the health care system. The volume of the use that we document is an indicator of the total use and helps the Agency identify implementation issues.

Data Issues

Data collection for Goal 3 falls into two basic categories: collection through Agency data systems and collection through routine program management.

AHRQ has implemented several computer-based reporting tools to monitor usage of Agency information systems and websites. Accurate statistics are recorded periodically on the usage of the National Guideline Clearinghouse™, Publications Clearinghouse, and various other Agency websites and systems using commercially available reliable and accurate tools, e.g., WebTrends. These tools are used by many corporations and government agencies nationwide to monitor usage and have been certified by various information technology testing and review groups. This category includes information on categories of grants. Information on all grants, which can be word searched, is included in the Agency Management Information System.

AHRQ tracks print media (newspapers, health care-related trade journals and newsletters, and consumer magazines) news about or involving the Agency through a contractor—Burrelle's—one of the Nation's largest and oldest news clipping services. Beginning in calendar year 2000, Burrelle's will begin monitoring online news services. AHRQ staff is currently doing this task. AHRQ also monitors TV and radio news reports on selected studies in major markets around the United States through another contractor, Video Monitoring Service.

The other statistics are maintained by Agency program staff during the normal monitoring of contracts and grants. Certain items, such as the release of a CONQUEST product, are documented on the AHRQ Web site when ready so that consumers are aware of the availability. Other items such as the statistics on the ULP program are monitored through the management of the support contracts, travel arrangements, and other records kept in administering the program. Anecdotal information is verified with the primary source before being used by the Agency in this report or for any other uses.

Previous Successes in Implementing Research

AHRQ research has identified numerous opportunities for improvement in the quality of care. Some recent findings from AHRQ research include the following:

Schizophrenia PORT

The Schizophrenia PORT produced the evidence needed for the development of treatment recommendations which have been adopted by Massachusetts in a statewide quality improvement program. Initiated by the State Mental Health Director, all treatment facilities will use the recommendations to guide their management of schizophrenic patients. The National Alliance for the Mentally Ill (NAMI) has also used the recommendations to develop a consumer booklet offering advice to patients and family members, which was disseminated to members nationwide.

One section of the treatment recommendations developed through the work of the Schizophrenia PORT is focused on Assertive Community Treatment (ACT) programs, a multi-disciplinary team approach that shares caseloads and offers 24-hour mobile crisis teams, assertive outreach for treatment in the community, individualized treatment, medication, rehabilitation and support services. HCFA has issued a letter to all State Medicaid Directors endorsing the use of ACT programs and confirming Medicaid coverage of additional costs if this treatment model is implemented. Again, NAMI has launched an initiative to promote these programs in the remaining 25 states that have no similar program.

Medical Errors

AHRQ's medical errors study by Leape helped influence three major facilities in the Boston-based Partners HealthCare System as they focused on reducing medication errors:

• Massachusetts General Hospital.
• Brigham and Women's Hospital (where the AHRQ study was conducted).
• Dana Farber Cancer Institute.

These facilities have added special software to their computerized information systems to reduce medication errors. Called the "Physician Computer Order Entry," the program allows doctors to enter their medication orders, including dosage, route and frequency, directly on computer terminals, thereby eliminating handwritten orders. The system also alerts doctors when an order contains a possible error, such as a potential drug interaction or allergic reaction by the patient.

A test of the software at the 714-bed Brigham and Women's Hospital found that it decreased the rate of serious "nonintercepted" errors—mistakes that could have or did cause an adverse drug event and which were not caught before reaching the patient—by more than half. In addition to protecting patients, the new software is estimated to save the hospital between $5 and $10 million annually, even after accounting for development, start-up and maintenance costs.

GPRA Goal 3 Fiscal Year 1999 Results

Objective 3.1: Promote distribution of AHRQ publications, products, and tools through intermediary organizations.

Indicator

Formation of a minimum of five partnerships to support dissemination of AHRQ products through intermediary organizations, such as health plans and professional organizations.

Results

Thirty public-private and public-public partnerships were formed in Fiscal Year 1999. See details below.

Public-Private Partnerships

Put Prevention Into Practice (PPIP)—Materials to support a national campaign to improve the delivery of clinical preventive services such as screening tests, immunizations, and counseling for behavior change. PPIP materials include a clinicians handbook and health guides for adults and children. Select for PPIP Materials.

Nine companies/organizations reprinted and disseminated PPIP materials. Examples:

• Texas Dept. of Health—Austin, TX.
• Presbyterian Health Care—Albuquerque, NM.
• UCare Minnesota—St. Paul, MN.
• OmniCare Health Plan—Memphis, TN.
• American Association of Family Physicians.

Public-Public Partnership

Staying Healthy at 50+—HRSA and AARP are partnering with AHRQ and have developed a PPIP Personal Health Guide for Adults Over 50, which was launched at the Healthy People 2010 conference on January 25, 2000.

Quality Navigational Tool (QNT)—an interactive tool designed to help people use evidence-based information on quality and to take a more active role in their health care.

Newsletters and Web sites—Fourteen companies/organizations disseminating the information to their employees, including it in their newsletters or on Web sites. Examples:

• Midwest Business Group in Health.
• United Parcel Service (UPS).
• Safeway.
• Henry Ford Health Plan.
• Blue Cross/Blue Shield of Michigan.
• IBM.
• Erie Insurance.
• National Consumers League.

Smoking Cessation—clinical practice guidelines for physicians issued in 1996 that provide evidence-based information on how to help patients stop smoking and patient brochures containing recommendations on how to stop smoking.

Fourteen companies/organizations reprinted and disseminated smoking cessation materials based on the guideline. Examples:

• The National Medical Association.
• Pharmacy Council on Tobacco Dependence (PCTD).
• The Utah Tobacco Prevention and Control Program.
• American Academy of Pediatrics.
• Michigan Department of Community Health.
• American Cancer Society.

Additionally, all evidence reports and technology assessments are undertaken only when partners have been identified to take the findings and use them in developing a practice guideline or some other tool that will facilitate their use in the health care system.

---

Objective 3.2: Maximize dissemination of information, tools, and products developed from research results for use in practice settings.

Indicator

Number of hits on the Web site (Baseline: 2.9 million per year in 1997, nearly triple the hits in 1996.)

Results

15.5 million hits.

Indicator

Number of inquiries handled on Web site. (Baseline in Fiscal Year 1997—1300; in Fiscal Year 1998—2500.

Results

2,950 inquiries.

Indicator

Number of uploaded documents. Baseline in Fiscal Year 1997—950; in Fiscal Year 1998—1450).

Results

4,000 files/docs uploaded.

Indicator

Number of State and local governments trained in the understanding and use of health services research findings through User Liaison Program (ULP) Workshops.

Results

48 states, 4 territories, 30 county governments from 14 states, and 9 city governments from 7 states.

Indicator

Number of ULP meetings held. Baseline—Meetings held. 10 held in Fiscal Year 1997; 9 held in Fiscal Year 1998; 12 scheduled in Fiscal Year 1999.

Results

• Thirteen 2 1/2-day national workshops were held: State and local health policymakers from all States were invited to attend 12 of these and only State and territorial legislators were invited to attend 1.
• Two 1 1/2-day national seminars were held: one included State and local health policymakers from all States and the other included only State rural health directors, directors of Aging, and one other official with a rural or aging focus from 10 States in HHS regions V and VII.
• One 1-day "Masters" seminar was held that included only very senior State health officials entitled, "What Can States Do to Foster Reengineering of the Health Care Delivery System."
• Two 1-day "State-specific" workshops were held to which only State representatives from the organizing State were invited to attend:
  • "Vermont: Uniting for Health Care" cosponsored by the Vermont Division of Health Care Administration, the Vermont Employers Health Alliance, the Vermont Program for Quality in Health Care, and AHRQ.
  • "Exploring Quality Consumer Health Information in Texas" cosponsored by the Statewide Health Coordinating Council, Information Ad Hoc Committee, the Texas Health Care Information Council, and AHRQ.

Indicator

Number of attendees. Baseline: 538 attendees in calendar year 1997.

Results

834 attendees.

Indicator

States represented. Baseline—Fiscal Year 1997 and 1998—all 50 states and Puerto Rico.

Results

48 States plus D.C., Puerto Rico, Virgin Islands, Micro Polynesian Islands, and Guam. (Hawaii and North Dakota are the only two States not represented.)

Indicator

Number of hits on National Guideline Clearinghouse™ (NGC) with analysis of use by page, type of guidelines accessed, whether the guideline was downloaded, and linkages to other sites.

Results

The measures listed in the original performance plan were developed prior to the development of the NGC contract. These were changed to indicators that provide the Agency with better indicators of actual usage.

Definitions:

• Hit: Any connection to an Internet site, including online images and errors.
• Request: Any hit that successfully retrieves content.
• Visit: A series of consecutive requests from a user to an Internet site.
• User: Anyone who visits the site at least once.

Total users: 329,715.

Average visit/user: 2.24.

Average users/organization: 9.68.

Average number of requests/user: 23.57.

Number of hits: 13,590,013.

Number of requests: 7,771,095.

Average number of requests/visit: 10.53.

Number of organizations: 34,064.

Number of U.S. organizations: 10,045.

Number of Canadian organizations: 158.

Number of International organizations: 3, 627.

Unknown: 20,227.

Indicator

At least five purchasers/businesses use AHRQ findings to make decisions.

Results

Evidence Reports:

• Use of Erythropoietin in Hematology/Oncology: The Health Care Financing Administration will revise its coverage decision guidance on the topic based on the evidence report.
• Criteria To Determine Disability In Patients with end-stage renal disease (ESRD) (ECRI EPC): The Social Security Administration will use to determine if more research is needed and if its coverage decision guidance on this topic requires revision.
• Criteria for Referral of Patients with Epilepsy: Will be used by the Centers for Disease Control and Prevention for inclusion in a clinical practice guideline.
• Diagnosis and Treatment of Dysphagia: Department of Veterans Affairs are considering initiating a study to fill some of the gaps in the research identified in this report. The Health Care Financing Administration will revise its coverage decision guidance on the topic based on the evidence report.
• Testosterone Suppression Treatment for Prostatic Cancer: the Health Care Financing Administration will use to update its coverage decision guidance on this topic.

Consumer Assessment of Health Plans Surveys (CAHPS®):

The following are examples of organizations using CAHPS® to inform consumers' choices of health plans:

• Daimler Chrysler, Ford and GM.
• Colorado Business Group on Health.
• Colorado Department of Health Care Policy.
• Delaware Health Care Commission.
• Employee Health Care Alliance, Wisconsin.
• Iowa Department of Personnel and Human Resources.
• Kansas Foundation for Medical Care.
• Maryland Health Care Commission.
• Minnesota Buyers Health Care Action Group and Minnesota Department of Employee Relations.
• New Jersey Medicaid.
• New Mexico Health Policy Commission.
• New York State Department of Health.
• Office of Vermont Health Access.
• Oklahoma Health Care Authority.
• Central Florida Health Care Authority.
• Texas Department of Health.

In all, there are a total of 25 States functioning as a purchasing agency for its employees or for Medicaid beneficiaries have used CAHPS® Those not listed above include: Alaska, Arkansas, California, Georgia, Lousiana, Massachusetts, Michigan, North Carolina, Oregon, Pennsylvania, Utah. CAHPS® has been used in a total of 41 States when those for which it has been used by the U.S. Office of Personnel Management are included.

---

Objective 3.3: Develop and facilitate the use of new tools, talent, products, and implementation methodologies stemming from research portfolio.

Indicator

Evidence-based Practice Centers (EPCs) produce a minimum of 12 evidence reports and technology assessments that can serve as the basis for interventions to enhance health outcomes and quality by improving practice (i.e., practice guidelines, quality measures, and other quality improvement tools). At least four reports are being used by customers to develop practice guidelines or other interventions. Baseline in Fiscal Year 1998—12 reports produced. Fiscal Year 1999 will be the first year any interventions will be in development based on the reports.

Results

In Fiscal Year 1999, 10 evidence reports were published and three more were "in press" at the end of the fiscal year. Thirty additional reports are currently under development. Nineteen evidence reports are being used to develop clinical practice guidelines by organizations such as the American Psychiatric Association, American Academy of Pediatrics, American College of Obstetrics and Gynecology, American Academy of Family Physicians, the Consortium for Spinal Cord Medicine, American Academy of Cardiology, and American Heart Association.

Indicator

The AHRQ software product, CONQUEST 2.0, was released in Fiscal Year 1999 and contained new measures, including measures for new conditions, and updated measures. A contract was awarded to create Web-based product for more timely updating of information contained within the product.

Results

CONQUEST 2.0 was released in March 1999. Over 3000 hard copies of the product have been distributed and also additional copies have been downloaded from the Web. The contract due to be awarded in Fiscal Year 1999 (the National Measures Clearinghouse) was canceled in order for the Agency to better assess the future integration of the Web-based CONQUEST product with the AHRQ National Guideline Clearinghouse™ effort. The contract to create a Web-based product is expected to be awarded by September 30, 2000.

Indicator

Funding of a minimum of five major projects that will develop products, tools, or methodologies for implementing research findings into practice in significant segments of the health care system (i.e., potential to be generalizable across health care systems, provider-types, or clinical areas.)

Results

Three enhancements to the CAHPS® instrument are underway: Group Practice Level CAHPS®, Medicare Disenrollment CAHPS®, Nursing Home CAHPS®. Three small business innovative research grants were funded. They will develop:

• A system to automate the management and delivery of clinical preventive services using an integrated approach.
• Health insurance purchasing decision-support tools for small employers.
• A home-based cardiac rehabilitation program utilizing the Internet as the primary link between case managers, patients, and family members.

Also, a number of grants funded under the Translating Research Into Practice initiative contribute to this indicator. Examples include grants that will:

• Validate a clinical guideline for community-acquired pneumonia one.
• Implement a computer-based health support systems.
• Explore methods for translating research on pain management into clinical practice with a specific focus on elderly hospitalized patients.
• Study methods to improve and increase screening for Chlamydia.
• Develop and test methods to evaluate the efficacy of acupuncture treatment for major depression during pregnancy.
• Develop analytical tools and methods for performing meta-analysis of findings from clinical studies that exhibit substantial heterogeneity to estimate treatment effects. (The findings will be useful to the Evidence-base Practice Centers and other groups responsible for analyzing data and providing evidence reports.)
• Develop patient-centered methods to assess the effectiveness of treatments for chronic neurologic diseases.

Indicator

At least two new tools, products, or methodologies become available from projects funded between Fiscal Year 1993 and Fiscal Year 1996. Baseline: Fiscal Year 1999 results.

Results

• HCUPnet (Health Care Costs and Utilization Project) is now available for public access on the Agency's Web site. HCUPnet allows users to tailor an online query of HCUP's National Inpatient Sample (NIS), the largest all-payer inpatient database in the U.S. (Select for more detail.)
• Eleven of the 22 Statewide Inpatient Databases (SID) from HCUP are now available from a single point of access, under the auspices of AHRQ. Prior to September l999, the only means to access SID data was to approach each HCUP partner state on an individual basis, determine if the data organizations released their SID, obtain information about state-specific application processes, and successfully complete the application processes. (Select for more detail.)
• Three products developed through the Small Business Innovative Research contracts:
  1. Johnston Zabor and Associates developed SmartChoice to help employees choose health insurance plans. It has been purchased by several large employers. OPM and NIH also purchased this product and developed a demonstration Web site that helped Federal employees in the Washington-Baltimore area choose health insurance plans during the Fiscal Year 1998 open season.
  2. Abacus developed a workbook and video in English and Spanish to assist low income workers choose health plans. The materials are available commercially and have been integrated into Abacus' benefits management services.
  3. Benova developed a computerized decision tool to help Medicaid beneficiaries choose health plans. The tool is available commercially as a stand alone product or can be included in Benova's Medicaid enrollment programs.

Indicator

Support a minimum of 150 pre- and post-doctoral trainees. Baseline: 150 trainees funded per year. Commitment is to maintain the current level of support in Fiscal Year 1999.

Results

In Fiscal Year 1999, 167 trainees were supported.

GPRA Goal 3—Fiscal Year 2000 and 2001 Indicators

Objective	Fiscal Year 2000 Indicator	Fiscal Year 2001 Indicator
Fiscal Year 2000 Objective 3.1: Promote distribution of AHRQ publications, products, and tools through intermediary organizations. (Merged with Objective 3.2 in Fiscal Year 2001)	Formation of a minimum of 5 partnerships to support dissemination of AHRQ products through intermediary organizations, such as health plans and professional organizations. Baseline in Fiscal Year 1999: 30 partnerships used to disseminate materials.	See Objective 3.2
Fiscal Year 2000 Objective 3.2: Maximize dissemination of information, tools, and products developed from research results for use in practice settings. Changed To: Fiscal Year 2001 Objective 3.1: Maximize dissemination of information, tools, and products developed from research results for use in practice settings.	Web site: Number of hits on the Web site. (Baseline: Fiscal Year 1999 - 15.5 million hits.) Number of inquiries handled on Web site. (Baseline in Fiscal Year 1999 - 2950.) Number of Uploaded documents. Baseline in Fiscal Year 1999 - 4000.) Reports from user surveys on how the information requested was used. User Liaison Program: Number of meetings held. (Baseline - 13+ meetings held in Fiscal Year 1999. See details of 1999 results.) Number of State and local governments trained in the understanding and use of health services research findings through User Liaison Program (ULP) Workshops. (Baseline - 834 attendees in calendar year 1999). Reports from annual participants on how the information was used in decisionmaking. National Guideline Clearinghouse™ (NGC): Statistics on usage of National Guideline Clearinghouse including number of hits, requests, organizations, and total users. (Baseline: See Fiscal Year 1999 results for details.) Use of research findings: At least 10 purchasers/businesses use AHRQ findings to make decisions.	Web site: Same indicators used. User Liaison Program: Same indicators used. Partnerships: At least five public-private partnerships are formed to implement research findings for decisionmakers. Budget: Commitment Base. Formation of a minimum of 10 partnerships to support dissemination of AHRQ products through intermediary organizations, such as health plans and professional organizations. Budget: Commitment Base.
Objective 3.3: Develop and facilitate the use of new tools, talent, products, and implementation methodologies stemming from research portfolio. Fiscal Year 2000 Priority (3), "Translating Research into Practice," focuses on the translation and dissemination of research findings, products, and tools to foster adoption and use in health care settings. Objective 3.2 in Fiscal Year 2001: Develop and facilitate the use of new tools, talent, products, and implementation methodologies stemming from research portfolio.	Demonstration of use of at least three AHRQ research findings in systematic efforts to Translate Research Into Practice. Baseline: Under development. Funding of a minimum of five major projects that will develop products, tools, or methodologies for implementing research findings into practice in significant segments of the health care system (i.e., potential to be generalizable across health care systems, provider-types, or clinical areas.) (Baseline: Under development.) At least two new tools, products, or methodologies become available from projects funded between Fiscal Year 1993 and Fiscal Year 1996. (Baseline: 16 projects identified in Fiscal Year 1999.) Support a 5-percent increase, at a minimum, in number of pre- and post-doctoral trainees. (Baseline: 167 trainees funded in Fiscal Year 1999.)	Provide evidence summaries for use in Federal direct care providers' efforts to create guidelines. Budget: Commitment Base. Evidence-based practice centers (EPCs) will produce a minimum of 12 evidence reports and technology assessments that can serve as the basis for interventions to enhance health outcomes and quality by improving practice. Budget: Commitment Base. Support a minimum of 165 pre- and post-doctoral trainees. Budget: Commitment Base. Support a minimum of 10 minority investigators through individual and center grants. Select for Budget: Strengthening Minority Health Services Research Capacity and Commitment Base. Fund at least 10 projects in tool development. Select for Budget: Strengthening the Nation's Ability to Improve Patient Safety, Web-Based Applications, and Commitment Base.

Return to Contents
Proceed to Next Section