Detailed Information on the
Pharmaceutical Outcomes Assessment

Continue analysis of program and modify measures and data, if necessary.

Produce reports on best practices in observational methods research in therapeutics and publish on AHRQ's website. Submit reports to health sciences journals for peer-reviewed publications.

Update 2006 targets

Develop annual targets for long-term measures for out-years.

Analyze 2006 trends to determine if targets for measures need to be adjusted.

Update performance targets.

Measure: The percentage of hospital readmissions within 6 months for congestive heart failure in patients between 65 and 85 years of age.

Explanation:The program's efforts to reduce readmissions among individuals with congestive heart failure (CHF) largely center on improving compliance and adherence of physician prescribing, and patient use of, medications such as beta-blockers and ACE inhibitors, which have been shown to have a positive effect in reducing hospitalizations, improving functionality, and decreasing mortality of CHF. Program activities related to congestive heart failure include funding educational materials such as brochures and videotapes for patients, web-based training and fact sheets for providers, summary guides for clinicians and patients comparing different hypertension treatment options, and research that is ultimately published in academic journals. The data source for this measure is the Healthcare Cost and Utilization Project (HCUP) databases of hospital admissions data. The average age among the 65 to 85 year-old population is increasing, so to risk-adjust targets, the 2014 target is only 30% (instead of a lower target readmission rate).

Measure: The decrease in the rate of hospitalization for upper gastro-intestinal bleeding due to the adverse effects of medication or inappropriate treatment of peptic ulcer disease in patients between 65 and 85 years of age.

Explanation:The program seeks to decrease gastro-intestinal (GI) bleeding (as well as other adverse bleeds, such as central nervous system hemorrhages) by improving the use of drugs that deliberately interfere with normal clotting mechanisms (such as warfarin) or incidentally decrease normal clotting as a side effect of their principal therapeutic intent (such as NSAIDs). Multiple projects address how to dose, monitor, and predict appropriate drug therapy levels. Program activities related to peptic ulcer disease include funding online courses for providers, summary guides for clinicians and patients comparing different treatment options for osteoarthritis and gastroesophageal reflux disease (heartburn), and research that is ultimately published in academic journals. The data source for this measure is the Healthcare Cost and Utilization Project (HCUP) databases of hospital admissions data. The percent change identified in the targets for this measure are relative to the previous year's actuals.

Measure: The number of prescriptions of antibiotics per child aged 1 to 14 in the U.S.

Explanation:Inappropriate or excessive antibiotic use contributes to antibiotic resistance (e.g. methicillin-resistant Staphylococcus aureus, and vancomycin-resistant enterococci). Antibiotics are one of the most commonly used drugs in the pediatric population, and are likely overprescribed. AHRQ research projects aimed at reducing the inappropriate use of antibiotics include: analysis of the actual effectiveness of preventive antibiotic use (such as for vesicoureteral reflux); educational efforts to decrease patients' demands for antibiotics for viral upper respiratory infections and other illnesses; and analyses of population data to explore individual and system factors that contribute to antibiotic resistance. The data source for these population-level estimates is AHRQ's Medical Expenditure Panel Survey.

Measure: The cost per capita of hospital admissions for upper gastro-intestinal bleeding among patients aged 65 to 84.

Explanation:This program aims to reduce adverse effects of medications or inappropriate treatment of peptic ulcer disease (which can lead to hospitalizations for upper gastro-intestinal bleeding especially in elderly populations) by educating health care providers and patients about appropriate uses of medications. A reduction in the per capita charges for gastro-intestinal (GI) bleeding for patients suggests that fewer GI bleeds requiring hospitalization occurred; this may be due to reduced use of medications that cause GI bleeding and increased use of medications which protect against GI bleeding. This program works to reduce GI bleeding hospitalizations by conducting research on appropriate uses of medications, and educating patients and health care providers about the program's research findings from relevant reports, such as "Comparative Effectiveness of Management Strategies for Gastroesophageal Reflux Disease." The data source for this measure is the Healthcare Cost and Utilization Project (HCUP) databases of hospital admissions data.

Is the program purpose clear?

Explanation: The objective of Centers for Education and Research on Therapeutics (CERTs), the central component of the Pharmaceutical Outcomes portfolio, is clear. It is to conduct-state-of-the-art clinical and laboratory research to inform clinical practitioners and policy makers about both the uses and risks of new drugs and drug combinations, biological products, and devices as well as of mechanisms to improve their safe and effective use. CERTs were first originally established as a short-term demonstration program under the Food and Drug Administration in 1997, and then made permanent in 1999 with the Agency for Healthcare Research and Quality (AHRQ) reauthorization. The FY 2004 appropriation language for AHRQ funds research on the comparative effectives, cost-effectiveness and safety of drugs, biological products and devices. In addition, the Medicare Modernization Act authorizes AHRQ to conduct and support research to meet the priorities and requests for scientific evidence and information.

Evidence: Evidence 1. Reauthorization ' Healthcare Research and Quality Act of 1999 (P.L. 106-129) 2. Consolidated Appropriations Act of 2004 (P.L. 108-199) 3. Medicare Prescription Drug, Improvement and Modernization Act of 2003 ' Section 1013 (P.L. 108-173) 4. Fact Sheet - Agency for Healthcare Research and Quality: Reauthorization Background The statute authorizes studies of the effectiveness and appropriateness of health care services and procedures in the prevention, diagnosis, treatment, and management of clinical conditions. The Agency for Healthcare Research and Quality's (AHRQ) Center for Outcomes and Evidence has lead responsibility for the development and administration of research programs related to patient outcomes associated with pharmaceutical therapy.

Does the program address a specific and existing problem, interest or need?

Explanation: Prescription and over-the-counter pharmaceuticals are central to many of the challenges in health services delivery. Understanding which therapeutics work for which patients and at what cost in the rapidly changing health care environment is needed. The Pharmaceutical Outcomes research portfolio addresses many of today's most critical health care issues, including those related to: the needs of the elderly, racial and ethnic disparities in service delivery, prevalent chronic conditions, health care prevention, treatment effectiveness, the cost and quality of patient care, and research and patient management tools that support evidence-based practice.

Evidence: Evidence Outcomes of Pharmaceutical Therapy Preliminary Update (November 2003)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: There is considerable duplication among the research efforts of the National Institutes of Health (NIH) and AHRQ. Although the research of these agencies occurs in different settings, and NIH's overall research focus is on molecular, biomedical, product development and pre-clinical focuses, the premise of much of NIH's pharmaceutical research is often the same--to compare affects and appropriateness of therapies. NIH's therapeutics activities consist of drug discovery and development, and clinical trials. In FY 2003, NIH funded an estimated $2.7 billion on clinical trials, which included a substantial investment on studies that compare one commonly available therapy to another, or a standard therapy to a newer investigational therapy. These comparisons are made in a controlled clinical setting and are commonly referred to as efficacy trials. A scan of the NIH website has identified several similarly funded activities. The Food and Drug Administration's (FDA) drugs program includes pre-marketing review of human and animal drugs and biological products in order to ensure their safety and efficacy and the post-marketing monitoring of drug experience. FDA conducts manufacturer inspections and sample examinations to ensure industry compliance. FDA's devices and radiological products program conducts pre-market review and post-market surveillance of medical devices to assure their safety and efficacy, and sets standards for the manufacture and use of biological products to protect the public from unnecessary exposure to radiation. AHRQ's activities related to therapeutics begin after product approval in "real world" settings through support of research on the relative effectiveness, appropriateness, and cost effectiveness of various strategies for prevention, diagnosis, treatment, and management of clinical conditions. Activities have included development and administration of a program to study patient outcomes, development of evidence based practice centers, and support of the development of quality measures.

Evidence: Evidence Outcomes of Pharmaceutical Therapy Preliminary Update (November 2003)

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program is free of major flaws that would limit its effectiveness. Available funds are announced through an open competitive process to attract the highest quality applications. Applications are peer-reviewed.

Evidence:  

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Legislation explicitly defines the audience for the CERTs as: 1) health care practitioners and other providers of health care goods or services, 2) pharmacists, pharmacy benefit managers and purchasers, 3) health maintenance organizations and other managed health care organizations, 4) health care insurers and governmental agencies, and 5) patients and consumers. For example, AHRQ awarded a five year grant (1993-1997) to the New England Medical Center for projects that focus on comparative outcomes of outpatient drugs. AHRQ's grants helped the Center to develop a computer-based, Real-Time Meta-Analysis System, which provides updates and displays randomized data regarding the effectiveness or comparative effectiveness of one or more drugs. Other grantees review such things as prospective drug use and patient outcomes relative to a given drug.

Evidence: Evidence 1. Reauthorization ' Healthcare Research and Quality Act of 1999 (P.L. 106-129) 2. CERTs Fact Sheet 3. Outcomes of Pharmaceutical Therapy Preliminary Update (November 2003)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program developed new long-term outcome goals that are directly linked to improved health outcomes. Goals focus on reducing congestive heart failure hospital readmission rates for those between ages 65 and 85 during the first six months after discharge, reducing the inappropriate use in children between ages one and fourteen, and reducing hospitalization for upper gastrointestinal bleeding.

Evidence: See "Measures" Tab

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has established ambitious targets and timeframes for the long-term performance measures. The targets are by 2014 to reduce hospital readmissions due to congestive heart failure from 38 to 20 percent, reduce prescriptions of antibiotics for children from .56 to .42 per child, and reduce hospitalizations for upper gastrointestinal bleeding from 55 to 45 per 10,000.

Evidence: See "Measures" Tab

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: For the FY 2006 PART, the program developed an efficiency goal that measures the annual cost of admission for upper gastrointestinal bleeding. Reducing the number of incidences of readmission will help to increase the efficiency of the use of federal resources, by reducing the cost to the health care system. The program also developed new annual measures focused on forming partnerships with leading national organizations that will work with their membership to ensure that changes are occurring in the health care system to help the program achieve the newly developed goals.

Evidence: See "Measures" Tab

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and targets have been established/timelines have been set for annual performance measures that support the long-term outcome goals for the program.

Evidence: See "Measures" Tab

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: AHRQ attempts to hold all grantees and contractors accountable by specifying in its Request for Applications (RFAs) which require grantees to "evaluate, develop options and methods, and conduct state-of-the-art, clinical, laboratory and health services research". Although GPRA goals are not explicitly referenced in the RFAs the terms of the RFAs and the grantees proposals include activities that help to facilitate improvements in health care as a result of improved understanding of the affects of therapeutics.

Evidence: Evidence CERTs Request for Applications

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations of sufficient scope have not been conducted on a regular basis, but AHRQ has committed to a comprehensive evaluation of the Pharmaceutical Outcomes portfolio due to the AHRQ's recently increased role in the comparison and effectiveness of medications in the Medicare legislation. The scope of the evaluation will be determined by mid-July. AHRQ anticipates the evaluation will begin in October. AHRQ has funded pharmaceutical outcomes research since the early 1990s. The first two independent evaluations were completed in 2001 and 2002. The evaluations varied in scope but addressed the following key areas of interest: data and analytic methods involved in the study of drug therapy effectiveness, factors affecting the appropriateness of drug prescribing, the role of the patient in drug therapy effectiveness, and economic analysis and the effects of changes in the health care environment. The Washington Consulting Group (WCG) published an AHRQ-funded evaluation of the CERTs program in January 2002. The evaluation was based on objectives set forth in the CERTs authorization. The evaluation also assessed the Coordinating Center's and Steering Committee's performance. The WCG assessed the quality and effectiveness of the program by considering its ability to expand therapeutics research capacity, its development of operational linkages and communication channels and its impact on research, practice and policy. The Health Systems Research, Inc. (HSR) released its report in 2001. HSR conducted an AHRQ-funded evaluation to assess the impact of pharmaceutical research studies funded during the 1990s by the Center of Outcomes and Effectiveness Research.

Evidence: Evidence 1. Washington Consulting Group Evaluation - "An Assessment of the Centers for Education and Research on Therapeutics Initiative" (January 22, 2002) 2. Health Systems Research, Inc. Evaluation - "The Impact of Studies Funded Under Outcomes of Pharmaceutical Outcomes Research" (October 2001)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program does not provide a presentation that makes clear the impact of funding, policy or legislative decisions on expected performance nor does it explain why a particular funding level/performance result is the most appropriate.

Evidence: Evidence DHHS Federal Fiscal Year Justification of Estimates for Appropriations Committees

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: To date, HHS/AHRQ has not tied its budget requests to the accomplishments of the annual and long-term performance goals. HHS does plan to submit a performance-based budget beginning in FY 2006, but is it unclear whether this budget will show the marginal impact of funding decisions. AHRQ

Evidence:  

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: AHRQ does not conduct periodic comparisons of the potential benefits of its pharmaceutical outcomes research portfolio with those of NIH who has similar goals.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Across AHRQ, the program conducted an inventory of all activities funded and categorized them according to overarching strategic goal areas and priority research topics. During this process it was determined that the Pharmaceutical Outcomes research activities were a priority to help inform clinical practitioners and policy makers about the risks and use of new therapeutics.

Evidence: Evidence AHRQ - FY 2005 Congressional Justification

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AHRQ requires every awardee to provide progress reports to Program Officers on a regular basis. This information includes a brief narrative on what was actually accomplished during the reporting period and a summation of the cost and level of effort expended for each task, preliminary or interim results and conclusions, problems or delays the awardee experienced adjustments that are being implemented to study plans, and planned activities during the next reporting period. Once an initial award has been made, it is the Office of Grant Management's (GM) role to review any post award changes requiring prior approval and to conduct an administrative review of non-competing continuation applications. Grant recipients must submit a non-competing continuation application each year as a prerequisite to continued funding. The awarding office may use the annual performance or progress report in lieu of a non-competing continuation application as the means of determining whether continued funding should be provided. Once GM is satisfied that all administrative requirements have been met by the grantee the award is released.

Evidence: Evidence Notice of Grant Award Letters

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: AHRQ's strategic plan guides the overall management of the agency. Each Office and Center has an individual strategic plan and annual operating plan. Cost, schedule and performance are part of the performance plans, including Division, Center, and Agency Directors. The annual operating plan identifies those things that contribute to AHRQ achieving its performance goals and internal management goals. These factors are incorporated into each employee's annual performance plan/review. At the end of each year, the Office and Center Directors review accomplishments in relation to the annual operating plans in preparation for drafting the next year's plans. The results of these reviews contribute significantly to Office and Center performance reports. Some managers' performance plans also take into consideration their staffs performance in managing program operation. In addition, contracts are performance-based.

Evidence: Evidence Agency for Healthcare Research and Quality Strategic Plan

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All appropriated funds are obligated in accordance with the annual operating plans, formulated for obligation and outlay on a quarterly basis. The administrative and funding instrument is a demonstration cooperative agreement award mechanism in which the Principal Investigator retains the primary responsibility and dominant role for planning, directing and executing the proposed project. AHRQ staff are involved as a partner. Grants Management staff monitor unexpended funds on an annual basis and notifies the program in the case of excessive annual carryovers. Upon review, program staff may recommend a reduction in the following year award. This has not been necessary in this program up to this point.

Evidence:  

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: AHRQ does not have procedures in place to measure and achieve efficiencies and cost-effectiveness. AHRQ does work diligently to ensure timely solicitations and release of awards; however, no processes are in place to measure improvements in timing.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: AHRQ often reviews the intent of its program relative to the activities funded by other agencies. Notices of Intent to fund research are published in the Federal Register and through Early Notice in the NIH Guide to Grants and Contracts. To this effect, AHRQ often closely coordinates its CERTs RFAs with the Food and Drug Adminsitration (FDA) and the Centers for Medicare and Medicaid Services (CMS). The CERTs Steering Committee Government representatives, in addition to AHRQ, consist of two members from FDA and one from CMS. The CERTs Steering Committee members actively work to keep agencies informed and identify areas of collaboration. The CERTs sponsor an annual meeting with government stakeholders. These activities have generated Inter-Agency Agreements.

Evidence: Evidence Inter-Agency Agreements with FDA and CMS

Does the program use strong financial management practices?

Explanation: In FY 2003, HHS OIG conducted an HHS financial statement audit. The audit reported that the Department had serious internal control weaknesses in its financial systems and processes for producing financial statements. OIG considered this weakness to be material. The audit recommended that HHS improve their reconciliations, financial analysis, and other key controls. AHRQ purchases its fund accounting, financial reporting, debt management and other related fiscal services from the Program Support Center on a fee-for-service basis. Because the Department prepares audited financial statements for its largest components only, AHRQ financial statements are not audited. In 2002, AHRQ engaged Clifton Gunderson LLP for technical support consultation for certain financial management practices.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: HHS' long-term strategic plan is to resolve the internal control weaknesses is to replace existing accounting systems and other financial systems within HHS with the Unified Financial Management System (UFMS). HHS plans to fully implement the UFMS Department-wide by 2007. AHRQ's strategic plan guides the overall management of the Agency. In the past, each Office and Center had an individual strategic plan and annual operating plan. AHRQ is revising these plans to address current portfolios of work. The revised versions will address cost schedule and performance along with identifying those things that contribute to AHRQ achieving its performance outcomes and internal management goals. The plans will track to the performance contract that the Director of AHRQ has with the Secretary of HHS. Program reviews focusing on performance results contribute significantly to Office and Center portfolio performance reports. Some managers' performance plans also take into consideration their staff's performance in managing program operation. AHRQ also has developed an efficiency measure as part of the FY 2006 PART process and is in the process of reviewing benchmarks for program management efficiencies in the areas of contracts and grants management. These efficiency measures will cut across portfolios. Draft measures will be available for review and approval in early August. In addition, AHRQ is establishing a Funding Decision Support System to track funding activity by portfolio of work. This IT improvement is designed to align each funded activity to associated outcomes within a portfolio. Data will be used to assess the cost of an outcome and annual accomplishments.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation:  

Evidence:  

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation:  

Evidence:  

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: All pharmaceutical program funding is based on competitive grant awards.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: There has been progress made toward achieving the newly developed long-term goals. For example, the average number of prescriptions for children in 1996 was .97 per child per year. It has fallen to .56 in 2001.

Evidence: See Questions 2.1-2.2

Does the program (including program partners) achieve its annual performance goals?

Explanation: There has been progress made toward achieving the newly developed annual goals as well. For example, the program already partners with several organizations, but will expand efforts to include commitments to ensure change and work toward the long-term goals.

Evidence: See Questions 2.3-2.4

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The number of readmissions for upper gastrointestinal bleeding has decreased which has caused the overall charge to the health care system to decrease as well.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: AHRQ does not conduct periodic comparisons of the potential benefits of its pharmaceutical outcomes research portfolio with those of NIH who has similar goals.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Overall, the WCG found that the program had been successful in contributing to the body of therapeutics research. WCG found also that over 100 partnerships had been formed with other federal agencies, foundations, providers, professional membership organizations pharmaceuticals and other private sector enterprises that led the WCG to conclude that the CERTS 'now have a national reach and importance.' Overall, HSR concluded the following: 1) the projects contributed important primary research in a number of key areas including treatment effectiveness, cost and economics of health care, tools for patient management, research tools and translating research for clinical care, special needs of target populations, and public health and prevention and chronic and persistent disease; 2) several of the projects demonstrated direct influence on the policies of one or more organizations including managed care organizations, State Medicaid programs, clinical associations, quality accreditation commissions, private insurance programs, integrated delivery systems and/or Federal agencies; 3) collectively the projects addressed a range of population groups that are important because they represent key public financing programs or because they address issues related to key populations; and 4) relevance of this body of work can be seen by comparing the issues it addresses to topics of current health policy interest.

Evidence: Evidence 1. Washington Consulting Group Evaluation - "An Assessment of the Centers for Education and Research on Therapeutics Initiative" (January 22, 2002) 2. Health Systems Research, Inc. Evaluation - "The Impact of Studies Funded Under Outcomes of Pharmaceutical Outcomes Research" (October 2001)