Executive Summary

The Partnerships for Quality (PFQ) program sponsored by the Agency for Healthcare Research and Quality (AHRQ) aimed to accelerate the translation of research findings into practice on a broad scale through partnerships led by organizations well-positioned to reach end users. Initiated in 2002, PFQ was one of AHRQ's first efforts to use partnerships to improve health care quality, safety and security.  Hence, AHRQ is very interested in understanding what can be learned from the experience. AHRQ contracted with Mathematica Policy Research, Inc. (MPR) towards the end of the program to evaluate PFQ and discern the lessons it might have for future efforts at translating research into practice. This final report provides the results of MPR's evaluation.

A. PFQ Program Goals and Context

AHRQ's solicitation for the PFQ program represented an important departure from its traditional health services research grants. It was designed to encourage applicants beyond the usual academic institutions the agency had historically funded. AHRQ wanted to fund "change agents" that not only possessed the evidence-based knowledge to improve care but also could create the partnerships and had the capacity to influence changes in health care organization and delivery. In addition to their own projects, grantee agencies were expected to participate in cross-grantee meetings and activities designed to foster learning and develop new knowledge on how to use partnerships to achieve quality goals.

AHRQ spent about $20.5 million on PFQ over the life of the program, of which about $17.6 million came from AHRQ appropriations and about $3 million from other DHHS funds. Most grantees received four years of funding, although a few were for shorter periods of time either by design or because problems arose. AHRQ originally awarded grants to 22 organizations, but only 20 remained after the first year. One of the 22 withdrew from the program before it received funding, and another grant was not renewed after the first year. These two are therefore not included in this evaluation.

The 20 projects that are the focus of this evaluation targeted a broad range of diseases, conditions, and health care issues or settings. Most (17) projects focused on clinical quality improvement and received grant awards of about $300,000 per year. Of the 17 in this first subset, 15 focused on improving provider quality of care, and 2 focused on purchasers' roles in influencing quality of care. The other three projects focused on improving the preparedness of health care providers to respond to bioterrorism and other emergencies, and received grant awards that were about $100,000 per year. Two of the 20 grantees had both bioterrorism preparedness and quality improvement components. In pursuing their goals, the 20 PFQ projects used a wide assortment of partnership models and partner organizations, and employed diverse strategies and techniques for increasing provider use and adoption of evidence-based practice. 

When PFQ was developed, AHRQ's mission was transitioning from focusing mainly on the production of knowledge to promoting the actual use of knowledge to improve care delivery. AHRQ senior executives involved at the outset indicated that they hoped the PFQ program would help promote a change in culture within the agency. Many AHRQ staff were involved in developing the grant solicitation, although several of them are no longer with the agency. The perceived novelty of PFQ's focus and the turnover in agency leadership involved in its development are important factors to understand in assessing PFQ's experience, since they affected the strength and clarity of the agency's direction for the program.

B. Key Evaluation Questions

The goals of this evaluation are to determine the effectiveness of the grant-funded projects, learn how partnerships could be used effectively to translate research into practice, and assess the overall contribution of the PFQ program to AHRQ's strategic goals. This evaluation addresses four key questions: 

1. What impact did PFQ project activities have on improved health care quality processes and outcomes, and on the dissemination of effective quality improvement methods? In other words, how effective were the projects in accomplishing what they proposed and what AHRQ funded?
2. Did PFQ generate partnerships and infrastructure important to sustaining change on an ongoing basis? How did the partnerships and networks created by the PFQ projects contribute to the project outcomes?
3. How adequate was AHRQ's support and oversight of the program? How well did the agency support the projects and generate synergy and collaboration across projects?
4. What contribution did PFQ make towards AHRQ's strategic goals, both through the individual projects and the program-wide activities?  

C. Evaluation Methods

To guide the evaluation, MPR developed a conceptual framework that identifies key participants, the way they are linked, and the critical questions of interest from each perspective. The framework is based on the premise that the success of PFQ in achieving its goals depends on productive interactions among four core participating groups: 1) AHRQ staff, 2) the lead grantee organizations, 3) the relevant collaborators and targets for each grantee's efforts, and 4) the coordinating activities put in place by AHRQ to foster program goals and link PFQ to AHRQ's broader quality agenda and objectives.  The evaluation framework also assumes that each actor/program component must successfully execute a set of relevant tasks, decisions, and communications for PFQ to achieve its goals.

The data for this evaluation were AHRQ and grantee documents, interviews with AHRQ staff and grantee leaders and partners, and observation of two AHRQ-grantee (AHRQ Council of Partners) meetings. The information derived from these sources was used to describe and assess the outcomes from the perspective of each set of actors and understand which factors facilitated or impeded their work. The evaluation is largely qualitative in nature. However, when grantee progress reports and self-assessments included concrete process or outcome measures of the reach and impact of their efforts, they were included in this evaluation.  

D. Major Findings

1. What Did PFQ Grantees Seek to Do?

The central focus of PFQ was to apply evidence-based practices to improve quality of health care. PFQ also provided grants to improve the health care system's readiness to address bioterrorism preparedness, although grants in this area were smaller. Of the 21 grants made initially, 18 received funds for the first purpose and 5 for the second. The latter five include two that included both components. The particular approach used by grantees varied substantially across grants, as did the conditions, settings, and populations they aimed to reach.

Clinically Focused Grants. Of the 18 grants focused on quality, 15 planned to work directly with providers (directly or through intermediary organizations) while 3 attempted to leverage purchasing power in ways that would change incentives to reward providers of high quality care. Of the 15 grants focused directly on changing provider behavior, 6 worked with hospitals, 4 with long-term care/home health providers, and 5 with office-based physicians. Most grantees planned to work through the full four-year period on interventions that were either sequenced and/or expanded to reach more providers and patients over time.  Of the 15 grants that sought to influence provider behavior, all but 3 hoped to measure changes in care processes as a way of evaluating their success. These three exceptions had less tangible aims related to the development of infrastructure and knowledge that might ultimately support improvements in quality or safety. For the most part, the three purchaser-led grants (one of which was discontinued after the first year, reducing this subset to 2 of 17 quality oriented grants) planned to gauge their success by their ability to modify incentives, rather than by the effects of changing incentives, although one pilot project in this group examined whether workers modified their choice of hospital in response to discounts for using high quality facilities. 

Bioterrorism Preparedness Grants. Bioterrorism preparedness projects typically defined their target audience more broadly than other grantees. The three projects devoted entirely to this goal sought to develop simulation models to test the utility of community response to bioterrorism threats or other vital emergencies, train practicing physicians on how to respond to threats, and assess bioterrorism readiness among provider systems in particular locales. Two grantees had dual purpose funding (quality improvement and bioterrorism preparedness) each of which had strong hospital links, which they sought to leverage in examining emergency and disaster preparedness more broadly. The bioterrorism preparedness grants did not typically include a formal evaluation component and instead proposed to judge their success by producing findings that would help to improve the health care system's ability to respond to disease outbreaks or disasters. 

Lead Organizations. Of the 20 PFQ grants that had more than a year of funding, 12 went to organizations of the type highlighted by the Request for Applications (RFA): 5 to provider-affiliated research groups, 5 to health professional organizations, one to an accreditation body, and one to an employer coalition.  Of the remaining eight, four went to independent research organizations, two to state government agencies, one to a university, and one to a private company that sells electronic medical record systems. Though AHRQ did not allow academic institutions to be the grant recipient (except in the case of bioterrorism preparedness), they could be involved in the leadership group; principal investigators based in academic institutions led six grants.

Partners and Affiliates. Consistent with the RFA, grantees proposed partnerships with a variety of organizations and individuals that could help them achieve their goals.  Both the number and types of organizations involved varied, as did their respective roles on the grants. Some partners were expected to work closely with the lead grantee on the overall leadership of the project. Instead, or in addition to this, others were chosen because of their ability to fulfill particular roles. Some were "intermediaries" who helped to recruit target organizations and create linkages for the grantee. Others were the "target organizations" themselves. Another group of partners included advisors with specialized expertise, such as clinical, health services research, or particular aspects of health delivery.

2. To What Extent Did PFQ Grantees Succeed?

For a program with limited visibility, PFQ does appear to have made a difference in health care security, quality and safety in some of the targeted health care organizations, and raised quality of care processes and outcomes for many Americans. Though final outcomes are not known yet for all projects, available results are encouraging, suggesting that some grantees made notable progress and others developed less striking, but important new knowledge. The report provides substantial detail about the projects' impact in four categories: reach, implementation, effectiveness, and sustainability as well as potential for broader diffusion. Overall results are briefly described here. 

Projects with Particularly Striking Outcomes. In terms of their ability to change clinical practice in ways consistent with evidence, four projects stand out based on the magnitude and scope of their effects: 1) Child Health Corporation of America, which improved clinical performance in several areas at 18 hospitals and has expanded quality improvement efforts at 42 children's hospitals; 2) International Severity Information Systems, which streamlined care processes in nursing facilities in ways that led to demonstrated reduction in pressure ulcers; and has launched a follow-up project to spread its approach more widely; 3) Physician Micro Systems/MUSC, which has expanded an effective strategy to get performance data into greater use in physician offices for improved process of care; and 4) the Visiting Nursing Service of New York, whose model for diabetes home care has shown positive effects and is being extended in 10 states.  

Projects Illustrating New Approaches That May Ultimately Generate Payoffs. Though less striking, four other projects developed new approaches to quality improvement that have the potential for attaining broader scope and merit greater attention:  1) the American Academy of Pediatrics, which has sustained its clinical improvement efforts through new projects that build on its practice-based, quality-improvement CME course, and has linked the approach to board certification; 2) the American College of Physicians, which had strong preliminary results in diabetes care improvement and is pursuing team-oriented CME projects in other clinical areas; 3) the AMA, which is now working with EMR vendors to integrate its performance measures into their systems; and 4) Catholic Healthcare Partners, whose work on improving heart failure care in hospitals is promising and is being disseminated nationally through the American Heart Association.

Projects That Generate Important Lessons Despite Disappointing Results. Other grants effectively pursued important areas but did not generate detectible positive improvements, though they have important lessons to share within their respective fields. For example, The Leapfrog Group's work on performance incentives may well be very important in enhancing understanding of the barriers to introducing these incentives. The Lehigh Valley Hospital and Health Network's approach to diabetes control proved it was financially feasible for primary care physicians, but little was done to replicate it beyond the 10 small practices where it was tested. Similarly, the Association of California Nurse Leaders work on falls prevention, though ultimately disappointing in its results, was important and will likely enhance support for performance monitoring in other clinical areas.  Others, like the work by JCAHO, while directed more at building knowledge than seeking immediate changes in practice, may have promise down the road in influencing care.

Bioterrorism Preparedness Project Outcomes.  Among this set of projects, the tools developed for training physicians in Connecticut were important, even though project leaders found that training had only a short-term effect on physician knowledge.  Findings from the other three bioterrorism preparedness projects may help some local health providers strengthen their plans, and produce new knowledge or tools for health system response planning, but their significance and overall contribution to the field are difficult to assess. 

Other Projects. A few grants, however, did not appear to be well-conceived from the start, even though they were well-intended. For example, the fact that nursing needs to be a focus in improving quality in nursing homes should not have been a surprise to the American Medical Directors Association Foundation. More thought could have been given to the goals and approach behind HealthFront's project, which achieved less than it originally planned. The impact of RTI's study of the science of partnerships remains difficult to evaluate.

3. What Role Did Partnerships Play in Contributing to Grantee Success in Accelerating the Translation of Research and Evidence-based Guidelines into Practice?

A key premise of the PFQ program and of this evaluation was that the success of the projects depended on effective partnerships and working relationships among the lead grantee organizations, key collaborators and target organizations or providers.  Without effective partnerships, the projects would be unlikely to achieve buy-in to evidence-based changes for improving health care quality, safety, and security. Without strong support from project collaborators and target organizations, health care improvements would be less sustainable.  

The evaluation examined the form and composition of the partnerships created in the 20 PFQ grant projects and assessed the role they played in project success and sustainability. The projects used different partnership models, most of which appeared to be appropriate to their aims and targets. The projects that set goals for changing clinical processes or outcomes were most likely to establish direct working relationships with the target organizations, and use intermediaries to provide training, technical assistance and support. In general, projects that worked closely with target organizations tended to have more tangible outcomes, as measured by the grantees' own results at the time of this evaluation.  However, it could be the scale and purpose of the projects, rather than the relationships with the target organizations, that made achieving concrete outcomes easier or harder. A few projects used intermediaries to increase the reach of the project and to sustain quality improvement activities beyond the grant period, suggesting a model that might be used when broad reach and sustainability are key goals.  

Certain characteristics and processes appear to contribute to effective partnerships in PFQ projects, based on some key themes that emerged from interviews with project PIs and their partners. These include:

• The position of lead organizations and intermediaries vis-a-vis the target organizations; professional associations and other national groups that represent the health care providers who were the targets were especially well-placed to command their respect and confidence.  
• PFQ leaders also had to have some prior experience and skill in managing partnerships to make them work effectively.
• Progress is easier when partners have a prior history of working together, though there are ways to build trust quickly without it. A participatory approach to decision making is also useful for gaining buy-in, and the involvement of target organization administrators and staff in deciding how to implement the intervention is particularly important in many situations. 
• Certain types of partners are needed to promote the sustainability and broader diffusion of an effective approach to quality improvement, who may be different than those needed for implementation at the local level.

While the PFQ projects all used varied forms of partnership as a mechanism to accelerate the translation of research into improved health care quality, safety, and security, they faced many of the same challenges confronting all efforts to diffuse innovation and change personal and organizational behavior. The most significant factors that appear to have enabled projects to overcome these challenges and make progress in meeting their goals include:

• Strong principal investigators and sponsoring organization leadership. 
• Good timing and a supportive external environment to motivate providers to use the interventions to meet performance expectations. 
• An ability to overcome provider resource constraints of competing priorities and limited time, staff or resources.
• Effective use of information technology for quality measurement and provider feedback.
• Effective leverage of AHRQ grant resources.

4. How Did the AHRQ Infrastructure and PFQ Program Components Contribute to Grantee's Success? 

The PFQ program contained several elements that sought to contribute both to the success of individual grantee efforts and to help the program achieve its overall goals. These included overall program oversight by AHRQ leadership, the PFQ program director, and the grants management office; grantee oversight and support from 10-12 AHRQ project officers over the course of the four-year program; meetings and collaborative efforts across project investigators through the AHRQ Council of Partners (AHRQCoPs) , working subcommittees, and other cross-grantee communication and networks. 

Overall Program Direction. Perhaps because of the turnover in AHRQ leadership at the start of PFQ (including the departure of a key PFQ champion) as well as competing priorities, senior executives at AHRQ do not appear to have given PFQ the kind of ongoing attention and guidance that tends to be important in shaping important projects like this. Agency leaders appear to have been more deeply invested in conceiving the PFQ program and designing the RFA than they were in providing strong leadership and support to the program once it was launched. Lack of senior leadership was particularly an issue because lead program staff were not involved in developing the program, were located relatively low down in the organization, and otherwise faced challenges in leveraging the efforts of associated PFQ project officers distributed across the many divisions and centers within AHRQ. Important program decisions, such as the content of cross-cutting collaborative activities, appear to have been made without strong guidance and input from the agency leadership, despite the recognition that the program had a novel and challenging goal.  While the program director sought to work together with individual project officers to define these parameters, critical decisions probably received less consideration and input than they could have. 

Project Officers. PFQ was structured so that AHRQ staff who functioned as project officers were the primary means of oversight for individual grants. Project officers (POs) were drawn from centers throughout the agency, one of a number of AHRQ programs that began to use this approach around that time.  PO assignments were usually but not always based on the focus and content of the grants, and appear to have been made by AHRQ management. Project officers had substantial flexibility to define their roles and the amount and kind of support they provided to each project. Some project officers, with expertise particularly matched to grantees, engaged with the projects in their portfolio frequently and substantively, providing suggestions on strategy and linking grantees to other initiatives and leaders in their fields, or helping to obtain additional funding and partners to expand their projects.  PFQ projects that received such dedicated support said this helped them to succeed. Another group of project officers provided more traditional oversight, reading progress reports and giving some feedback to project investigators, though the amount of interaction varied, with some project officers providing little or no input or support to projects. Grantees typically appreciated it when their project officers were available and encouraging. Most were disappointed if they received little feedback on reports, though some seemed to desire more interaction than others. AHRQ could do better at providing guidance to project officers, but AHRQ's structure also limits the rewards for good performance in this area.

Grants Management. For the most part, grants management appears to have operated smoothly from a fiscal perspective within PFQ.  Some grantees expressed concern over the reporting needed to support annual approval of the following year's funding. PFQ award amounts were set at the outset but re-approved annually, and grantees had to submit an annual report and justify any carry-over funds.  Because PFQ was structured as a cooperative agreement, the program director decided to require quarterly reporting, a first for the agency though now more common. The grants management office experienced problems tracking these reports that were initially submitted to project officers.  Some grantees, particularly with less AHRQ experience, found the requirements demanding and many expressed dissatisfaction with submitting reports for which they obtained little feedback.  PFQ's effort to create a database for electronic web-submission of data was unsuccessful as grantees found the web interface cumbersome and duplicative of other efforts. 

Program-wide Elements. With the goal of creating a program-wide focus for cross-fertilization, PFQ required what turned out to be twice-a-year meetings of grantees, organized into a group called the AHRQ Council of Partners (AHRQCoPs).  The Council divided the group into subcommittees on functional aspects of the projects—implementation, dissemination, partnerships, evaluation, and sustainability. While the meetings and subcommittee work were valued by some PIs, the majority of PIs expressed frustration with them, because they took away valuable funding, time and attention of the PIs from their projects and were not well-structured to foster synergy among the projects.  The AHRQCoPs and its subcommittees will be producing a set of articles, to be published in a forthcoming special journal supplement, on partnership functions and lessons. However, these activities and any learning was linked only tangentially to the work grantees sought to carry out in their projects, and hence provided limited benefits to most efforts. While the meetings sought to foster cross-grantee collaboration and some examples of this occurred, the relationships formed as a result of the AHRQCoPs meetings seem fairly similar to what one might have expected from any meeting that allowed networking opportunities. Over time, a few principal investigators either assigned responsibility for attending to junior staff or stayed for only a portion of the meeting, sometimes due to scheduling conflicts. Many PIs, however, were very enthusiastic about the work of the group.

5. How Significant Overall Was PFQ in Contributing to AHRQ's Broader Strategic Goals?

PFQ grantees clearly did not have the scale of impact originally expected by AHRQ's program developers, or promised in the RFA or the program announcement.  Such expectations were somewhat unrealistic, given the nature of the grants funded and the scale of the projects' goals, which—though not trivial—did not match original ambitions.  Yet, despite the relative invisibility of the program now within AHRQ and an infrastructure that was not very well- developed to provide all grantees with the level of support to amplify and diffuse their efforts more widely, many PFQ grantees attained substantial accomplishments, generating lessons which appear to be highly relevant to AHRQ's priority of translation of research to practice.

While the theme of partnerships has bound these projects together, it is not the only, or perhaps even the most important outcome of the program. In many projects, the use of partnerships was one of several means to an end; and a focus just on partnerships would overlook some of the most important lessons to be mined from them to inform AHRQ's strategy for closing the gap between evidence-based knowledge and actual practice in health care delivery.

In part because final results are still pending for a number of projects, little has been done to date to extract the lessons of PFQ and take advantage of the opportunities they present.  The next six to nine months (January 2007 to September 2007) is a critical period for AHRQ senior managers to consider how to leverage the lessons and results of the PFQ projects, because the final outcomes and reports from nearly all projects will be submitted to the agency during this time. AHRQ has an opportunity to reap the benefits from its earlier investment in PFQ.

However, doing so will require agency leadership and commitment of resources in a number of ways: 

• Elements of Effective Partnerships. PFQ grantee experiences and lessons can help AHRQ learn how to create effective partnerships for scaling and speeding up the translation of research into practice. Critical elements that need attention, among other things, include:  1) national organizations and individual leaders appropriate to the health care issue or topic of focus, 2) selection of well-connected intermediaries and target organizations, 3) skills and experience in partnership management, and 4) use of strategies and tools that overcome provider barriers to change. 
• Health Care Setting, Condition, or Issue-Specific Lessons. A few of the AHRQ project officers that oversaw the PFQ grants have taken the initiative to connect principal investigators and their partners to other public and private quality improvement initiatives in their specific fields.  All of the projects' results should be assessed both individually, and collectively, to identify opportunities and avenues to apply their lessons and quality improvement capacity to other AHRQ initiatives and efforts. However, not all PFQ project officers at AHRQ have the level of expertise or connections to do this. In addition, staff workloads and incentive structures do not reward staff well for this type of grant oversight.  Training and support would be valuable to help project officers maximize their contribution to grantee work within the time and other constraints they face. AHRQ should also pursue strategies to direct more attention to PFQ project results by key audiences through various dissemination vehicles that directly reach the providers and professionals in relevant fields
• New Quality Improvement Tools and Techniques.  Several PFQ projects made important advances in testing and demonstrating the effectiveness of new tools and techniques for helping providers adopt or more fully implement clinical care guidelines.  From the effective use of appropriately-scaled information technology, to the development of practice-based CME, to the integration of performance measures into electronic health records, to purchaser's design of incentive programs, the PFQ projects have important lessons to share about how these strategies can be used to help providers measure, report, and improve care quality.  While some PFQ principal investigators have already begun to translate their success into lessons for those in these other fields, AHRQ staff can provide further support for these efforts.
• Internal Agency Leadership and Support. PFQ reinforces the importance of agency leadership to the successful transition of new approaches to funding and translation work. New programs warrant as much attention over the full course of their lives—including follow up after the grants officially end—as they do in their formation.  The way AHRQ is structured makes the role of program manager very challenging, especially in programs without a "coordinating center" and sufficient staff resources, because success in this role requires skills of strong leadership and the ability to use informal support structures.  Only a small subset of AHRQ staff is likely to have these skills, and AHRQ's leadership would do well to nurture and support staff who can fulfill this role. 

In sum, PFQ generated capacity and knowledge that can support other AHRQ's efforts to translate research into practice. Harvesting its potential will further leverage the agency's $20 million investment in PFQ and enhance the strategic value of this program as an early pioneer whose experience and lessons can inform attempts to translate research to practice on a broad scale.

Return to Contents

Structure of the Report

Chapter I provides background on the origins and purpose of the PFQ program, the grant solicitation process and grants funded; and the infrastructure AHRQ created to oversee the program. Chapter II provides more detail on the evaluation approach, methods and data sources. Chapter III describes what grantees sought to accomplish in their PFQ projects and how they structured their partnerships. Chapter IV assesses the PFQ projects' accomplishments and outcomes.

The next two chapters assess the contribution to PFQ projects' successes of AHRQ's oversight and program infrastructure (Chapter V) and partnerships and other factors (Chapter VI).  Both chapters assess how these factors facilitated or hindered projects' progress and outcomes. Finally, Chapter VII contains conclusions regarding the PFQ program's overall contribution to AHRQ strategic goals, and what the outcomes and lessons from the program mean for any future efforts by AHRQ to use partnerships to translate research into practice on a broad scale.

While this report tries to identify common themes and lessons across the 20 PFQ projects, it cannot capture the richness and diversity of their experiences over the last four years.  Appendix B partially fills this gap by providing brief summaries of the 20 projects' goals, major activities, partners and partnership structure, key findings and products, and plans for continuation, where relevant.

Return to Contents
Proceed to Next Section