Chapter VII. How Significant Overall Was PFQ in Contributing to AHRQ's Broader Strategic Goals?
A. Accomplishments and Limitations
From our perspective, PFQ was reasonably successful as a
grant program taking into account the varied objectives of the diverse grantees
that were funded. Most grantees did what they said they would, although the
overall impact of all 20 projects was not as fully realized as AHRQ program
initiators had hoped.
PFQ had a core set of 12 grants focused on directly changing
clinical practice and outcomes, at least 8 of which already have some evidence
of positive outcomes. Most of these efforts produced sustainable changes in
day-to-day practice that will enable and foster regular quality monitoring and
continuous quality improvement in nursing homes, primary care physician
offices, hospitals and home health agencies. While five other projects had
goals that also focused on improving clinical quality and outcomes but stopped
short of trying to directly change practice, they did generate valuable lessons
about how to provide an infrastructure and set of financial incentives for such
efforts. The bioterrorism preparedness grants, whose goals were to improve the
health system's ability to respond to emergencies, also appear to have
generated valuable knowledge.
For a pioneering program, these accomplishments are
impressive. They provide a foundation of learning that AHRQ can build on for
improving the safety, quality and effectiveness of health care delivery. The
partnerships created have leveraged resources from national and community-based
organizations for promoting improvement, and forged stronger linkages between
researchers and those on the front line of health care delivery.
While relatively successful on these metrics, PFQ had
some shortcomings. First, a few of the grants probably were not, with the
benefit of hindsight, well-conceived originally, despite their best
intentions. Second, PFQ grantees did not have the scale of impact originally
expected by AHRQ's program developers, or promised in the RFA and the program
announcement.19
While the grantees' interventions reached a meaningful number of providers,
they clearly reached fewer than one would expect solely by the membership of
major organizations involved with PFQ (e.g. AMA, AHA etc). However, those
initial expectations on the part of AHRQ were probably unrealistic, given the
nature of the grants funded and the scale of the projects' goals. Third, this
evaluation suggests that PFQ's efforts to promote collaboration and mutual
learning across PFQ grantees through AHRQCoPs and other cross-grantee work was
not very successful in supporting grantees, though it may generate some useful
publications.
Return to Contents
B. Future Opportunities and Lessons
Particularly because PFQ was an early initiative to support
one of AHRQ's current priorities—transforming research into practice—the formal
ending of the PFQ program provides an important opportunity to harvest lessons
that may be valuable to AHRQ for the future. While AHRQ could expect some
failures in a program that aimed to encourage innovation, AHRQ can learn from
its experience on how it managed the PFQ program and apply the lessons to
current and future initiatives designed to translate research into practice,
and to use partnerships to extend the reach of its quality, safety, and
security improvement efforts.
To date, little has been done to extract the lessons
about what worked well and take advantage of the opportunities they present.
The lessons learned about what did not work are equally useful. The initial
lessons and findings presented in this report can help AHRQ achieve many of its
goals. We review here four of the most significant lessons and insights from
PFQ, and offer several avenues for AHRQ to apply the lessons to its current
priorities.
1. Elements of
Effective Partnerships for Translating Research into Practice
PFQ grantee experiences and lessons can help AHRQ create more
effective partnerships for bringing to scale and speeding up the translation of
research into practice. Critical elements of effective partnerships, listed at
the conclusion of Chapter VI, include: national organizations and individual
leaders with expertise and well-regarded reputations in the health care issue
or topic of focus, selection of well-connected partners at all levels—grantees,
intermediaries and target organizations—and strategic use of each one's
resources and connections; skills and experience in partnership management; and
the ability to overcome provider barriers to change.
Partnerships, and how to promote them, are important to many
AHRQ programs. For example, the concepts have immediate relevance to the ACTION
program, AHRQ's latest effort to use "field partnerships" to translate research
into practice. In fact, several PFQ grantees are participating in ACTION
networks, offering an opportunity for them to share their own lessons in
partnership building with the collaborators in each network. But other ACTION
grantees also would benefit from knowing more about the collective experience
and lessons from the PFQ projects to gain insight in fostering teamwork and
partnerships. Other AHRQ programs involving partnerships, such as the 17
projects funded by the Partnerships in Implementing Patient Safety (PIPS), may
also benefit from learning more about the PFQ experience and lessons.
AHRQ managers and staff also participate in a wide
variety of partnerships with other federal agencies and private health care
organizations, from work on CAHPS to leadership of the AQA-HQA efforts to
develop standardized performance and quality measures. Lessons about effective
partnerships are potentially applicable to AHRQ's work in these other efforts
as well. A work group within the agency might be created to distill the lessons
on effective partnership management from these initiatives, and determine how
they could be applied to strengthen AHRQ's existing and future partnerships and
programs.
2. Leveraging AHRQ's
Internal Resources to Help Translate Research into Practice
The PFQ program provides good examples of the way an
effective project officer can help leverage the work of grantees.
Specifically, those AHRQ project officers that brought with them connections
and deep knowledge of particular issue areas took the initiative to connect
principal investigators and their partners to other public and private quality
improvement initiatives in their specific fields. With AHRQ's focus on
portfolios, such support for grantees creates potential synergies across
programs and connections between similarly focused grants that may be conducted
under diverse auspices. Although not all project officers have such skills, it
may be valuable to encourage AHRQ staff to think more creatively about how best
to use the knowledge and enthusiasm they bring to help leverage the work of individual
grantees.
AHRQ also may benefit from a more careful review of findings
in particular topic areas, with a view towards forming tighter connections with
other AHRQ initiatives and efforts in the same areas. For example, the results
of the three PFQ projects that focused on long-term care could be examined to
determine how their results could be leveraged with current quality initiatives
in the long-term care field. AHRQ staff with expertise in diabetes prevention
and control could examine the results of the five PFQ projects that focused on
control of diabetes in primary care or home health settings to assess
opportunities for spreading effective approaches more broadly. They might also
help the PIs of those projects connect with leaders in the diabetes prevention
and control field. The experience of the purchaser-led grants similarly
provides important input on the factors that promote or inhibit purchasers from
leveraging their influence to promote change in quality incentives and care
within communities.
AHRQ could widely share the synthesis of findings and
lessons in this evaluation, supplemented by final results from PFQ grantees
that will be produced by September 2007. Aside from this report and
publications by individual grantees, there are currently no other ways to
easily obtain information on what the PFQ program was and what grantees
accomplished. The program remains relatively invisible, a belief shared by
grantees and many of those directly associated with the program at AHRQ. AHRQ
staff in various parts of the organizations should consider how best to
translate the results of the most promising projects to relevant providers and
professionals in the field.
3. Appropriate Use of
Quality Improvement Tools and Techniques for Translating Research into Practice
Several PFQ projects made important advances in testing and
demonstrating the effectiveness of new tools and techniques for helping
providers adopt or more fully implement clinical care guidelines. They include
the effective use of appropriately scaled information technology, the
development of practice-based CME, the integration of performance measures into
electronic health records, and the design of quality reward and incentive
programs by purchasers. National and local quality improvement leaders wishing
to replicate these strategies on a bigger scale can draw on the lessons of the
PFQ projects. While some PFQ principal investigators have already begun to
translate their success into lessons in these other fields, AHRQ staff can provide
further support for these efforts.
To take one example, several PFQ projects made
important advances in introducing information technology to health care
facilities or to individual physician practices to aid in tracking adherence to
clinical guidelines or to performance standards. The ISIS-sponsored PFQ project
is a featured case study in the AHRQ National Resource Center on Health
Information Technology20 and leveraged its success in the PFQ project to obtain new funding under AHRQ's
Transforming Healthcare Quality through Information Technology (THQIT) Implementation
Grants. Other PFQ projects have had some success as well, but efforts are
needed to bring them to the attention of experts in the HIT field, so their
lessons or implications for IT development in particular settings can be more
carefully assessed. Assessment of the PFQ project results might also be
performed to determine if they should be included in AHRQ's new Innovations
Clearinghouse.
4. Future Design of
Programs for Translating Research into Practice
PFQ provides valuable insights about the importance of
agency leadership and program structure to the successful transition to new
approaches to funding and translation work. From our review of the PFQ
experience, we suggest several lessons important to the success of future
programs seeking to translate research to practice.
- PFQ highlights the importance of senior leadership guidance on
refining program strategy over time, not just when new programs are being
conceived.
- The selection and placement of program directors is important.
AHRQ can do a better job of considering explicitly the structural constraints
associated with the program director's role in order to pick and position
individuals to increase their effectiveness in working with staff across the
agency, and in communicating with top leadership. AHRQ's current structure
makes the role of program director in a cross-center program like PFQ very
challenging. Since center directors are not held accountable for the program's
success, responsibility rests with the program director to marshal the
necessary resources, guide and motivate project officers to oversee grantees,
and maintain commitment to the program's vision and goals through staff
turnover and changing center and agency-wide priorities. The challenges are
similar for project officers who get little support for actively supporting
grantees in a cross-center program. Consideration might be given to adjusting
staff workloads and incentive structures to reward staff for this type of grant
oversight work.
- Agency managers need to think through more clearly their
expectations for cross-grantee work. While some PIs valued the AHRQCoPs
meetings and subcommittee work, the majority of them expressed frustration with
the meetings, because they took away valuable funding, time and attention from
their own projects and were not well-structured to foster synergy among the
projects. AHRQCoPs and its subcommittees are producing a set of articles on
partnership functions and lessons, to be published in a forthcoming special
journal supplement. However, these activities and any learning they produced
were linked only tangentially to the grantees' work and hence provided limited
benefits to most of their projects. If AHRQ expects grantees in a program to
work collaboratively, the final products should be more clearly defined and
communicated to applicants in advance, and the agency should make clear to
prospective grantees the amount of time and effort this activity will require.
- AHRQ needs to better match grantee selection criteria to the
goals of cross-grantee work. The PFQ grantees were too heterogeneous to foster
significant collaboration, particularly without a strong content or focus that
was relevant to all their needs. Programs like PFQ that seek to attract
well-connected national and regional organizations whose base is outside of
research also need to appreciate better the demands on the time of these
individuals, which may mean limiting reporting requirements and collaboration
work to the essential core.
In sum, PFQ generated capacity and knowledge that
can support broader AHRQ's efforts to translate research into practice.
Harvesting its potential will further leverage AHRQ's $20 million investment in
PFQ and enhance the strategic value of this program as an early pioneer whose
experience and lessons can inform attempts to translate research to practice on
a broad scale.
19. Partnerships for Quality. Fact Sheet. AHRQ Publication No. 04-P004, March 2004. Agency for Healthcare
Research and Quality, Rockville, MD. http://www.ahrq.gov/qual/partqual.htm
20. "Long-Term Care Facilities Embrace Health Information Technology," located on the AHRQ HIT National
Resource Center Web site: http://healthit.ahrq.gov; select AHRQ-Funded Projects, and then on Case Studies.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850