Chapter I. Background on the PFQ Program and Evaluation Goals
The Partnerships for Quality (PFQ) program sponsored by the Agency for Healthcare
Research and Quality (AHRQ) aimed to accelerate the translation of research
findings into practice on a broad scale through public-private partnerships led
by organizations well-positioned to reach end users. PFQ was one of AHRQ's
earliest efforts to structure work in ways designed to support this goal. As a
result, AHRQ is very interested in understanding what can be learned from the
experience. To support this interest, AHRQ contracted with Mathematica Policy
Research Inc. (MPR) in the last few years of the program to evaluate PFQ and
the lessons it might have for future efforts in translation.
In this first chapter of the final evaluation report, we review: 1) why
partnerships are important to AHRQ's goals, 2) the origins and purpose of PFQ,
3) the grantee solicitation process and grantees selected, 4) the
infrastructure AHRQ created to promote and oversee the success of the program,
and 5) the key evaluation objectives.
A. Relevance of Partnerships to AHRQ Goals
The Agency for Healthcare Research and Quality (AHRQ) is increasingly focused on
improving health care delivery and outcomes (Gray et al. 2003; Clancy 2004b).
In its efforts to improve quality, AHRQ engages in four types of work: research
to support evidence-based decision making; use of data to drive quality;
accelerating the pace of quality improvement; and improving the infrastructure
for quality health care (for example, informatics). (Clancy 2004a). AHRQ also
views itself as the "science partner" to the Centers for Medicare and Medicaid
Services and the states with respect to quality improvement. Collaboration is
essential, given what AHRQ's director Dr. Carolyn Clancy has termed the
"Quality Challenge," as reflected in the gap between current practices and what
we know from research to be effective (Clancy 2005). This is what commonly is
referred to as the challenge of "translating research to practice."
A critical strategy used by AHRQ to reduce the gap is to accelerate the pace of
quality improvement through partnerships with public and private sector
organizations that can move research on effective care into practice across the
health care system. Through these partnerships, AHRQ seeks to encourage the
adoption of practices that research has shown to be effective. Examples of such
partnerships include programs such as the Primary Care-Based Research Network,
the ACTION program (formerly the Integrated Delivery System Research Network),
the Put Prevention into Practice program, and the Partnerships for Quality
program, which is the focus of this study. Through these and other programs,
AHRQ seeks to strengthen its ties to organizations that are well-positioned to
reach providers and other important parties able to influence health care
delivery.
Research suggests that partnerships such as those AHRQ is investing in are critical to
enhancing the use of evidence-based practices (Greenhalgh et al. 2004). For
example, the diffusion of effective practices is more likely to occur if, among
other things, it has the support of early adopters (opinion leaders receptive
to change and well-integrated into the appropriate networks) (Berwick 2003). If
early adopters make their practices observable and gain the trust of relevant
networks that are perceived as subscribing to similar values, further diffusion
is much more likely to occur. Thus, involving key leaders who are respected in
health care or influential in its practice is vital to encouraging practice
changes that improve health care delivery.
Return to Contents
B. Origins and Purpose of the PFQ
The process through which the Partnerships for Quality (PFQ) program was developed
involved many people, including some no longer with the agency. Through the
solicitation, AHRQ was seeking to move beyond its original efforts at
translation to reach a broader set of providers and others who were
well-positioned organizationally to effectively translate research to practice.
While AHRQ had previously attempted some work of this kind through the
Translating Research into Practice Programs (TRIP I and II) and PFQ could be
considered "TRIP III," staff also viewed the two sets of programs as
distinct.
As some characterized it, the TRIP program begun in 1999 was more about
small-scale researcher-driven studies that worked with health care
organizations to determine which techniques led to effective use of research in
the delivery of care. PFQ, on the other hand, aimed to encourage change in
practice on a broad scale so that care was more consistent with emerging
research evidence. One AHRQ staff member, for example, said that while TRIP
was trying to translate research into practice, TRIP ended up funding rigorous
studies on how to change outcomes in well-defined populations and didn't have
the reach intended by PFQ, which was meant to be broader to include the next
generation. PFQ could reinforce, for example, ongoing partnerships between AHRQ
and groups like the American Medical Association (AMA) and others that might be
key to encouraging adoption—but unlikely to apply for a grant—and hence would
not otherwise have a way to work closely with AHRQ on translation.
At the time the PFQ program was developed, AHRQ's strategy was evolving from one
that supported quality improvement by funding the production of
knowledge to funding promotion of broader use of knowledge. The
development of the PFQ initiative was indicative of a change in culture; the
agency saw PFQ as the beginning of a series of projects with demonstrably broad
impacts through which the agency could show "look, we are touching America," as
one former AHRQ executive characterized it. Through the PFQ program, AHRQ
hoped to find out what could be accomplished and how sustainable it could be
after the grants ended.
Top leaders of the agency (especially Lisa Simpson, who then was deputy and Carolyn
Clancy who now is the director) say they conceived of the idea for the program
and developed a one-page summary of it that reportedly was approved at AHRQ's
Executive Management Meeting (EMM) (AHRQ's senior management group). Staff
members were then tasked to develop the concept into a Request for Application
(RFA). A senior staff member (Elinor Walker, since retired) was assigned to
write the solicitation, working with a committee of AHRQ staff set up for the
process. Staff involved in the effort said that designing the solicitation was
challenging because the goals of the program were so ambitious in relation to
the limited funds available for it.
The final RFA resulted from an iterative process between AHRQ leadership and the
RFA development committee. One AHRQ senior official described the RFA
development process as difficult and contentious. Though details reported by
participants in the process are now somewhat vague and inconsistent, we
understand that AHRQ leadership and the committee had to grapple with competing
views on a range of issues including: who to target with the solicitation
(traditional researchers versus others), whether to allow the substantive focus
to vary each year based on emerging research results from AHRQ or elsewhere
(versus maintain a single focus over the years), whether to focus on clinical
concerns only or broader strategies for quality improvement, and how to balance
the desire for nontraditional grantees who had broad reach with concerns that
such grantees were not used to working under a grant mechanism that held them
accountable for the funds and could have limited experience with evaluating
their projects. AHRQ leadership wanted internal grantee evaluations that might
help the agency show that its findings were reaching or being adopted by health
care providers nationwide—information that would be invaluable in gaining
support for funding agency programs.
As ultimately released, the solicitation for PFQ applications was designed to
encourage applicants beyond the usual academic institutions the agency had
historically funded. AHRQ wanted to fund "change agents" that not only possessed
the evidence-based knowledge to improve care but could also create the
partnerships and had the capacity to influence changes in health care
organization and delivery. The agency's desire to fund a different kind of
grantee, these "change agents," required reworking the usual processes by which
grantees were solicited, reviewed, and chosen. AHRQ barred universities from
serving as grantees, though researchers affiliated with universities might be
involved, even as project investigators.
Our ability to describe the origins of the PFQ program (or the decision process on
awards) in more strategic terms is limited by the fact that many people who
developed PFQ are no longer with the agency and many key decisions on strategy
either are undocumented or not retrievable for the evaluation. Though we were
able to interview several current or former staff involved in the program,
these interviews did not take place until several years after the program was
initiated. By then, some details were forgotten and some perceptions modified
by more recent events.
Staff turnover made it hard to cleave to the original PFQ vision. Current PFQ
experience needs to be understood in this context as do past pressures to
address other priorities after the grants were awarded. Without strong
guidance from leadership, AHRQ's ability to ensure the original program vision
and concept into the day-to-day work of program implementation was hampered, as
discussed later in the report.
Return to Contents
C. Grantee Solicitation and Awards
1. Solicitation Process
Because PFQ reflected a new program strategy for AHRQ, it required changes in the usual way
grants are reviewed. To facilitate PFQ's agenda, the agency had to ensure that
1) the RFA was different from previous solicitations, encouraging more health
system leaders to apply than in the past, 2) the review panel maintained a good
balance of academics and people with operational experience with health care
delivery, and 3) the funding committee balanced both rigor and relevance in its
funding decisions.
The PFQ request for applications, released on May 10, 2002, sought applicants for
cooperative agreements1 to
conduct projects designed to "accelerate the pace with which research findings
are translated into improved quality of care and the health system's ability to
deliver that care."2
The solicitation encouraged applicants with the capacity to influence health care
organization and delivery and evaluate the impact of their efforts, such as
health care professional organizations, accrediting agencies, practice
networks, employer coalitions, and health insurers. Academic institutions could
be one of the partners, but not the grantee. The multi-year projects had to:
- Identify high-priority areas that
are important to core audiences and for which evidence-based findings can guide
improvements.
- Translate, disseminate, and
implement evidence-based findings, with a preference for those supported by
AHRQ research.
- Annually update opportunities for
collaboration and efforts to respond to issues on security, safety, quality,
effectiveness or outcomes of care.
- Estimate the impact of
implementation efforts on policies, processes, or outcomes and stakeholders.
- Facilitate AHRQ's understanding of
research needs as perceived by diverse stakeholders relevant to the PFQ.
The solicitation described a program structure that envisioned an initial planning
phase of one year for each grant and a second phase of up to three additional
years of funding for grantees able to show potential. By the end of the sixth
month in the second phase, grantees would have to demonstrate that the aims
would be accomplished if funding continued. The solicitation also encouraged
shorter projects with more limited objectives. Budgets for the initial year
were not to exceed $100,000, with subsequent annual funding potentially two to
four times that amount. Funding for the second phase would depend on what had
been achieved in the initial phase. Each project was to include an evaluation,
as well as progress reports, at stipulated intervals. In a late modification,
AHRQ decided to expand the focus of PFQ to include projects relating to
bioterrorism. In contrast to the original grants, which were made with AHRQ's
direct funds, bioterrorism grants were funded with money AHRQ received from
other parts of the Department of Health and Human Services to address
bioterrorism readiness needs. The amount of the bioterrorism
grant awards after the first year (about $100,000 per year) was considerably
lower than the other awards under the PFQ program (about $300,000 to 400,000
per year).
We have little information on the selection process. Our interviews with staff suggest
that AHRQ succeeded only partially in its efforts to recruit a more diverse
review panel than was typical. However, there was enough diversity of the
panel to create some discomfort among those more experienced in the traditional
review panel process. For example, one participant told us:
There were people that felt that the reviewers were too researchy. I think I felt uncomfortable during a lot of the review because there was a lot of conflict, a lot of
inconsistency. For some of the reviewers, the whole emphasis of the review
would be on the research. And for some of the others, research wasn't
sufficient. I didn't feel that a great many of them gave adequate attention to
all the aspects—the feasibility, the likely value of the program, the
evaluation. My feeling is that unless you've got a decent evaluation, you
aren't going to learn much that you can use. I was kind of uncomfortable. But
again, we didn't have a mode.
The panel had to have some researchers and some doers. The researchers didn't have a lot of meat
to chew on and I was uncomfortable with what the doers were really bringing to
the table. There were a lot of arguments and in each case, you weren't sure who
to believe because you couldn't be sure what they were basing their comments
on. It's not that researchers don't have disagreements like that, but it's
generally clear what they are basing their arguments on.
Not surprisingly, differences of opinion carried over into how participants on the
panel viewed the ultimate decisions on awards. Some staff told us with concern
that AHRQ's final decision on PFQ awards did not strictly follow the ranked
technical scores. Some said that the review summary did not reflect the
panel's richer views. AHRQ leaders, however, say that adjustments between
proposals' ranking based on technical scores and actual awards are now routine
to achieve a balance in work across topic areas.
Another factor that complicated the grant selection process was that AHRQ planned to
allocate funds to PFQ's overall budget for projects focused on children's mental
health from a dedicated source. Even though several applications that planned
to focus on this area scored well, available funds were insufficient to fund
all of them. They were therefore "skipped" in order to fund projects focused
on a broader set of health issues and conditions.
Establishing an appropriate set of reviewers for grants as path-breaking for the agency as
those envisioned under PFQ must have been challenging. As one interviewee
noted, awards needed to balance rigor against relevance, with an applicant pool
other than "the usual suspects." Balancing traditional grant reviewers that
focus on the rigor of design with other reviewers looking more at operational
practicality probably was not easy. We heard, for example, one AHRQ staffer say
that some of the latter were not "objective" whereas another felt the panel
didn't have enough experience with quality improvement. It is unfortunate that
detailed documentation of the review process is not available as it could have
helped us provide more concrete feedback to the agency on lessons for future
reviews of this sort.
2. PFQ Grantees
AHRQ spent about $20.5 million on PFQ grants over the program's life, of which about
$17.6 million came from AHRQ appropriations and about $3 million from other
departmental funding (Table I.1). Twenty-two grants were made originally,
with 20 remaining after the first year. One of the 22 withdrew from the
program before it received funding and another grant was not renewed after the
first year.3 Grantees received
an initial award and then up to three additional annual awards over the
remaining period of the program. The initial grants were awarded in late
September 2002, with federal FY 2002 funds used to support work in federal FY
2003. The final fourth-year grants were made in September 2005 with an end date
of September 2006. Of the 20 multi-year grants, 14 had an end date of September
30, 2006, although some of these grantees have applied for or received no-cost
extensions so their work will continue into next fiscal year.
Table I.2 lists the 21 grantee organizations, the principal investigators affiliated with each grant, the total award, and predicted end date and status as of
September 2006. Most grantees ultimately received the full four years of
funding though funds were dispersed on an annual basis based on renewal
application. A few were for shorter periods of time, either by design or
because problems arose. Since one of the original 21 grants was terminated
after the first year, this evaluation focuses on the 20 grant projects that
continued for more than a year. We defer describing the characteristics and
focus of these 20 grantees until Chapter III.
Return to Contents
D. Program Infrastructure and Oversight
AHRQ executives said that there was not a lot of discussion in advance of PFQ on how
the program would be administered. Establishing an administrative infrastructure
was further complicated because several of the staff involved most closely with
the program in its formation would soon be retiring or were otherwise
unavailable.
The PFQ infrastructure that was ultimately established appears to be a blend of the
way AHRQ traditionally oversees grants with some program-wide elements designed
to encourage synergy across grants on issues of mutual interest.4 This infrastructure relied on internal
AHRQ staff and was not heavily resourced. The basic elements of the
infrastructure are as follows.
1. Organizational position of PFQ within AHRQ
PFQ was housed within one of AHRQs main operational centers—the Center for Primary
Care, Prevention, and Clinical Partnerships (CP-3). That was at least in part
because Charlotte Mullican, who headed the program and monitored several
grants, was located in that center, as were two project officers for six
additional PFQ grants.
Seven additional project officers who oversaw individual PFQ grants came from a
variety of centers within AHRQ. PFQ was one of the first programs in the
agency, in addition to TRIP I and II, to draw its project officers from across
the agency (rather than from a single center), reflecting the scope of the
program. Individual project officers appear to have been assigned by dividing
grants across AHRQ's line centers based on the grant focus. Specific project
officers were assigned by center directors based on availability of appropriate
project officer staff. This resulted in a matrix form of organization in which
individual project officers had lead responsibility for individual grants,
while the PFQ program director managed program-wide meetings and tasks that
would benefit from consistent efforts across grantees. Early in the program,
the program director had weekly meetings with project officers to discuss
common elements of the program and issues of mutual concern (for example,
grantee reporting requirements) though such meetings ended well before our
evaluation began.
In interviews, project officers conveyed different approaches to their oversight
tasks. From our perspective, there appear to be two different strategies taken
by project officers. The first, typically preferred by project officers with a
strong substantive interest in a given topic area, was to work closely with
their grantees to help form linkages with others involved in the same issue.
The second was what can be viewed as a more generic oversight role that focused
on overseeing adherence with grant requirements rather than seeking involvement
in the substance of the work. Project officers pursuing the first strategy
typically focused more on work with individual grantees rather than
program-wide activities, though they might do both. Regardless of strategy,
the amount of time spent by project officers on oversight varied substantially
based on their interests and competing work assignments.
2. Program-wide Structure and Elements
AHRQ desired to encourage a program-wide focus with communication across grantees.
The infrastructure to accomplish this included 1) periodic meetings of all
grantees serving as a "Council of Partners;" and 2) a Web site where materials
could be placed to foster communication. The concept behind the AHRQ Council of
Partmers (AHRQCoPs) was not well-developed in the original RFA, though grantees
were asked to include funds to attend an annual meeting.
AHRQ leadership appears to have left the decision on how to form AHRQCoPs to staff
who, we were told, decided to model it on the structure used for the
Translating Research into Practice (TRIP) grants. At the initial AHRQ CoPs
meeting, grantees were asked to elect leadership and approve a charter. The intent
was that AHRQCoPs was to be grantee run with AHRQ support. Early meetings
involved grantee presentations. Later on, the group divided into five
subcommittees perceived to reflect the main challenges shared across all
grantees: 1) science and partnership, 2) evaluation, 3) implementation, 4)
dissemination and impact, and 5) sustainability. Each subcommittee took
responsibility for structuring one of the semi-annual COP meetings and set an
agenda that addressed each subcommittee's area of interest (for example,
implementation). The meetings included a combination of speakers and time for
subcommittee work. Later on, participants on AHRQCoPs suggested that they work
together on a journal supplement that would complement their work by
documenting what had been learned about their experience. This supplement was
under active development at the end of the program. The decision to focus on
subgroups by cross-cutting challenge rather than substantive focus areas of the
grantees was made after some debate among the program director and project
officers.
In Chapter V, we provide additional details on the way AHRQCoPs functioned and how
AHRQ staff and grantees viewed it as contributing to the success of their
individual grants and the program as a whole.
Return to Contents
E. Evaluation Objectives
PFQ is a complex program involving a multiplicity of organizations and substantive
foci. AHRQ asked that the evaluation not just document the richness of the
program, but sort through the experience of diverse grantees to answer questions
of interest to AHRQ. These questions are:
- What impact did PFQ project
activities have on improved health care quality processes and outcomes, and on
the dissemination of effective quality improvement methods? In other words, how
effective were the projects in accomplishing what they proposed and what AHRQ
funded?
- Did PFQ generate partnerships and
infrastructure important to sustaining change? How did the partnerships and
networks created through the PFQ projects contribute to the project outcomes?
- How adequate was AHRQ's support
and oversight of the program? How well did the agency support the projects and
generate synergy and collaboration across projects?
- What contribution did PFQ make
towards AHRQ's strategic goals, both through the individual projects and the
program-wide activities?
In addition, AHRQ leadership expected that the
evaluation would inform internal management and operations of programs similar
to PFQ. For example, the results of the evaluation could inform the
development of future RFAs and their review, funding processes for projects
similar to PFQ, appropriate leadership structures for AHRQ programs that are
cross-center versus those owned by a single center, and the roles and
responsibilities of project officers in overseeing and documenting impact of
grantee projects.
1. Agencies
signing cooperative agreements with AHRQ are not "grantees" in the traditional
sense, since the cooperative agreement connotes a more collaborative
relationship. "Cooperative activities are intended to strengthen individual
projects and at the same time generate collaboration across projects." However,
cooperative agreements are a type of grant and in practice, lead agencies were
referred to as grantees, so we use this term hereafter.
2. AHRQ
Partnerships for Quality, Request for Applications (RFA): HS-02-010, Release
Date: May 10, 2002. http://grants1.nih.gov/grants/guide/rfa-files/RFA-HS-02-010.html
3. The
American Board of Family Medicine was approved for work with NCQA to
incorporate validated quality measures into recertification requirements for
family physicians but the application was withdrawn before funding. In
addition, the Pacific Business Group on Health received funding for one year
before mutually agreeing with AHRQ to terminate due to its inability to obtain
CMS data that was needed to implement its project.
4. Staff told us these were modeled after the formal councils that were part of AHRQ's ongoing work with the Translating Research
into Practice program in Phase I and II.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850