Chapter VI. Contribution of Partnerships and Other Key Factors to Project Success and Sustainability (continued)
C. Role of Other Key Factors in Project Success/Sustainability
While the PFQ projects all used some form of
partnership to accelerate the translation of research into improved health care
quality, safety, and security, they faced many challenges to changing
professional and organizational behavior. Below are the most significant
factors that appear to have enabled or hindered progress in the PFQ projects,
and how they tried to overcome these challenges.
1. PI Leadership
Many of the partners interviewed for this evaluation stressed
the contribution of the leadership by the principal investigators (PIs) and
others in the leadership team as a key factor in their perceived success in
implementation and diffusion. The particular qualities of leadership differed
from person to person, but they all functioned as champions in one way or
another. Some partners mentioned the PIs' energy and enthusiasm for the project
as a key factor in the success of the project, while others cited his or her
expertise in the subject matter. Several partners credited their projects'
successes to the support and ideas provided by the lead organization staff,
their willingness to work collaboratively with providers, and their flexibility
in dealing with problems that emerged. In contrast, one project partner
mentioned the PI's lack of organization as a detriment to greater success,
another said turnover in leadership slowed the project's progress, and a third
said that one of the partners didn't really play a strong leadership role,
leading to failure to launch a pilot project in one site.
However, to succeed, PIs need more than a stellar
record of research published in peer-reviewed journals. As the previous section
stressed, PIs and their leadership teams must have experience in partnership
management to structure and use them effectively. PIs that had these skills,
or could invest the time to develop them, appeared to be more effective in
harnessing their partners' contributions towards the attainment of project
goals.
2. Good Timing and a
Supportive Environment
Some projects benefited from external developments and forces
that lent their efforts greater relevance or urgency with the target
organizations. The bioterrorism preparedness projects had an initial advantage
in this regard, since memories of terrorist and anthrax attacks in September
and October 2001 were still fresh when the PFQ projects began in September
2002. The Katrina and Rita hurricanes in the fall of 2005 represented
important reminders of the need for the health care system to be prepared to
deal with emergencies, and increased interest by partners in working with
Altarum Institute's and Texas A&M University's projects.
As the drive to implement pay-for-performance and electronic
health record systems gained momentum, driven by CMS and the Office of the
National Coordinator for Health Information Technology, as well as large
national health plans and employer purchaser groups, the PFQ projects that
worked with providers to help them measure and report their performance against
national standards also gained relevance. One PI said, "Our timing for the
project was also right because the grant started just before
pay-for-performance got big, and we had it up in time before the P4P angst
started. At that time, our [physician] members were tired of the talking-head
learning experience and were ready to do something in their practices."
Another PI affirmed this sentiment: "People are cognizant of the IOM studies and
realize that we're not doing as good a job as we should be, but then people
don't know how they should be doing things differently. This project came in
and offered to show the physicians how to do it." Increased expectations for
physicians to use electronic medical records had the same constructive effect.
"It also helped that the practice sites knew that EMR was where all the big
groups were headed. It helped to have a mix of a few small sites and few big
organizations because that reinforced to the small sites that rather than being
just another academic exercise, this was where the industry was going." Such
forces help to overcome resistance to change, though they do not always
succeed. Hospitals' resistance to being held accountable for performance outcomes
blocked progress in several of the Leapfrog Group's pilot projects, for
example.
Several projects' experiences reinforce the importance
of picking the right health condition for focus. AAP was glad it decided to
focus on ADHD because "it was an easy sell—the interest was very high...
the topic had a lot to do with it, so we did not have much of a problem with
recruitment." The long-term care projects' focus on pressure ulcers in LTC
facilities, and primary care practices' focus on diabetes care benefited
because these are conditions on which providers are more likely to be measured
and reported in current or emerging public reporting systems.
3. Ability to
Overcome Provider Resource Constraints
To secure provider participation, and successfully
implement their interventions, all projects needed to overcome common barriers
confronting providers. Most health care organizations face the pressure of
limited funds, time, staff, and other resources needed to test new approaches
to quality improvement, patient safety, and emergency preparedness. Even if
they recognize its potential value, natural resistance to behavior change and
uncertainty about the benefits of new ways of working can be powerful
deterrents to adopting new practices. And even when change begins to take hold,
staff turnover at all levels can affect the pace of progress. As the following
quotes show, these issues presented enormous problems in nearly every project:
- Time and Competing Priorities. "Lack of time and
money and an overwhelmed environment were the challenges that most hindered our
progress... the practicing physicians are incredibly overwhelmed.
People do not want to take on this kind of [work] because it will increase the workload..." "The competing
priorities of the organizations were a huge barrier to trying to get anything
done. They've got so many things people are telling them they've got to
get done..." "Practices are just so busy, and even the highly motivated
practices see this as an add-on to their daily routine." "To some facilities,
this just seemed like "another project" that would take a lot of time without
being certain it would improve their quality measures." "At the end of
the day, when someone is volunteering and there are multiple demands on their
time, we can't dictate the progress they make. That's our biggest
stumbling block—that we don'thave a command and control
scenario."
- Funding. "[Although] the program was 'free' it
required them to devote staff time to something that didn't have a guaranteedreward
or positive outcome." "The business case is very difficult... there
are many hospitals where even if they wanted to do this, they can't afford to
do it upfront." "While the pot of money at the top [for bioterrorism
preparedness] looks big, by the time it gets to the states and the states divvy
the money up to their regions, there isn't much left."
- Turnover: "An inhibiting factor is turnover at the
senior leadership level. If you get turnover at the chief nursing officer
or nurse manager level, you potentially have to start over, so that hinders us
at the longitudinal level." "The turnovers are tough. The turnover
at the ______ plan caused us to lose momentum, and even though [a project
collaborator's move to another organization] was a blessing in disguise, the
project lost time because of it." "In some cases, we would get all ramped
up but then go back a month later and the person was gone." "At one
hospital, the CEO left and a new person took over who didn't buy into the
[program]..."
- Speed and willingness to change. "One of the
challenges for all agencies... was getting the nurses to change what we
wanted them to change at the speed that we wanted them to—having to
continually get people to buy-in. "... different doctors went through the
stages of change differently. Some went through the stages easily and other
took much longer. Some doctors tested us by giving us the toughest
patients first so they could see what we did with them. Eventually, when
they saw that we dealt with those patients well, they were persuaded to engage
more."
Successful
efforts to overcome provider resistance required flexibility and smart use of
available resources. For example, some projects modified their interventions
to reduce barriers to participation, or gave providers the ability to adapt the
intervention to their organizational culture or practice. By design, some
projects sought to provide more support than others, especially when their
interventions required more significant change in organization policies or
operations. While most projects overcame the challenges associated with
recruitment, they varied in their ability to provide sufficient flexibility and
support to providers, which may have affected the degree of success in achieving
project goals.
While some projects provided
intensive training and support to target organizations to implement new quality
measurement and improvement tools and techniques, other projects intentionally
limited the amount of support they offered to providers after an initial
training course, believing that more intensive follow-up support would not be
sustainable after AHRQ grant funds ran out. Examples of the latter model
included projects run by NYSDOH and AHA/HRET, which provided target organizations
with brief training courses or site visits, but had minimal follow-up, except
for collecting data for evaluation purposes. Preliminary results suggest that
the first strategy—intensive follow-up support—was more successful in making or
sustaining changes. It may be that such support enabled participants to
realize the benefits of the intervention more quickly, generating greater
commitment. However, as final results are not yet known, this warrants further
investigation.
Since staff turnover is inevitable, it is important to
learn from those projects that found ways to minimize its impact on their
interventions. The most successful projects appeared to be those that worked
with teams from organizations, rather than with one person. That way, even if
one of the team members left, the others were already on board and could train
new staff.
4. Effective Use of
IT for Quality Measurement and Provider Feedback
Projects that made effective use of information technology to
measure and motivate care process improvements had more measurable, and
possibly better, progress in improving adherence to clinical guidelines or
yielding higher scores on clinical outcome indicators. Eight projects (AAP,
ACP, AMA, CHP, ISIS, Lehigh Valley, Physician Micro Systems/MUSC, and VNSNY)
used IT-based measurement systems to give practitioners the measures and the
tools to compare their own performance with others.
When the IT systems were working well, the ability to provide
feedback on an immediate and regular basis gave providers "actionable
information" that they could use in their day-to-day patient care and practice
management, as well as strong motivation to improve if their scores were below
national standards or those of their peers. When combined with a rapid cycle
quality improvement approach, such as IHI's learning collaboratives, projects
could use the data to accelerate the testing and refinement of quality
improvement methods. For example, according to one interviewee, "there needs
to be an IT system in place for data collection... You need to be able to
do real-time data collection that will show you whether you are doing the right
thing for patients." A physician participating in one of the projects said
that success was largely attributable to "the report that we receive quarterly
100-page pamphlets with all of the graphs." Projects that worked with EMR
vendors, such as the PMSI/MUSC project, had an advantage in this regard,
"Because of the way we've developed this network and they all use electronic
records,there'sno work to get the data...."
Having available IT tools was not enough though,
unless grantees could make effective use of them. Logistical issues still
present hurdles as the AMA project discovered. "Physician practices had
difficulty getting their data into an HL-7 format to get it transferred.
Thatwas alessonon needing standards for data
transfer..." Other projects found that just making tools available on a
Web site doesn't guarantee people will access or use them, suggesting the
importance of making web-based tools more interactive and a part of the
learning/quality improvement cycle.
5. Effective
Leverage of Grant Resources
The fact that all projects were grant-funded sometimes worked
for, and sometimes against, efforts to make progress. On the positive side,
the grant funds obviously provided financial support for many activities and
infrastructure development that could not have been achieved without the grant.
"We definitely would not have been able to pay for or support the coaches...
or the hierarchical analysis without the AHRQ grant [and it] provided us with
support to establish some things that we'll be able to continue," said one
interviewee. Another said, "By giving the chapters some money, we were
providing them with a lot of infrastructure support."
Many PIs and their partners also said that the external
deadlines and deliverables associated with the grant had a salutary effect.
Several of them said that providers and partners, especially those
participating in learning communities, had more incentive to implement quality
strategies, if only to be able to report their progress at the next meeting or
teleconference. For example, said one PI, "Anytime you have a deadline, that's
helpful.You had an element of peer pressure there as well [as motivation]
to get things done in relation to this project." One of their partners
affirmed that "Having ___ hold you accountable with the conference calls
[wasa motivation to do the work]. We had other meetings and conference
calls that were held internally... which [also] helped the individual
practices stay in line." Having deadlines, said another PI, "made us report
back and provide data and say what we're doing at a level of scrutiny that
pushed us forward... the external deadlines we had... [made us] continually
focus."
On the negative side, the amount of grant funds needed to
make large-scale change was limited in relation to the overall goal. Projects
funded for clinical quality improvement projects had between $300,000 and
$400,000 for each of the four years, while those conducting bioterrorism
preparedness projects had just $100,000 for each of the four years, so it was
unrealistic to expect the 20 projects to reach millions of people as the AHRQ
RFA envisioned.
In addition, the requirement to evaluate the project's impact
led grantees to spend funds on research and data collection activities that
reduced the amount available for project infrastructure or partnership
management. Several PIs complained about the need to prepare and obtain
Institutional Review Board approval for their data collection activities.18 For example, one
said, "Dealing with IRBs was an enormous problem...in quality improvement work,
we're being asked to adhere to standards of research, but we're not really
doing research. This needs to be looked at in a big way." Others
ran into resistance from providers in submitting data needed for the
evaluation. "The data collection was always a big problem... [it was a
burden for practices and we haven't figured out how to make it easier," said
one PI.
This suggests the need to revisit how best to document
the impact of QI interventions while not running afoul of patient rights.
Whether or not grantees could have designed their work to avoid these problems
is something AHRQ may want to consider in formulating future projects of this
type.
Return to Contents
D. Lessons on Elements of an Effective Partnership for Quality
If one is planning to use a partnership to accelerate
the translation of evidence-based research into health care practice on a wide
scale, there are a few things that appear to be necessary to the success of
such an endeavor, with implications for other AHRQ efforts to fund projects
involving partnerships.
- Partnership structure. The composition, size, and
form a partnership takes should fit the goals and scale of the project. If the
goal is to make large-scale change, projects should seek intermediaries to help
with provider recruitment, training, and ongoing support for quality
improvement; and efforts should be made to build capacity of these
intermediaries to continue this work on their own over the long-term.
Partnerships should try to recruit participants who are committed to the
project and are well-connected to their peers.
- Leadership. National organizations and project
directors that have strong credibility with, and influence on the target,
should take the lead in partnerships. This affirms the importance of taking the
PI's reputation and track record into account when reviewing grant
applications. It also supports AHRQ's practice of allowing PFQ projects to
travel with the PIs when they switch employers, or transfer to different
sponsoring organizations. In the context of partnerships, though, leadership
does not equate solely with a record of scholarship and peer-reviewed journal
articles; it also means having the enthusiasm for this sort of work, as well as
commitment to, and flexibility in working collaboratively with partners.
- Partnership management skills. Leaders need skills
and experience in partnership management, and make a commitment to spend time
on forging consensus, fostering regular communication, sharing lessons, and
resolving problems at all partnership levels. Partnerships that involve all
partners in decision making and staff at all levels in the target organizations
in tailoring the intervention to their own organization may be more successful
in building commitment and sustaining activities in the long-run.
- Strategies to overcome provider constraints.
Partnerships should anticipate and prepare tools and strategies to address the
needs and constraints of providers. They should also decide in advance how
much room to allow providers to adapt the intervention so that it fits each
organization's culture, and can be adjusted to each provider's pace of change.
- Effective use of data and IT. Partnerships to
improve quality should consider seriously how best to make effective use of IT
and data collection to measure and motivate providers to make care process
improvements in "real-time".
- Regular interaction. Partnerships should organize
regular opportunities for organizations and providers to talk or meet with each
other, since the need to report progress, share successes, and learn what works
and what does not appears to accelerate providers' progress.
- Timing. If at all possible, the initiative should
be timed to take advantage of external demands on providers that make the
intervention more relevant and responsive to those demands.
This list mirrors most of the criteria that AHRQ set out in
the RFA for applicants to the PFQ program, affirming to a large extent the
assumptions and thinking that went into the program's initial development. When
one looks at the qualifications and proposals of the grantees that were originally
funded in 2002, most met the majority of these criteria.
Projects that met the PFQ applicant criteria closely and put
into practice these elements of effective partnerships appear to be most
successful in achieving their goals or those of the overall program. Projects
that did not meet the criteria as well, or were not able to apply these
elements of effectiveness, appear less successful. As a new program for AHRQ,
PFQ represented a form of venture capital, and as with all such investments, one
can expect a certain number of failures. Despite the fact that some projects
did not succeed as much as program architects may have hoped, they too have the
potential to shed insight into the challenges of doing this type of work.
18. It is unclear whether IRB approval was required by AHRQ or by the sponsoring institution for many of the
PFQ grant projects.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850