Chapter IV. What Did the PFQ Projects Achieve?
AHRQ sought projects that aimed to make a "significant improvement in quality of
care for a substantial part of the population of the United States. AHRQ is
seeking projects that will, in aggregate, affect the quality of care of
patients numbering in the hundreds of millions." (PFQ RFA, May 2002) This
chapter assesses the achievements of the PFQ grantees over the course of their
projects. After a brief overview of the project's overall outcomes, it reviews
the experiences and results of all 20 grants by areas of common focus.
A. Overall Outcomes
For a program with limited visibility, PFQ does appear to have made a difference in
health care security, quality and safety in some of the targeted health care
organizations, and raised quality of care processes and outcomes for many
Americans. Though final outcomes are not known for all projects, it appears
that some projects achieved better results than others (Table IV.1).
In terms of their ability to change clinical practice in ways consistent with
evidence, four projects stand out based on the magnitude and scope of their
effects: 1) Child Health Corporation of America, which improved clinical
performance in several areas at 18 hospitals and has expanded quality
improvement efforts at 42 children's hospitals; 2) International Severity
Information Systems, which streamlined care processes in nursing facilities in
ways that led to demonstrated reduction in pressure ulcers; and has launched a
follow-up project to spread its approach more widely; 3) Physician Micro
Systems/MUSC, which has expanded an effective strategy to get performance data
into greater use in physician offices for improved process of care; and 4) the
Visiting Nursing Service of New York, whose model for diabetes home care has
shown positive effects and is being extended in 10 states.
Though less striking, four other projects developed new approaches to quality
improvement that have the potential for attaining broader scope and merit
greater attention: 1) the American Academy of Pediatrics, which has sustained
its clinical improvement efforts through new projects that build on its
practice-based, quality-improvement CME course, and has linked the approach to
board certification; 2) the American College of Physicians, which had strong
preliminary results in diabetes care improvement and is pursuing team-oriented
CME projects in other clinical areas; 3) the AMA, which is now working with EMR
vendors to integrate its performance measures into their systems; and 4)
Catholic Healthcare Partners, whose work on improving heart failure care in
hospitals is promising and is being disseminated nationally through the
American Heart Association.
Other
grants effectively pursued important areas but did not generate detectible
positive improvements, though they have important lessons to share within their
respective fields. For example, The Leapfrog Group's work on performance
incentives may well be very important in enhancing understanding of the
barriers to introducing these incentives. The Lehigh Valley Hospital and Health
Network's approach to diabetes control proved it was financially feasible for
primary care physicians, but little was done to replicate it beyond the 10
small practices where it was tested. Similarly, the Association of California
Nurse Leaders work on falls prevention, though ultimately disappointing in its
results, was important and will likely enhance support for performance
monitoring in other clinical areas. Others, like the work by JCAHO, while
directed more at building knowledge than seeking immediate changes in practice,
may have promise down the road in influencing care.
In
the area of bioterrorism preparedness, the tools developed for training
physicians in Connecticut were important, even though project leaders found
that training had only a short-term effect on physician knowledge. Findings
from the other three bioterrorism preparedness projects may help some local
health providers strengthen their plans, and produce new knowledge or tools for
health system response planning, but their significance and overall contribution
to the field are difficult to assess.
A
few grants, however, did not appear to be well-conceived from the start, even
though they were well-intended. For example, the fact that nursing needs to be
a focus in improving quality in nursing homes should not have been a surprise
to the American Medical Directors Association Foundation. More thought could
have been given to the goals and approach behind HealthFront's project, which
achieved far less than it originally planned. The impact of RTI's study of the
science of partnerships remains difficult to evaluate.
Return to Contents
B. Outcomes of Projects Seeking to Change Clinical Practice
The concepts of the RE-AIM evaluation framework—reach, effectiveness, adoption,
implementation and maintenance—are particularly relevant to assessing the
impact of the 17 PFQ grants seeking to affect clinical quality of care.12 The RE-AIM framework is oriented toward assessing the potential for translating research to practice, and for
wider dissemination. While this framework can be used to assess interventions
at both the individual and organizational levels, in this evaluation we focus
on the PFQ projects' effects at the organizational level, since the PFQ
projects were intended to scale up proven health care interventions already
demonstrated as effective for individuals. This section assesses 17 PFQ
grantees' impacts in the RE-AIM framework domains relevant to these projects—reach, implementation, effectiveness, and maintenance/sustainability.13
1. Reach
When it announced the original 22 projects to be funded, AHRQ stated that they would "involve more than 88,000 medical providers; 5,800 hospitals, nursing homes, and other health care facilities; and 180 health plans."14 Although these estimates were based on
overly optimistic predictions at the start of the program, PFQ did not achieve
short-term effects on the delivery system on this scale.
The number of organizations targeted ranged widely across the PFQ projects, even
among those targeting the same type of organizations. (Go to Appendix Table A.4
for a visual display of the number of organizations, patients, or other targets
chosen by each project.) For instance, in projects targeting hospitals for
their interventions, the number initially targeted ranged from just a handful
(Catholic Healthcare Partners) to between 10 and 40 (CalNOC and CHCA) to 100
(AHA/HRET, Leapfrog) Among those targeting nursing homes, the number targeted
ranged from 8 (ISIS) to 30-50 (NYS-DOH and AMDA). In projects targeting
physician practices, the number ranged from 8 (Lehigh Valley) to 10-35 (AMA
ACP) to more than 100 (PMSI and AAP).
Projects
meeting or exceeding planned reach/participation. Among 17 projects that
specified the number of target health care delivery organizations, physician
practices, or other local partners they planned to recruit for an intervention,
14 enlisted at least the number of entities projected in their original
proposals. This is not an insignificant accomplishment, since few of the
projects paid provider organizations anything for participating other than
nominal fees to offset the cost of data collection or travel to project
meetings. The only participation incentives project leaders could offer were
the free training or technical assistance to improve care quality, and in some
cases, the opportunity to learn from others.
Some
projects had low targets, so they attained them easily. For example, ISIS
enrolled 12 nursing homes, VNSNY enrolled 8 home health agencies, and Catholic
Healthcare Partners recruited 6 hospitals. Other projects set substantially
higher targets, but still met them. For example, the PMSI project, conducted
with the Medical University of South Carolina, expanded the number of primary
care practices participating in its performance measurement system from 40 to
about 100. Recruiting the practices was part of the PMSI's regular operations,
and participation was relatively easy for provider practices, once they
purchased the electronic medical record system sold by PMSI. ACP met its target
of about 35 physician practices for its team-oriented, practice-based CME
training programs, which required practices to send 3 staff members out of the
office to participate in training, implement workflow redesign in their
practices, and submit data regularly.
Some
projects had to revise their recruitment or research design strategy to reach
their target. For example, when AMDA realized that the best way to gain
nursing facilities' participation was by persuading the Director of Nursing,
rather than the Medical Director, it switched its focus. AMDA also loosened
its participation criteria and allowed "rolling" enrollment, rather than all at
one time. Even Catholic Healthcare Partners initially had a hard time
recruiting its own hospital CEOs to participate in its program, when they
couldn't see "clear hospital revenue and profitability gains." They overcame
the CEOs' resistance by asking the system's ultimate decision makers—the nuns
who govern the system—to persuade the CEOs to cooperate.
One
project far exceeded the participation target it had originally projected. In
the third year of its four-year project, CHCA significantly expanded the number
of hospitals eligible to participate in its QI efforts from the original 14
CHCA participating hospitals to all 42 member hospitals. This expansion
occurred in part because non-participating sites realized the value of the
quality improvement efforts and early PFQ interventions, which coincided with
member hospital CEOs recognizing that QI was not just something for the quality
department; rather that "quality was the business they were in."
Projects falling
short of planned reach/participation. Three PFQ projects did not recruit
the targeted number of participating organizations, primarily due to difficulty
in overcoming barriers to provider involvement. For example, the American
Hospital Association–Health Research and Education Trust (AHA-HRET) sponsored a
project that worked with seven hospital-based palliative care units to offer
on-site visits and support to other hospital teams wishing to develop or
enhance their own palliative care units. This project found that even when
most program costs were subsidized, the difficulty of making the business case
to hospital administrators dampened interest.
NYS-DOH
did not recruit all of the adult care facilities it planned to participate in
its training program, largely because these organizations are not required to
provide staff training and resource problems make it hard for them to spare
staff to participate. Long-term care facilities, especially those that are
small, appear to be less willing or able than hospitals to take on any "extra"
activities, particularly when the incentives or rewards for doing so are
long-term or uncertain. The Connecticut Department of Health/Yale New Haven
Health System found it very hard to persuade physicians to take its
bioterrorism preparedness course, and as a result did not expand the effort to
target other groups of professionals or to hospitals and practitioners in other
parts of the state as originally intended.
2.
Implementation of the Intervention Model/Strategy
Implementation
in the RE-AIM framework refers to the fidelity to the core elements of an
intervention protocol, that is that they are implemented consistently with the
design or model. In this evaluation framework, the question of fidelity is
framed as whether the intervention was delivered as intended. While most
grantees were successful in this regard, a few encountered problems that
required they modify original plans and adapt models.
One
of these, the American Medical Association project, had to change its strategy
significantly from one that planned to test and compare two models for
collecting data from physicians on performance measurement, to a focus on just
one of the models. This change occurred after the groups involved in testing
the so-called "community model" for collecting data from payers encountered
resistance to sharing data on physician quality measures. The project shifted
gears to focus exclusively on the "practice model," in which physicians transfer
data electronically to a central data repository. In making this change, AMA
expanded beyond its original focus to invite a variety of physician practices—a
large specialty group, a university-based outpatient group, and a
publicly-sponsored ambulatory care network—to test the model and help it learn
how different types of electronic health information systems could be adapted
to export data for measuring performance against AMA-developed standards.
Also
encountering operational constraints, the New York State Department of Health
reduced the number of best practices it expected nursing homes and adult care
facilities to implement to make it easier for them to participate and increase
their ability to train staff in the best practices.
HealthFront
also encountered operational problems that challenged the original project
concept. Originally hoping to develop a nationally recognized provider
performance measurement system, the grantee decided to focus more intensively
on supporting purchaser capacity in two markets (Minneapolis and Colorado)
after one of the key partners had to withdraw. Key partners in these markets
had competing obligations; they supported the work of the grant but couldn't
provide the fast response originally assumed. As a result, this project
transitioned into a strategy focused more on generating information on how
financial incentives to doctors could be aligned and how providers perceived
incentives than its original focus on introducing these incentives over the
course of the grant.
Use
of IT to support quality improvement. While nearly all PFQ projects
collected data from target organizations to track progress and evaluate
outcomes, three projects (AAP, ISIS, PMSI) sought to introduce new information
technology into facilities or provider practices as a tool for quality
improvement or quality measurement. Several others (AMA, Lehigh Valley, ACP,
CHCA, VNSNY) collected data from providers and used third parties to deliver
timely reports to provider organizations to provide frequent feedback on the
success of quality-related efforts.
Most
solved the difficulties of incorporating the new technology or data collection
and reporting tools into daily workflow. But some ran into problems that
slowed their progress or caused them to make significant shifts in strategy.
For example, the American Academy of Pediatric's intervention relied on
pediatricians' use of a new online tool for reporting care processes, called
eQIPP. When the PFQ project began, this tool was still new and not completely
reliable. The American College of Physicians found that the data coordinating
center it used was slow to produce results needed by the participating
physician practices to assess changes in their patients' clinical indicators.
Adapting
interventions to each participating organization/group. Several projects
found it challenging to identify essential elements of their intervention
versus those that could be modified to adapt to each organization's culture, IT
infrastructure and staffing patterns. For example, RTI's project found that
many health care innovations are complex and have multiple elements, but
evaluations of their effectiveness do not distinguish between elements that are
required or optional. ACNL/CalNOC's project allowed each hospital to select
which evidence-based practices to implement to reduce hospital-based falls, but
when its results did not show a significant reduction in hospital falls or
falls-with-injury, variation in the interventions may explain the lack of
impact.
3. Effects on
Health Care Delivery Processes and Clinical Outcomes
Of
the 17 grants focused on health care quality or patient safety, 12 set
measurable goals related to change in clinical practice or outcomes. Of these
12, 8 had preliminary results to report by September 2006.15 Go to Appendix Table A.5 for a brief
summary of all projects' preliminary outcomes. All but one of the eight
detected some improvement in the measures examined, suggesting the majority
were at least somewhat successful. However, the magnitude of the changes is
not consistent across measures and in some cases, is difficult to assess from
the information provided by project staff.
- The American College of Physicians examined process of care
measures, such as eye and foot exams and flu vaccines, and clinical outcome
measures, such as blood pressure, LDL below recommended levels, and so on among
patients with type 2 diabetes that were tracked in 35 physician practices
participating in the team-oriented, practice-based CME program. Early results
from a four-practice pilot program showed
that 75 percent of patients' blood pressure scores improved from baseline, and
an average of 3.6 new patients participated in group sessions each month.
- Association of
California Nurse Leaders/CalNOC tracked data reported to the California
Nurse Outcomes Coalition data repository before and after interventions in
about 90 participating medical-surgical units in 32 hospitals to reduce falls
and fall-related injuries, compared to 260 non-participating units in the same
hospitals. Pre-post data analysis found mean change in falls and mean change
in falls with injury were not significantly different between participating and
non-participating units. While the falls per 1,000 patient days in
participating units decreased slightly after the intervention, project
researchers are trying to determine if the lack of a statistically meaningful
difference is due to improved reporting, widespread attention to falls due to a
JCAHO focus during the intervention period, or the interventions not having
sufficient impact on a relatively rare event.
- When PFQ began, Catholic
Healthcare Partners already had a system to report quality of care
processes for treatment of heart failure patients via MIDAS, a proprietary data
warehouse for hospital benchmarking. It collected data on ACE inhibitors
prescribed at discharge, left ventricular function assessment, smoking cessation
counseling, and appropriate discharge instructions. The PFQ project, called
Heart Failure (HF) Guidelines Applied in Practice (GAP), aimed to attain a
score for each of the four measures at or above 75 percent of all HF patients
or the top 25th percentile in MIDAS, whichever was greater. It also
set an organization-wide goal of reducing 30-day all-cause readmission rates
for patients with an HF admission. About 18 months after implementing
interventions in six hospitals, preliminary results indicate that patients
under the care of HF advocates experienced a 41 percent drop in readmissions,
and almost a doubling of the period between readmissions.
- Child Health
Corporation of America (CHCA)'s quality improvement strategies focused
on several areas, including hospital patient safety, medication safety and pain
management, and initiated many QI projects involving different subsets of CHCA
member hospitals. One of the most successful projects involved an effort to
reduce adverse drug events (ADEs) related to narcotics. Over an 18-month
period, the 18 hospitals participating in this project showed a 49 percent
decrease from 39.1 to 17.1 ADEs per 1,000 narcotic doses. Another successful
project focused on reducing bloodstream infections by implementing best
practices in 29 hospitals. The results showed 57 percent improvement in
infection rates for 18 of the 29 hospitals, a drop in bloodstream infections
from 6.9 to 4.8 per 1,000 line days for all 29 hospitals, and 88 percent
compliance with IHI and CHCA-created "best practice" guidelines.
- The International
Severity Information System (ISIS), whose PFQ project streamlined
nursing facility documentation of patient care processes, tracked operational
measures related to interventions and clinical care measures for pressure
ulcers. Seven facilities that implemented interventions starting in April 2005
reduced the number of high-risk patients with pressure ulcers by 33 percent.
Pressure ulcer prevalence in participating facility units dropped over the
project period to 8.7 percent on average, compared to the national average of
14 percent, which remained flat over the life of the project. Facilities that
implemented the interventions more completely, such as regularly submitting
care process forms and using the reports in care planning meetings
had better results—pressure ulcer prevalence of about 5-6 percent—than those
that partially implemented the interventions.
- Lehigh Valley
Hospital and Health Network (LVHHN), which provided a package of
educational interventions to physicians and patients to improve care of type 2
diabetes patients, monitored process of care measures and clinical lab scores
for selected patients in participating primary care physician offices at
baseline, six months and 12-months post-intervention. About 18 months after the
start of the project, it reported improvements in the percent of physicians
screening for glycosylated hemoglobin (HBA1c) and lipids (but not
micro-albuminuria) in a timely manner relative to ADA guidelines. Patients also
showed progress in adherence to recommended practices and statistically
significant improvements in blood pressure, lipid levels, cholesterol,
triglycerides and hemoglobin.
- Physicians Micro
Systems, Inc. (PMSI)/Medical University of South Carolina (MUSC) sought
to improve adherence to clinical guidelines for more than 70 indicators in
eight sets of medical conditions, including heart disease/stroke, diabetes,
cancer screening, immunizations, respiratory disease, mental health and
substance abuse, nutrition and obesity, and drug prescribing for the elderly.
Participating practices all used PMSI's electronic medical record system, which
made it easy to extract data and generate quarterly reports. MUSC staff and
consultants provided educational services and support to physician practices on
clinical guidelines in each area. Preliminary results indicate statistically
significant improvements in the summary index measure for the percent of
eligible targets met in the 78 indicators, rising from 33 percent at baseline
(9/02) to 46 percent three years later. According to the project investigator,
the results are not as large as they could have been if the project had focused
on a smaller number of practices and fewer quality indicators.
- Visiting Nurse
Service of New York (VNSNY) worked with eight home health agencies from
around the country on its first phase of quality improvement efforts, focused
on care for diabetic patients. Each agency submitted monthly data from chart
reviews on clinical measures related to glycemic control, foot care, and
medication management. The proportion of people with diabetes receiving a
comprehensive foot exam by a nurse within 10 days of admission to home care
increased more than 50 percentage points over the course of the project. Also,
patients with blood pressure in their target range most or all of the time
increased 30 percentage points, with similar increases in patients who received
and an individualized glycemic control plan, foot care education and a review
for medications with possible contraindications. The second phase of the
project, which focuses on reducing hospitalization in home care patients, has
preliminary data suggesting a drop of 2.5 percentage points for the 70 home
health agencies.
4. Effects of
Projects Focused on Infrastructure and Learning
Among
the 17 projects that were trying to improve clinical quality of care, three
that focused on health care providers (AMA, JCAHO, RTI) and two that focused on
purchasers (The Leapfrog Group and HealthFront) had goals that could not be
measured quantitatively. As mentioned in Chapter III, only two of these five
projects—the AMA and The Leapfrog Group— tried to formally evaluate their
success, so we have limited ability to judge the effects of the other three
projects.
Of
the three provider-based grants focused on infrastructure and learning, two
involved major national organizations (AMA and JCAHO). AMA's work to examine
electronic transfer of data for performance measurement had, sponsors say,
important lessons about the practical issues and challenges to data extracting
exporting and validation. With CMS and others calling for the introduction of
performance measures for physicians in office-based practice, these findings
have the potential to be very important. JCAHO's work involved a survey of
hospitals about their perceptions of the value of performance measures, as well
as a comparison of self-abstracted data on performance measures with data
abstracted by third parties. They found that the self-abstracted and
third-party abstracted data is essentially similar, which may help build
confidence that hospitals' own data is reliable enough to use in pay-for-performance
systems.
Among
purchasers, The Leapfrog Group worked with purchasers in six markets to
encourage use of quality information in selecting hospitals. Though Leapfrog
sought to evaluate the effects of these efforts, only three of its six pilot
projects were implemented and evaluation results were available from only one
of the pilots for this report. That pilot involved a differential patient
co-payment to encourage use of hospitals meeting Leapfrog's quality and patient
safety practices. Preliminary results show no effects on choice because
physicians' admitting privileges appear to play a stronger role in influencing
patients' hospital selection. Leapfrog continues to evaluate these efforts and
says that it has gained valuable experience in establishing pay-for-performance programs.
There was no information on impacts of the projects led by RTI and HealthFront,
although HealthFront reports that stakeholders in the two markets it targeted
have been interested in the results from surveys of providers' perception of
incentive and reward programs.
12. RE-AIM
is a "systematic way for researchers, practitioners, and policy makers to
evaluate health behavior interventions. It can be used to estimate the
potential impact of interventions on public health," according to its developers.
For more information, go to http://www.re-aim.org/index.html and Glasgow, et al.,
1999. AHRQCoPs Subcommittee on Dissemination and Impact also found the RE-AIM
framework useful in examining the impact of three PFQ projects.
13. For
example, in the RE-AIM framework adoption refers to the percentage and
representativeness of the sites or providers that agree to participate. The
representativeness of the participants is important because the results cannot
be generalized or may not be broadly replicable if those who participated are
more motivated or ready to change than those who did not. This is difficult to
assess in the PFQ projects. Because these were applied research projects,
virtually none of them randomly selected organizations to participate. A few
projects tried to compensate for this by randomly assigning those who agreed to
participate to an experimental or control group, or to one or another
intervention. A few stated that they tried not to recruit those who were
innovative or best-in-class, but they were not able to verify this with any
data. Thus, this analysis does not address adoption.
14. Partnerships for Quality. Fact Sheet. AHRQ
Publication No. 04-P004, March 2004. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/qual/partqual.htm
15. Among the four projects with clinical practice or outcome goals whose results are not yet known (AAP, AHA-HRET, AMDA, NYS-DOH), one has indicated it expects positive impact, but implementation delays and problems with the other three indicate that they may not have as positive results to report as those in the eight projects with preliminary findings.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850