Chapter IV. What Did the PFQ Projects Achieve? (continued)
C. Outcomes and Findings from Bioterrorism and Emergency Preparedness Projects
Four
of the five projects that aimed to improve the health system's preparedness for
bioterrorism events and other emergencies had findings to report from their
studies or modeling exercises in time for this evaluation.16 It is inherently difficult to measure
the utility of these findings in the absence of real events or disaster
response exercises that show whether and how health care providers and public
health officials actually use the information to prepare and implement plans.
For this reason, the utility of the findings is based on the perceptions of
project staff. The one exception is the Connecticut Department of Public Health
project that included a formal study of the effectiveness of the training
provided through their PFQ grant.
- Altarum
Institute, which used two models to simulate the flow of patients into
health care facilities in the event of smallpox and other disease outbreaks,
provided information to public health officials in the San Antonio area, which
they say helped them accurately
estimate the number of smallpox vaccinations and distribution sites needed to
control an epidemic. The information was also used to develop a purchasing
strategy for bioterrorism preparedness supplies.
-
The Connecticut Department
of Public Health/Yale New Haven Health System project's online
training program for front-line physicians showed that it effectively increased
the knowledge of those who took the course; but six months later, their exam
scores declined almost to their pre-test scores. Project investigators
speculate that since physicians have no opportunity to use the information, it
quickly dissipates. Annual training or drills may be needed to retain the
information.
-
One of two studies conducted by
the Joint Commission on Accreditation of Healthcare Organizations
(JCAHO) under the PFQ program focused on the existence and effectiveness of
linkages for community-wide bioterrorism preparedness among health care
organizations, and public health, public safety and other governmental
agencies. According to the article that published the results (Braun, et al.,
2006), while the majority of hospitals conducted drills or exercises, had plans
to acquire additional supplies or equipment, and were prepared for
decontamination needs, only 40 percent had 24-hour access to a live voice at
their local health department. The survey's list of 17 elements of an
effective emergency preparedness plan is regarded as a useful checklist for
hospitals.
-
Texas A&M University
System Health Science Center
conducted a number of studies on factors affecting bioterrorism and emergency
preparedness. A case study of federal bioterrorism funding allocation in the
San Antonio area showed the importance of formal and informal communication
networks throughout the region. A study of disease surveillance and reporting
systems on the U.S.-Mexico and U.S.Canada borders showed that communication
infrastructure at the local level needs to be improved; that funds should be
targeted to disease surveillance methods with the greatest potential for
mitigating disease burden; and that bi-national organizations are needed to
overcome the problems created by the existence of public health bureaucracies
in three national governments, dozens of U.S. states, Mexican states, and Canadian
provinces, as well as numerous county and local jurisdictions.
Return to Contents
D. Sustainability and Broader Diffusion of Project Activities
In the RE-AIM
framework, sustainability is called "maintenance," and it means the extent to
which a program or innovation becomes institutionalized in organizational
policies and practices. Both sustainability and broader diffusion were
important goals for the PFQ projects. AHRQ's RFA for the program expected
project-initiated improvements in health care security, safety, and quality to
be sustained and further disseminated. Sustainability would be shown if
PFQ-initiated activities became part of ongoing practice in the targeted health
care providers or if these providers "invest[ing] their own resources
sufficiently to show commitment and the likelihood of sustained [quality]
improvement." (RFA HS-02-010, May 2002). Dissemination could be shown by
efforts to diffuse the improvement strategy or model beyond the initial target
population or providers.
1.
Sustainability Indicators
Although
final results are not known for all projects, at least 13 of them have led
already to sustainable improvements in health care security, safety or quality
if one uses a minimal benchmark—reports that some or most of the target
organizations have integrated the improvements initiated by PFQ projects into
ongoing or routine practice. Details for each project are shown in Appendix
Table A.6. Though some of them will need support from lead agencies or
partners to continue these activities, others will continue to build on
effective practices without outside support. For example:
-
Six of 10 AAP chapters report that
they will continue collaborating with physicians on practice-based educational
programs to improve their care of patients with ADHD. AAP also gained
recognition of the practice-oriented quality improvement CME program it developed
for new American Board of Pediatrics "maintenance of certification"
requirements.
-
Midwest Heart Specialists and the
Northwestern University Medical Faculty will continue working with AMA to
refine electronic data transfer for performance measurement.
-
Five of the six Catholic
Healthcare Partners hospitals will continue to employ the Heart Failure
Advocates using their own funds, rather than AHRQ's PFQ funds.
-
Effective diabetes care
interventions reportedly remain in place in: 1) the 10 primary care practices
that participated in the Lehigh Valley's program two years after it ended, 2)
in many of the practices that were involved in the American College of
Physician's project, and 3) in the 8 home health agencies in VNSNY's project.
-
A few of The Leapfrog Group's
pilot project partners are implementing the reward and incentive programs
initiated by the PFQ project without PFQ funding support.
-
Lasting changes in workflow,
documentation, and care planning processes have been made in all 11 of the nursing
facilities that participated in the ISIS-led project.
Cost Savings. Another
important indicator of the potential for sustainability is the cost of the
interventions, and specifically, any savings that the interventions yield for
providers. Lehigh Valley Hospital and Health System, for example, calculated
the financial costs of the intervention to physician practices and showed that
the patient diabetes education groups with a minimum number of patients could
generate enough billable revenue to sustain the program without the PFQ-funded
certified diabetes educators. CHCA demonstrated that the adverse drug events
prevented saved between $1.7 and $3.1 million. The catheter-associated
bloodstream infections avoided by one of CHCA's collaboratives was estimated to
save the hospitals almost $1 million. Catholic Healthcare Partners program,
however, showed the difficulty of introducing a program that reduces hospital
admissions because it lowers hospital revenue.
2. Indicators
of Broader Diffusion
Almost
all PFQ projects have begun to disseminate the results of their projects to via
journal publications and presentations at conferences. This is important to
establish the credibility of the project's approach in professional circles,
and it may be very useful to project investigators when they seek another AHRQ
grant, or funds from other sources. However, this is arguably the most passive
approach to dissemination, one that AHRQ was trying to diverge from in the PFQ
program. Moreover, its impact on diffusion is difficult to measure.
Twelve
projects are making more significant efforts to diffuse the security, quality
or patient safety approaches tested in the PFQ project to organizations or
providers beyond those targeted. They are using three strategies to accomplish
this, listed below from the least to the greatest potential for spread.
- Making widely available and easily accessible tools/toolkits, resources, or
training materials developed by the project, via Web sites and other media.
A slightly greater effort is required to disseminate the materials developed by
the projects to wider audiences by making them available on Web sites. For
example, Yale New Haven Health System made available online its
bioterrorism/emergency preparedness course and reportedly 300 physicians have
taken it and the exam for CME credit. Texas A&M University is making
available the disaster preparedness training exercises developed in the PFQ
project to medical students and rural hospitals in Texas. CHCA plans to use its
Web site and conferences to spread project results and make the NICU trigger
tool and other resources available to its members. The ACNL/CalNOC team
executed an agreement with the American Nurses Association to use the ANA
National Database for Nursing Quality Indicators Web site to transform live
coaching into a self-directed online process. While this dissemination
strategy is easy and relatively inexpensive, it does not guarantee use and
uptake of the resources, if not accompanied by aggressive and ongoing efforts
to publicize the availability of the tools and resources, and support for their
implementation.
- Securing commitments and funds from new partners, organizations, providers, and
funders to promote and diffuse evidence-based improvements more broadly.
Several grantees have already initiated new efforts to spread the quality,
safety or security improvement models embodied by their PFQ projects. A few
began these diffusion efforts with PFQ grant funds in the latter years of the
projects, but most sought and received new funds either from AHRQ, or other
sources for this work.
- New funds and new target
organizations. The American
College of Physicians obtained funds from a drug manufacturer to conduct two
additional team-oriented practice-based CME programs to improve care for
patients with diabetes and cardiovascular disease, with 20 physician practices
participating in each group. The AMA and Midwest Heart Specialists obtained an
AHRQ Health Information Technology grant to spread the MHS model for reporting
quality information to six other physician practices, using different EHR
systems. AMA also received another grant to work with MHS, Northwestern, and
other sites on related activities.
- Spread via QIO
collaborations. Both ISIS and VNSNY decided that the best way to
diffuse their quality improvement approaches was to train and work with Quality
Improvement Organizations, as part of QIOs' nursing home and home health
quality improvement initiatives. With
support from an AHRQ HIT grant, ISIS is now working with six QIOs around the
country and 30 nursing facilities to implement "real-time optimal care
planning" using digital pen or internal IT systems to streamline documentation.
VNSNY obtained funds from the Robert Wood Johnson Foundation to continue
working with 10 QIOs and 69 home health agencies on techniques to reduce acute
care hospitalization among home care patients. QIOs involved in AMDA's project
may use its approach to clinical guideline implementation as part of its
nursing home quality improvement work, but AMDA is not actively promoting it
like ISIS and VNSNY.
-
Replication in facilities
within health care systems. An
especially significant by-product of the ISIS project is that a large nursing
home chain and a large health system which had one or more of their facilities
participate in the project are spreading the model to their systems' other
nursing facilities—240 in the large chain. It is not known, however, whether
the model is being fully implemented in all facilities in the systems.
-
Creating new coalitions and
adding new partners. Catholic
Healthcare Partners decided that the best way to expand and spread heart
failure quality improvement efforts was to establish a state-based coalition in
Ohio with key stakeholders. It is also encouraging the American Heart
Association's Heart Failure "Get with the Guidelines" program to use CHP's
Heart Failure Advocates as teaching faculty.
- Developing capacity for future quality improvement projects and
institutionalizing that capacity in host organizations. PFQ projects are
also trying to diffuse their quality improvement approaches more widely through
the creation of infrastructure that can support ongoing and possibly larger QI
initiatives.
-
Adding QI infrastructure. Based largely on the successful response to, and
outcomes from, their PFQ projects, both the American Academy of Pediatrics and
CHCA recently decided to expand their QI departments and staff that were hired
to work on PFQ projects. These organizations have committed operational funds
for permanent staff, data system infrastructure, and QI support to member
providers. AAP is developing additional eQIPP modules to support online
quality reporting and a measurement system and has recently hired new staff.
CHCA is also expanding its staff and quality reporting systems. This enhanced
capacity portends well for ongoing national QI support to pediatricians and
children's hospitals in the short to medium term. The AMA's AHRQ-funded HIT
project is also creating a data warehouse for feedback and benchmarking
purposes for physician-directed QI that may become a resource for wider use.
-
Enhancing QI capacity. Other membership associations, including AMDA, ACP,
and ACNL report that their experience working with state chapters and members
on "real" QI projects through PFQ projects has enhanced their ability and
credibility to undertake similar projects in the future.
16. The
fifth, RTI, did not provide information on findings or results of their
bioterrorism preparedness projects.
Return to Contents
Proceed to Next Section