Evaluation of AHRQ's Partnerships for Quality Program

You Are Here: AHRQ Home > About AHRQ > Mission & Budget > Evaluation of AHRQ's Partnerships for Quality Program > Chapter IV (continued)

Chapter IV. What Did the PFQ Projects Achieve? (continued)

C. Outcomes and Findings from Bioterrorism and Emergency Preparedness Projects

Four of the five projects that aimed to improve the health system's preparedness for bioterrorism events and other emergencies had findings to report from their studies or modeling exercises in time for this evaluation.¹⁶ It is inherently difficult to measure the utility of these findings in the absence of real events or disaster response exercises that show whether and how health care providers and public health officials actually use the information to prepare and implement plans. For this reason, the utility of the findings is based on the perceptions of project staff. The one exception is the Connecticut Department of Public Health project that included a formal study of the effectiveness of the training provided through their PFQ grant.

Altarum Institute, which used two models to simulate the flow of patients into health care facilities in the event of smallpox and other disease outbreaks, provided information to public health officials in the San Antonio area, which they say helped them accurately estimate the number of smallpox vaccinations and distribution sites needed to control an epidemic. The information was also used to develop a purchasing strategy for bioterrorism preparedness supplies.
The Connecticut Department of Public Health/Yale New Haven Health System project's online training program for front-line physicians showed that it effectively increased the knowledge of those who took the course; but six months later, their exam scores declined almost to their pre-test scores. Project investigators speculate that since physicians have no opportunity to use the information, it quickly dissipates. Annual training or drills may be needed to retain the information.
One of two studies conducted by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) under the PFQ program focused on the existence and effectiveness of linkages for community-wide bioterrorism preparedness among health care organizations, and public health, public safety and other governmental agencies. According to the article that published the results (Braun, et al., 2006), while the majority of hospitals conducted drills or exercises, had plans to acquire additional supplies or equipment, and were prepared for decontamination needs, only 40 percent had 24-hour access to a live voice at their local health department. The survey's list of 17 elements of an effective emergency preparedness plan is regarded as a useful checklist for hospitals.
Texas A&M University System Health Science Center conducted a number of studies on factors affecting bioterrorism and emergency preparedness. A case study of federal bioterrorism funding allocation in the San Antonio area showed the importance of formal and informal communication networks throughout the region. A study of disease surveillance and reporting systems on the U.S.-Mexico and U.S.Canada borders showed that communication infrastructure at the local level needs to be improved; that funds should be targeted to disease surveillance methods with the greatest potential for mitigating disease burden; and that bi-national organizations are needed to overcome the problems created by the existence of public health bureaucracies in three national governments, dozens of U.S. states, Mexican states, and Canadian provinces, as well as numerous county and local jurisdictions.

Return to Contents

D. Sustainability and Broader Diffusion of Project Activities

In the RE-AIM framework, sustainability is called "maintenance," and it means the extent to which a program or innovation becomes institutionalized in organizational policies and practices. Both sustainability and broader diffusion were important goals for the PFQ projects. AHRQ's RFA for the program expected project-initiated improvements in health care security, safety, and quality to be sustained and further disseminated. Sustainability would be shown if PFQ-initiated activities became part of ongoing practice in the targeted health care providers or if these providers "invest[ing] their own resources sufficiently to show commitment and the likelihood of sustained [quality] improvement." (RFA HS-02-010, May 2002). Dissemination could be shown by efforts to diffuse the improvement strategy or model beyond the initial target population or providers.

1. Sustainability Indicators

Although final results are not known for all projects, at least 13 of them have led already to sustainable improvements in health care security, safety or quality if one uses a minimal benchmark—reports that some or most of the target organizations have integrated the improvements initiated by PFQ projects into ongoing or routine practice. Details for each project are shown in Appendix Table A.6. Though some of them will need support from lead agencies or partners to continue these activities, others will continue to build on effective practices without outside support. For example:

Six of 10 AAP chapters report that they will continue collaborating with physicians on practice-based educational programs to improve their care of patients with ADHD. AAP also gained recognition of the practice-oriented quality improvement CME program it developed for new American Board of Pediatrics "maintenance of certification" requirements.
Midwest Heart Specialists and the Northwestern University Medical Faculty will continue working with AMA to refine electronic data transfer for performance measurement.
Five of the six Catholic Healthcare Partners hospitals will continue to employ the Heart Failure Advocates using their own funds, rather than AHRQ's PFQ funds.
Effective diabetes care interventions reportedly remain in place in: 1) the 10 primary care practices that participated in the Lehigh Valley's program two years after it ended, 2) in many of the practices that were involved in the American College of Physician's project, and 3) in the 8 home health agencies in VNSNY's project.
A few of The Leapfrog Group's pilot project partners are implementing the reward and incentive programs initiated by the PFQ project without PFQ funding support.
Lasting changes in workflow, documentation, and care planning processes have been made in all 11 of the nursing facilities that participated in the ISIS-led project.

Cost Savings. Another important indicator of the potential for sustainability is the cost of the interventions, and specifically, any savings that the interventions yield for providers. Lehigh Valley Hospital and Health System, for example, calculated the financial costs of the intervention to physician practices and showed that the patient diabetes education groups with a minimum number of patients could generate enough billable revenue to sustain the program without the PFQ-funded certified diabetes educators. CHCA demonstrated that the adverse drug events prevented saved between $1.7 and $3.1 million. The catheter-associated bloodstream infections avoided by one of CHCA's collaboratives was estimated to save the hospitals almost $1 million. Catholic Healthcare Partners program, however, showed the difficulty of introducing a program that reduces hospital admissions because it lowers hospital revenue.

2. Indicators of Broader Diffusion

Almost all PFQ projects have begun to disseminate the results of their projects to via journal publications and presentations at conferences. This is important to establish the credibility of the project's approach in professional circles, and it may be very useful to project investigators when they seek another AHRQ grant, or funds from other sources. However, this is arguably the most passive approach to dissemination, one that AHRQ was trying to diverge from in the PFQ program. Moreover, its impact on diffusion is difficult to measure.

Twelve projects are making more significant efforts to diffuse the security, quality or patient safety approaches tested in the PFQ project to organizations or providers beyond those targeted. They are using three strategies to accomplish this, listed below from the least to the greatest potential for spread.

Making widely available and easily accessible tools/toolkits, resources, or training materials developed by the project, via Web sites and other media. A slightly greater effort is required to disseminate the materials developed by the projects to wider audiences by making them available on Web sites. For example, Yale New Haven Health System made available online its bioterrorism/emergency preparedness course and reportedly 300 physicians have taken it and the exam for CME credit. Texas A&M University is making available the disaster preparedness training exercises developed in the PFQ project to medical students and rural hospitals in Texas. CHCA plans to use its Web site and conferences to spread project results and make the NICU trigger tool and other resources available to its members. The ACNL/CalNOC team executed an agreement with the American Nurses Association to use the ANA National Database for Nursing Quality Indicators Web site to transform live coaching into a self-directed online process. While this dissemination strategy is easy and relatively inexpensive, it does not guarantee use and uptake of the resources, if not accompanied by aggressive and ongoing efforts to publicize the availability of the tools and resources, and support for their implementation.
Securing commitments and funds from new partners, organizations, providers, and funders to promote and diffuse evidence-based improvements more broadly. Several grantees have already initiated new efforts to spread the quality, safety or security improvement models embodied by their PFQ projects. A few began these diffusion efforts with PFQ grant funds in the latter years of the projects, but most sought and received new funds either from AHRQ, or other sources for this work.
New funds and new target organizations. The American College of Physicians obtained funds from a drug manufacturer to conduct two additional team-oriented practice-based CME programs to improve care for patients with diabetes and cardiovascular disease, with 20 physician practices participating in each group. The AMA and Midwest Heart Specialists obtained an AHRQ Health Information Technology grant to spread the MHS model for reporting quality information to six other physician practices, using different EHR systems. AMA also received another grant to work with MHS, Northwestern, and other sites on related activities.
Spread via QIO collaborations. Both ISIS and VNSNY decided that the best way to diffuse their quality improvement approaches was to train and work with Quality Improvement Organizations, as part of QIOs' nursing home and home health quality improvement initiatives. With support from an AHRQ HIT grant, ISIS is now working with six QIOs around the country and 30 nursing facilities to implement "real-time optimal care planning" using digital pen or internal IT systems to streamline documentation. VNSNY obtained funds from the Robert Wood Johnson Foundation to continue working with 10 QIOs and 69 home health agencies on techniques to reduce acute care hospitalization among home care patients. QIOs involved in AMDA's project may use its approach to clinical guideline implementation as part of its nursing home quality improvement work, but AMDA is not actively promoting it like ISIS and VNSNY.
Replication in facilities within health care systems. An especially significant by-product of the ISIS project is that a large nursing home chain and a large health system which had one or more of their facilities participate in the project are spreading the model to their systems' other nursing facilities—240 in the large chain. It is not known, however, whether the model is being fully implemented in all facilities in the systems.
Creating new coalitions and adding new partners. Catholic Healthcare Partners decided that the best way to expand and spread heart failure quality improvement efforts was to establish a state-based coalition in Ohio with key stakeholders. It is also encouraging the American Heart Association's Heart Failure "Get with the Guidelines" program to use CHP's Heart Failure Advocates as teaching faculty.
Developing capacity for future quality improvement projects and institutionalizing that capacity in host organizations. PFQ projects are also trying to diffuse their quality improvement approaches more widely through the creation of infrastructure that can support ongoing and possibly larger QI initiatives.
Adding QI infrastructure. Based largely on the successful response to, and outcomes from, their PFQ projects, both the American Academy of Pediatrics and CHCA recently decided to expand their QI departments and staff that were hired to work on PFQ projects. These organizations have committed operational funds for permanent staff, data system infrastructure, and QI support to member providers. AAP is developing additional eQIPP modules to support online quality reporting and a measurement system and has recently hired new staff. CHCA is also expanding its staff and quality reporting systems. This enhanced capacity portends well for ongoing national QI support to pediatricians and children's hospitals in the short to medium term. The AMA's AHRQ-funded HIT project is also creating a data warehouse for feedback and benchmarking purposes for physician-directed QI that may become a resource for wider use.
Enhancing QI capacity. Other membership associations, including AMDA, ACP, and ACNL report that their experience working with state chapters and members on "real" QI projects through PFQ projects has enhanced their ability and credibility to undertake similar projects in the future.

^16. The fifth, RTI, did not provide information on findings or results of their bioterrorism preparedness projects.

Return to Contents
Proceed to Next Section