Facilitated Look Backs

A New Quality Improvement Tool for Management of Routine Annual and Pandemic Influenza

Julia E. Aledort, Nicole Lurie, Karen Ricci, David Dausey, Stefanie Stern

TR-320-DHHS

February 2006

Prepared for the U.S. Department of Health and Human Services Office of the Assistant Secretary for Public Health Emergency Preparedness

Contents

Preface
Summary
Acknowledgments
Acronyms
Chapter One.  Introduction and Methods
What Can We Learn From Routine Annual Influenza?
What is a “Look-Back”?
How Did We Develop the “Look-Back”?
Chapter Two.  Conducting a Look-Back at Routine Annual Influenza
General Framework
Discussion Topics and Probes
Core Discussion Questions
Chapter Three.  Translating Knowledge into Practice
Eliciting Ways to Improve Routine Annual Influenza and Pandemic Influenza Management
After Action Reports
Developing an Action Plan or Improvement Plan
Chapter Four.  Design Issues and Implementation Challenges
Chapter Five.  Selected Lessons Learned by State Public Health Agencies
Appendix A.  Look-Back Planning Checklist
Appendix B.  Sample Look-Back Presentation
Appendix C.  Template for an After Action Report
References
Copyright Information: Rand Corporation

Figures

2.1. General Framework for a Look-Back at Routine Annual Influenza
A.1. Sample Agenda

 Preface

One of the challenges of pandemic influenza preparedness is the relative infrequency of pandemics and the consequent inability of public health agencies to improve preparedness by learning from direct experience.  The occurrence of predictable, yet variable, routine annual influenza offers important opportunities for public health agencies to identify lessons from each influenza season that may be broadly applicable to some aspects of pandemic influenza and even to other public health preparedness activities.

This document describes a tool that public health agencies can adopt to regularly “look back” at each routine annual influenza season, with the goals of  (1) systematically institutionalizing knowledge from one influenza season to the next; (2) continuously  improving the public health response to routine annual influenza; and (3) incorporating lessons into preparedness activities for pandemic influenza and other public health emergencies. We developed and piloted this “Look-Back” methodology with three state public health agencies (SPHAs) in different regions of the United States between June and August 2005. This document provides SPHA leaders with the basic information necessary to conduct influenza Look-Backs at routine annual influenza with their staff members and to tailor the discussion to their own needs and experiences. In addition, we report on some design issues and lessons that emerged from our pilot tests that may be of interest to SPHAs.

This report was prepared for the Office of Public Health Emergency Preparedness, but it should be of interest to policymakers, public health professionals and individuals who are involved in routine annual influenza management and pandemic influenza and public health emergency preparedness. This work was sponsored by the U.S. Department of Health and Human Services and was carried out from October 2004 through October 2005 within the RAND Health Center for Domestic and International Health Security. RAND Health is a division of the RAND Corporation. A profile of the Center, abstracts of its publications, and ordering information can be found at www.rand.org/health/centers/healthsecurity. More information about RAND is available at our Web site at http://www.rand.org.

 Summary

Since influenza pandemics¹ are relatively infrequent, public health agencies have limited opportunities to learn from direct experience with them.  In contrast, routine annual influenza occurs each year with some predictability, and this routine occurrence offers important opportunities for public health and pandemic preparedness. Each routine annual influenza season presents unique challenges and lessons that may be applicable to pandemic influenza and other public health preparedness activities. For example, unanticipated events in the past decade, such as influenza vaccine shortages, high numbers of influenza-associated deaths among U.S. children, and unexpected surges in demands for patient care, have underscored the need for better preparation—specifically, the importance of early planning and responsiveness to different contingencies that may arise. 

With the specter of pandemic influenza before the American populace, the RAND team sought to harness opportunities arising from the annual occurrence of influenza to help public health agencies improve their response to routine annual influenza and to incorporate lessons into pandemic and emergency public health preparedness planning.  Specifically, we set out to develop and test a “Look-Back” methodology to assist state public health agencies (SPHAs) in systematically assessing their experiences from each routine annual influenza season and translating those experiences into future public health practice.

A Look-Back is a quality improvement tool that seeks to identify relevant and applicable lessons for routine annual and pandemic influenza preparedness by asking a broad and diverse group of participants who have been involved in routine annual influenza activities to critically evaluate their management of the past routine annual influenza season. We developed the Look-Back by focusing primarily on an SPHA’s role in routine annual influenza activities. Because of the SPHA’s role in these routine activities, we pilot-tested the Look-Back with three SPHAs and relevant stakeholders between June and August 2005. 

By involving a broad and diverse group of participants, Look-Backs seek to foster open, candid, no-fault, systems-level analyses of routine annual influenza management. A Look-Back traces the series of events that unfolded during the past influenza season, key decisions that were made by various stakeholders, and how those decisions were perceived and acted upon by others.

The general framework for a Look-Back includes an independent, objective facilitator or discussion leader who helps to stimulate the discussion and guides the participants in a critical evaluation of recent historical events and activities based on such general topic areas as the organizational structure of decision making; influenza surveillance; vaccine procurement and distribution; routine annual influenza vaccination campaigns; vaccine administration; priority groups and implications of changing priorities; non-vaccine and public health strategies; communication; and unanticipated events.

In advance of the Look-Back, the facilitator and the SPHA will need to select from three to six discussion topics, as well as suggested questions, or probes, that the facilitator may use to keep the participants on track and to ensure that particularly relevant or timely issues are not overlooked.  In addition to topic-specific issues, we developed the following core questions, which are broadly applicable to all of the discussion topics:

• What are activities, roles, and responsibilities during routine annual
  influenza season?
• What are specific issues that came up last year?
• What went well, and are past successes sufficiently institutionalized? What specific problems emerged?
• What might have been done differently?
• What should be done differently in the future?
• What are lessons for an influenza pandemic?

Appendix A provides advance-planning and logistical checklists for Look-Backs. Suggested presentation slides that may be used to guide the Look-Back are contained in Appendix B.

To translate the new information that emerges from the Look-Back into concrete steps to improve future public health practice and preparedness, we ended each topic session with a discussion dedicated to eliciting specific lessons learned. We then ended the overall Look-Back with a session to review, prioritize, and document all such lessons.  An after action report (AAR) is a summary of the general discussion that highlights specific strengths and systems-level improvements identified by the group. It addresses all participants’ needs and serves as an important vehicle for broadly disseminating past successes, strengths, and lessons learned.  It also facilitates incorporation of new individuals or functions into routine annual influenza season management. A suggested outline for an AAR is provided in Appendix C.

The AAR also informs the development of an Action Plan, or Improvement Plan. Following development and review of the AAR, specific strengths, lessons, and systems-level ideas for improvement identified during the Look-Back can be developed into a detailed, comprehensive plan that outlines specific steps, or actions, and that identifies individuals accountable for those actions, as well as explicit performance benchmarks.  The Action Plan is the means by which lessons learned are translated into concrete, measurable steps that result in improved response capabilities.

We tested the Look-Back methodology with only three SPHAs and their community stakeholders. Nevertheless, we identified four design issues and implementation changes that were common across sites:

1. Advance planning and investigation allow for customized Look-Backs.
2. Facilitator objectivity and independence are critical.
3. It is a challenge to produce effective and broadly agreed-upon AARs.
4. AARs can generate valuable dialogue if they are broadly disseminated and reviewed by individuals not typically involved in routine annual influenza activities.

The fourth point is especially apt because it is inclusive of more than those who participate in a specific Look-Back: In at least two cases, both individual and organizational tensions resulted from the functional separation of routine annual influenza management and influenza pandemic preparedness. In one case, broad dissemination of the AAR resulted in a new level of engagement by the state epidemiologist in routine annual influenza. Previously, epidemiologists tied functionally to the division of emergency planning and preparedness viewing routine annual influenza as outside of their purview.

In addition to having overall design and implementation issues in common, our three pilot SPHAs shared several important lessons and areas of improvement with respect to the 2004-2005 influenza season. Look-Backs with all three SPHAs revealed that several areas of planning are particularly important for enhancing routine annual influenza management and pandemic preparedness in the future, including (1) leveraging state emergency preparedness resources and infrastructure; (2) establishing clear lines of communication with the SPHA and between and among SPHAs, local public health agencies (LPHAs), healthcare providers, and the public; and (3) facilitating the distribution and administration of vaccine.

Look-Backs have utility as a relatively simple, effective quality improvement tool that any level jurisdiction can use to systematically assess actual recent past events to strengthen management and communications systems relevant to future routine annual and pandemic influenza. Adoption and implementation of Look-Backs with regular frequency (i.e., annually) by public health agencies will capitalize on routine annual influenza to better prepare for pandemic influenza (a rare opportunity in public health); document and formalize learning from successes as well as from problems; encourage follow-through on lessons learned; and reinforce the role of public health during annual and pandemic influenza, as well as during other public health emergencies. Although we pilot-tested the Look-Backs only with SPHAs, the tool may also be relevant to LPHAs. This guide should therefore be of broader interest to policymakers, public health professionals, and individuals who are involved in routine annual influenza management and emergency preparedness.

 Acknowledgments

We acknowledge and thank the entire RAND team on this project, many of whose efforts helped shape our thinking and approach to this work.

Developing this guide involved the participation of public health professionals from state public health agencies across the country, and that of numerous other influenza stakeholders.  We are deeply grateful for their willingness to participate in the pilot tests and to provide us with constructive feedback about the design of these discussions. We would like especially to thank Kathleen Toomey, MD, MPH, Center for Homeland Defense and Security, Naval Postgraduate School, and Melinda Moore, MD, MPH, Senior Natural Scientist at the RAND Corporation, for their in-depth reviews of this document. 

We also acknowledge the assistance and guidance of Dr. William Raub, Deputy Assistant Secretary for Public Health Emergency Preparedness and Science Advisor to the Secretary, Lara Lamprecht, Program Analyst, Office of Public Health Emergency Preparedness, U.S. Department of Health and Human Services, and Dr. Pascale Wortley,  Medical Officer, U.S. Centers for Disease Control and Prevention. Their commitment to developing tools and resources to help public health agencies improve the country’s public health preparedness was the driving force behind this work. 

 Acronyms

 Chapter One

Introduction and methods

 What Can We Learn From Routine Annual Influenza?

Pandemic influenza² is potentially catastrophic, but also, fortunately, very rare. Only three pandemics occurred during the 20th century.  The most notorious of these was the “Spanish influenza” of 1918-1919, which is estimated to have killed between 500,000 and 675,000 people in the United States and about 50 million people worldwide. Two other 20th century pandemics are the Asian influenza of 1957 and the Hong Kong influenza of 1968. Increasing global anxiety today about human-to-human transmission of influenza A (H5N1)—an influenza A virus subtype that is highly contagious and deadly among poultry—and the likelihood that vaccine and antiviral supplies will be inadequate in a pandemic significantly increase the importance of traditional public health measures to prevent disease spread. However, since pandemics are relatively infrequent, there are limited opportunities for public health agencies to learn from direct experience with them.

Unlike pandemic influenza, routine annual influenza occurs each year with some predictability and therefore offers important opportunities for public health and pandemic preparedness. Routine annual influenza is a contagious viral illness marked by abrupt onset of constitutional (e.g., high fever, myalgias), and respiratory (e.g., cough, sore throat) symptoms; it causes an estimated 36,000 deaths and 200,000 hospitalizations each year in the United States (U.S. Centers for Disease Control [CDC] and Thompson et al., 2004).  Peak incidence typically occurs between the months of December and February.  Most people who contract influenza recover in one to two weeks; however, some people develop serious and potentially life-threatening medical complications, such as pneumonia.  People over the age of 65, people with chronic medical conditions, and children under the age of 2 years are at greatest risk of developing complications.  For several decades, the influenza vaccine has helped people who come in contact with the virus avoid or mitigate influenza illness and its complications. Moreover, influenza antiviral medications are an important adjunct to influenza vaccine in the prevention and treatment of influenza. Antiviral drugs are most often used to help control influenza outbreaks in institutions, for example, in nursing homes or in hospital wards, where people at high risk for complications from influenza are in close contact with each other.

Each recurrent influenza season presents unique challenges and potential lessons relevant to some aspects of pandemic influenza and other public health preparedness activities. For example, in October 2004, when one of two major vaccine manufacturers announced that it would not release any vaccine for the 2004-2005 season because of potential contamination issues, the United States lost nearly half of its anticipated influenza vaccine supply.  As a result, the CDC issued recommendations to the states that doses of the vaccine be restricted to high-risk priority groups only. Although health officials took actions to distribute the limited supply of vaccine, reports persisted of shortages among persons in need, and of individuals who were turned away and never returned when supplies became available. 

The influenza vaccine shortage in 2004-2005 was an unexpected challenge for those managing influenza that season. In turn, it underscored the need for better preparation for routine annual influenza, specifically, the importance of early planning, responsiveness to unexpected contingencies (in the 2004-2005 case, mechanisms to make vaccine available), and effective communication to ensure that available vaccine is targeted to those who need it most.  Moreover, it revealed significant fissures in pandemic preparedness at the federal, state, and local levels, and motivated new investigations into the current state of the Strategic National Stockpile (U.S. GAO, 2005).

Other influenza seasons in our recent past have similarly presented unanticipated problems.  For example, in the 2003-2004 season, a substantial number of influenza-associated deaths occurred among young U.S. children (Bhat et al., 2005).  In October 2000, the U.S. Food and Drug Administration (FDA) announced that lower-than-expected production yields of the new influenza A H3N2 strain and manufacturing problems at two companies would cause a delay and temporary shortages in the availability of influenza vaccine (U.S. FDA, 2000).  And during the 1999-2000 season, an earlier-than-anticipated arrival of influenza caused an unexpected surge in demand for patient care and subsequent overcrowding of emergency rooms, insufficient bed supply, and ambulance diversions around the country (Schoch-Spana, 2000).

With the specter of pandemic influenza before the American populace, the RAND team sought to harness opportunities arising from routine annual influenza to help state public health agencies (SPHAs) learn from the challenges, lessons, and solutions that arise each year and apply them to future influenza seasons and, more broadly, to pandemic influenza and other public health preparedness activities. Specifically, we set out to develop and test a “Look-Back” methodology to help SPHAs systematically assess their experiences after each routine annual influenza season and translate them into future public health practice. Adoption of this tool by public health agencies might also help to educate members of the community about the role of public health during routine annual influenza and to reinforce its significance in managing a pandemic emergency.  The contents of this guide should be of interest to policymakers, public health professionals, and individuals who are involved in routine annual influenza management and pandemic and public health emergency preparedness.

 What is a “Look-Back”?

A Look-Back at routine annual influenza is a facilitated discussion that occurs at the end of each influenza season with SPHA leaders, key staff, and relevant community stakeholders. Look-Backs seek to foster open, candid, no-fault systems-level analyses of routine annual influenza management by asking a diverse group of participants involved in routine annual influenza activities to critically evaluate their management of the past influenza season. To meet the objective of identifying relevant and applicable lessons for routine annual and pandemic influenza preparedness, a Look-Back traces the series of events that unfolded during the past influenza season, the key decisions that were made by various stakeholders, and how those decisions were perceived and acted upon by others. 

Unlike tabletop exercises, which simulate a hypothetical public health emergency and ask participants to describe how they would respond in the future to a specific issue (e.g., surge capacity) if the situation were real, a Look-Back by definition reviews a past real-world event, critically examines how participants did respond, and draws on a wealth of practical experience and a broad range of perspectives to inform future responses. 

 Although these Look-Backs focus on the SPHA’s role in routine annual influenza management, the framework is generally applicable to any public health agency. For instance, a Look-Back could easily be tailored by many state, local, and/or district public health agencies and their community partners to evaluate their joint ability to coordinate across jurisdictional lines (e.g., national, state, county, city).  State and local public health agencies may elect to set up a regular Look-Back schedule with key influenza and emergency preparedness partners and stakeholders.

The breadth and strength of community engagement in the Look-Back will depend in some measure on how well SPHAs are able to reach out to a wide range of community stakeholders who have participated in any aspect of routine annual influenza management or who may offer assistance in the future.  For example, relevant stakeholders outside of public health who could participate in a Look-Back could include hospital representatives; nursing home and long-term care representatives; professional medical organizations; managed care organizations; insurers; pharmacies; the American Association of Retired Persons (AARP) and other advocacy groups; university-based health services; representatives from commercial enterprises offering influenza vaccine to the public (grocery stores provide a substantial proportion of vaccinations in some areas); emergency response groups that might have a role to play in influenza; and leaders from the business community, including large employers that provide vaccinations for their employees through worksite wellness programs. Many such businesses have a medical director or other representative who could be part of the planning process.  (See Appendix A for a suggested list of Look-Back participants.)

 How Did We Develop the “Look-Back”?

In early 2005, RAND researchers met with federal and state public health leaders to discuss what tools would be most useful to help states institutionalize knowledge from previous influenza seasons and prepare for future ones.  Based on these discussions, we developed a discussion framework to help SPHAs cultivate lessons learned during each routine annual influenza season, and we referred to this approach as a “Look-Back.”  To test the feasibility of this approach, we piloted the discussion framework in three SPHAs in different regions of the United States between June and August 2005. At the end of each pilot test, we solicited feedback about the process from participants and incorporated new guidance into the next iteration and pilot test. Below, we provide a brief description of the development of the Look-Back methodology.

First and foremost, our effort to foster open and candid discussion among a diverse set of individuals about past and future management of routine annual influenza was guided by the following four assumptions:

1. Although not a pandemic, routine annual influenza activities provide potential recurrent lessons for some aspects of pandemic influenza and other public health preparedness activities.
2. While many people can contribute important information about the immediate past and other previous routine annual influenza seasons, no one person knows the entire story or can represent all angles of the narrative. It is therefore useful to engage a wide variety of perspectives.
3. A systems-level, “no fault” analysis of what happened by a representative group of people who were involved and continue to have responsibilities in the area will identify opportunities to learn from experience and adopt improvements that can be useful in the future.
4. Organizational learning is not complete without an action plan that specifies responsibilities for change.

We made every effort to customize each Look-Back to reflect the three states’ particular experiences with routine annual influenza and to meet specific needs of the SPHA and its community stakeholders. To this end, we collaborated with SPHA leaders to gather as much background information as possible about the state’s activities during a typical influenza season, to identify key issues in the management of the previous influenza season, and to select salient topics for the upcoming discussion. Two weeks before the Look-Back discussion, we conducted a 60-minute call with one or two SPHAs and/or immunization program officials who were intimately involved in routine annual influenza season activities. In this call, we addressed the following broad questions:

• In a typical influenza season, who are the key players internally within the SPHA and in the community, and what are their activities, roles, and responsibilities?
• What were some challenges in the management of the last influenza season (e.g., surveillance, vaccine procurement and distribution, communication)?
• Considering both routine annual influenza and pandemic preparedness activities, what are specific issues that would benefit from in-depth review?

Given this information, we collaborated with the SPHA leaders and/or their representatives to define a specific set of discussion areas and topics for the Look-Back.  Many issues arise in the course of a typical influenza season; it is difficult to cover all of them in great detail in a single focused discussion. We therefore generated a comprehensive, but by no means exhaustive, list of routine annual influenza season activities and topics to review with the SPHA as a starting point for discussion. These topics are presented in detail in Chapter Two.  Each SPHA that piloted this process selected anywhere from three to six discussion topics based on its own past experiences and the time allotted for the Look-Back.

During the meetings with the SPHA, we also solicited a list of participants, which necessarily depended on the structure of the SPHA and its relationship with local public health agencies (LPHAs) and community stakeholders during a routine annual influenza season. We encouraged the SPHAs to invite individuals who would ideally be involved in future routine annual influenza season activities, even if they had not been involved in the past (e.g., emergency management officials, commercial pharmacies, private physician organizations).  Finally, we asked the SPHA leaders to personally enlist specific individuals to provide a very brief (i.e., two-minute) overview of the typical and relevant influenza activities surrounding each discussion topic on the day of the Look-Back.  At the end of each Look-Back, participants were asked to fill out an evaluation form. RAND staff took detailed notes at all Look-Backs, guided participants in the development of initial Action Plans, and generated state-specific, de-identified after action reports (AARs) for feedback and review by each SPHA.

The remainder of this report is organized as follows:  Chapters Two and Three provide SPHA leaders with the basic information necessary to conduct annual influenza Look-Backs with their staff members and to tailor the discussion to their own unique needs and experiences.  Chapter Two provides general information about the structure and organization of a Look-Back, potential issues or challenges that may arise during routine annual influenza season that may be relevant for discussion, and facilitator probes to motivate the dialogue. Chapter Three presents tools for translating information derived from the Look-Back into public health practice, such as Action and Improvement Plans and AARs. Appendices supplement Chapters Two and Three with more specific guidance, including advance-planning and logistical checklists, a list of suggested participants, a sample agenda, sample presentation slides, and a template for AARs. 

In addition to offering detailed guidance to SPHAs about how to conduct a Look-Back at routine annual influenza, we also report on some illustrative issues and observations that emerged from our pilot tests and that may be of interest to SPHAs.  Chapter Four focuses on issues related to the design and implementation of a Look-Back; Chapter Five reviews a select sample of lessons learned as they were identified by the pilot SPHAs.

 Chapter Two

Conducting a Look-Back at Routine Annual Influenza

 General Framework

This chapter is designed to show SPHAs how to conduct Look-Backs after each routine annual influenza season.  Look-Backs can be tailored to an SPHA’s unique experiences and future needs, and they can be customized to meet specific time and resource constraints.  For example, the Look-Backs we designed and tested focused on three to six discussion topics relevant to routine annual and pandemic influenza management (presented below). They lasted between three and five hours and involved 10 to 25 participants, depending on the reach of the state’s routine annual influenza activities and partners. Participants included SPHA leaders and staff with key responsibilities related to influenza and pandemic planning; healthcare partners typically involved with routine annual influenza activities, or who should or could play a role in future years; and other partners from the community.  Appendix A includes a suggested list of Look-Back participants and outlines ways to collaborate with SPHA leaders in advance of a Look-Back to gather relevant background about the state’s activities during a typical influenza season, and to identify salient topics for the upcoming discussion. 

There are many ways to structure a Look-Back.  Figure 2.1 presents one suggested framework that we developed during our pilot tests. First, a Look-Back is led by an independent, objective facilitator, or discussion leader, who helps to stimulate the discussion and guides the participants in a critical evaluation of relevant historical events and activities based on the general topic areas presented below. To maximize the ability of the facilitator to avoid bias and ensure that all sides are adequately heard, and to prevent possible conflict of interest, someone external to the public health department employees should serve as the Look-Back facilitator.Appendix A discusses the facilitator role in greater detail and provides a reference for facilitator training; the importance of facilitator independence is addressed in Chapter Four. 

 Figure 2.1. General Framework for a Look-Back at Routine Annual Influenza
 

The facilitator opens with introductory comments, an overview of the objectives and process, and participant introductions.  Each predetermined routine annual influenza season topic, e.g., surveillance, communications, is then discussed in turn.  For each topic, the SPHA or facilitator may elect to enlist a specific participant in advance of the discussion to briefly introduce the typical state-level routine annual influenza activities. Assigning individuals ahead of time to kick off each discussion provides participants with a common ground for each discussion and engages a variety of participants.

At the end of each topic’s discussion, the facilitator elicits from participants a list of practical lessons learned for both routine annual influenza and pandemic influenza that can form the basis of an Action Plan and an AAR. A summary review and prioritization of action items then concludes the Look-Back. Suggestions for initiating and developing follow-up AARs and specific Action Plans can be found in Chapter Three.

The next section provides discussion topics and sample questions that facilitators may find helpful in moderating a Look-Back discussion.  Additionally, facilitators may elect to lead the Look-Back with the aid of a PowerPoint presentation.  Appendix B provides sample templates that can be modified for this purpose.  No preparatory work is required of the participants, although they are expected to share their roles and experiences from the preceding influenza season in a way that stimulates learning and feedback.

 Discussion Topics and Probes

Several weeks to a month before the Look-Back, the facilitator and the SPHA will need to select the discussion topics.  Many complex issues and activities arise in the course of each influenza season, some of which are particularly relevant for pandemic planning and preparedness.  Below, we present an overview of broad categories of routine annual influenza season activities that give rise to relevant topics from which SPHAs may choose to structure the Look-Back.  These topics were identified with the assistance of officials from the CDC, the Association of State and Territorial Health Officials (ASTHO), the National Association of County and City Health Officials (NACCHO), and the three SPHAs that piloted this methodology.

Following each topic described briefly below is a list of suggested questions (probes) that the facilitator may use to keep the participants on track and ensure that particularly relevant or timely issues are not overlooked. Discussion probes are best used on a case-by-case basis, since too many leading questions may inhibit the discussion.  On the other hand, without some assistance, important points worthy of discussion might be neglected by participants.³  

1.  Organizational Structure of Decisionmaking

In past years, unanticipated events have led to the use of formal decisionmaking channels to manage routine annual influenza, e.g., Incident Command Systems (ICS)⁴ or directives invoking public health emergency authority.  At other times, more routine or typical decisionmaking channels have been appropriate.

• Was an ICS activated or a directive under the public health emergency authority issued last year? Why or why not?
• How was the decision made to go to ICS or to issue a directive?
• In the absence of a formal decisionmaking rule or process, how were key decisions and/or conflicts resolved, e.g., advisory committees, coalitions?
• Were there critical gaps in organizational structure and decisionmaking, and what ideas do you have about how these gaps might be filled?

2.  Influenza Surveillance

Through the efforts of public health agencies, facilities participating in influenza research, and private physicians and laboratories, the CDC develops a national picture of influenza virus activity, the geographic distribution of influenza viruses, and the clinical impact of the circulating viruses.  State public health agencies participate in this surveillance effort by reporting the estimated level of influenza activity to the CDC every week during the influenza season (October to May).

• How do you know when and where influenza season arrived in your state?
• Which surveillance indicators are—or would be—most useful to you for managing routine annual influenza in your jurisdiction (e.g., national versus local data; influenza-like illness (ILI) versus laboratory-confirmed cases, versus deaths; proxy indicators, such as pharmaceutical purchases or school/work absenteeism)?
• In determining which surveillance information to collect in your jurisdiction, how do you balance the value and cost of different surveillance indicators, such as ILI (nonspecific but easier, cheaper, and potentially faster) versus laboratory-confirmed influenza (more specific but costlier and with potentially greater delays)?
• How do you measure the effectiveness of the current surveillance system, and how effective is your surveillance?
• How are influenza incidence data obtained from private providers?
• Approximately what percentage of influenza cases is laboratory-confirmed as opposed to clinically diagnosed?
• How many labs (public and private) in your jurisdiction have the capability to perform influenza testing?
• If testing were to become widespread, what are likely vulnerable points in the process (i.e., specimen collection and storage; specimen transport; effective and timely lab tests; reporting of results to SPHA, providers)?
• Were you able to assess vaccine effectiveness (i.e., whether people receiving vaccine were more or less likely to become sick with laboratory-confirmed influenza or ILI)?  If not, would this capability be helpful to you in managing routine annual influenza? Would it be feasible?

3.  Vaccine Procurement and Distribution

Influenza vaccine procurement and distribution in the United States are largely a private-sector responsibility, although approximately half of states handle procurement and distribution of the influenza vaccine through the SPHA during the routine annual influenza season.  As with other pharmaceutical products, influenza vaccine is sold to thousands of purchasers by manufacturers, numerous medical supply distributors, and other resellers, such as pharmacies. In a pandemic situation, these procurement and distribution systems might require adjustment to ensure that vaccine reaches priority groups.

• Does the SPHA purchase vaccine and, if so, when, by what mechanism, and for whom?
• Is there a process by which the SPHA determines how much public- and/or private-sector vaccine is needed in the state/jurisdiction? 
• Does the SPHA have the legal authority to enforce specific administrative recommendations for vaccine and/or to redistribute public and private stores of vaccines?
• Who or what other entities (e.g., LPHAs, private providers, pharmacies, healthcare groups, employers) procure vaccine in your state?
• Are you able to track how much vaccine is ordered, how much is purchased in a given year, and to whom it is delivered?
• Do you have ways to track vaccine delivery and distribution in close to real time, so that you can determine where it is wanted and/or needed?
• If there were problems with uneven distribution, such as spot shortages, how did you respond? For example, did you adapt messages for providers and the public regarding vaccine availability and priority groups?
• If there were vaccine shortages, did you attempt to redistribute excess private sector vaccine supply (e.g., from providers) to those without vaccine?  If so, how did you go about it, and with whom did you partner?  If not, how did you respond to unmet demand? 
• What other comments do you have about problems/gaps in vaccine procurement and distribution, and what ideas do you have about fixing them?

4.  Routine Annual Influenza Vaccination Campaigns

Efforts to promote the benefits of influenza vaccine before the influenza season through partnerships with medical societies, social service organizations, and the private sector are important components of routine annual influenza management.

• How does your state/jurisdiction stimulate demand for influenza vaccine, including outreach to different groups (e.g., healthcare workers, nursing home workers/residents, minorities, different ethnic groups, disabled)?
• What is the role of immunization coalitions in your state?  How do they function in a routine year, and in the presence of a vaccine shortage?
• Did past vaccine shortages alter demand for vaccine, regardless of the supply?
• What other comments do you have about problems/gaps related to routine annual influenza vaccination campaigns, and what ideas do you have for fixing them?

5.  Vaccine Administration

Once vaccine arrives in the state and at a specific site (e.g., physicians' offices, public health clinics, nursing homes, and less traditional locations, such as workplaces and various retail outlets in the state), there is the additional step of ensuring that individuals who need or want vaccine receive it. 

• What are the usual sites for vaccination administration (e.g., private providers, community-based sites, pharmacies, employers, public health clinics, long-term care facilities, commercial entities)?
• What was the vaccination coverage target last year? Was it met? Why or why not?
• Are there typically, or have there been, mass vaccination clinics or other strategies to deliver vaccines?  Who is targeted in mass vaccination efforts, e.g., high-risk groups and special subpopulations, non-high-risk individuals, the entire public without distinction of risk?
• Were your mass vaccination efforts successful?
• What are some problems that you have experienced with mass vaccination programs (e.g., long lines, elderly/disabled waiting for many hours)?
• Have vaccine registry systems been used for children and/or for adults, and, if so, how well did they work? How adaptable were they for routine or pandemic influenza? Did they help you monitor vaccination coverage over the course of the season?  
• What other comments do you have about problems/gaps related to vaccine administration, and what ideas do you have for fixing them?

6.  Priority Groups and Implications of Changing Priorities

When there is an adequate supply of vaccine, all those for whom it is recommended should get vaccinated.  However, in the presence of a vaccine shortage or other events that affect vaccine supply, vaccine prioritization is more problematic.  The following questions are specific for the management of routine annual influenza under vaccine shortage conditions:

• Have you used priority groups in the past when there was a vaccine shortage?
• If there was a vaccine shortage last year, did you use a construct of priority groups for targeting influenza vaccination?  Did you use and/or modify CDC guidelines? How and in what way?
• Were you able to identify priority individuals before vaccination?
• What mechanisms did you use to ensure that vaccine went differentially to the highest priority groups and that providers followed the recommended guidance?

• In the event of another vaccine shortage or unusual patterns of influenza illness, how would high-risk and special populations be prioritized to receive vaccine?  What would be the most effective and efficient mechanism(s) to ensure that these populations are adequately served?
• What other comments do you have about problems/gaps related to priority groups, and what ideas do you have for fixing them?

7.  Non-Pharmacological and Public Health Strategies

Past influenza vaccine shortages or delays, and the limited availability of vaccine supply for pandemic influenza, elevate the need for the promotion of non-pharmacological and traditional public health interventions. Such interventions include antiviral prophylaxis and treatment, and infection control strategies, which include respiratory hygiene/cough etiquette, hand hygiene, and other social distancing measures, such as closing schools and canceling large public events.

• Are antivirals typically promoted as part of routine annual influenza management, and, if so, how? Were they promoted last year? Do you know how effective they were?  How important is it for you to know this effectiveness?
• Are infection control strategies, such as the use of masks, hand hygiene, staying home from work/school when ill, other social distancing measures, promoted as part of influenza season management?  If so, how were such measures implemented last year, and do you know how effective any such measures were?
• Under what circumstances would more drastic public health measures be taken (e.g., school closings, cancellation of public gatherings and events)? Who would make those decisions?
• Do you have experiences from any past influenza seasons that shed light on the feasibility and effectiveness of non-vaccination public health measures to control disease spread in the population, or in specific population groups?
• What other comments do you have about problems/gaps related to non-vaccine public health strategies, and what ideas do you have for fixing them?

8.  Communication

Communication and coordination between SPHAs, LPHAs, the Federal Government and community stakeholders plays a critical role in the management of routine annual influenza, especially when responding to changing circumstances or messages.

• In a typical year, with whom and how do you communicate about routine annual influenza issues (e.g., within SPHA divisions, across state and local jurisdictions, with physicians and other providers, private-sector entities, other key stakeholders)?
• How well did your communications system work last year?
• What were the key issues about which you needed to communicate, with whom, and when?
• How well did you communicate with the Federal Government (e.g., CDC)?
• How have you managed risk communication and the problem of changing messages with the public?
• Based on your experiences, what key communication gaps exist, and what ideas do you have for fixing them?

9.  Unanticipated Events

Each influenza season presents unique challenges and affords opportunities for improvements for both routine annual influenza and pandemic influenza management.

• What unforeseen events (e.g., vaccine delay or shortage, excess child mortality, limited clinical or laboratory surge capacity) challenged the management of routine annual influenza last year?
• How have unanticipated events shaped the management of routine annual influenza in your state/jurisdiction?
• How have processes and communication channels been modified to allow for flexibility in the event of unforeseen challenges?
• What other comments do you have about problems/gaps related to communications, and what ideas do you have for fixing them?

 Core Discussion Questions

In addition to the topic-specific issues described above, we also developed the following core questions, which are broadly applicable to all of the above discussion topics. We found it useful to show these core questions to the participants (e.g., on a PowerPoint slide) to provide some context for each discussion session:

1. What are activities, roles, and responsibilities during routine annual influenza season?
2. What are specific issues that came up last year?
3. What went well, and are past successes sufficiently institutionalized?
4. What specific problems emerged?
5. What might have been done differently?
6. What should be done differently in the future?
7. What are lessons for an influenza pandemic?

 Chapter Three

Translating Knowledge into Practice

This document presents a framework to help SPHAs capitalize on the predictable recurrence of routine annual influenza and identify lessons that may improve routine annual influenza and pandemic influenza planning.  In this chapter, we briefly review some suggested strategies to translate new information that emerges from the discussion into concrete steps to improve future public health practice and preparedness.

 Eliciting Ways to Improve Routine Annual Influenza and Pandemic Influenza Management

If it is successful, a Look-Back will reveal a variety of important strengths, gaps, and ideas for improving the management of routine annual influenza.  It is therefore important to end each discussion section by eliciting specific lessons learned, and to end the overall Look-Back with a session to review, prioritize, and document all such lessons.  One suggested process that actively engages each individual participant is similar to a “hot wash,” a common practice following public health and military tabletop exercises that reserves dedicated time for participants to review the discussion and the group’s responses.⁵  For example, at the end of the Look-Back, the facilitator may ask each individual discussant to identify the two or three most important gaps in the management of the preceding influenza season and specific steps the SPHA and community stakeholders can take to improve performance.

This dedicated review allows participants to receive feedback from one another, provides them with an opportunity to critically review strengths and gaps in the management of the past influenza season, and encourages them to translate past activities and events into meaningful, actionable steps with routine annual and pandemic influenza planning and preparedness in mind.  This translation means both institutionalization of successful activities as well as changes to address gaps or problems.  Finally, this process reinforces learning by giving participants time to reflect on their individual responses and on comments made by their peers. 

 After Action Reports

An AAR is a summary of the general discussion that highlights specific strengths and systems-level improvements identified by the group and that informs the development of an Action Plan, or Improvement Plan.  The resulting official AAR should be the product of the SPHA divisions and community stakeholders that participated in the Look-Back.  The AAR addresses all participants’ needs and serves as an important vehicle for broadly disseminating past successes, strengths, and lessons learned. To this end, the AAR also facilitates incorporation of new individuals or functions into routine annual influenza season management. Given the sensitivities associated with documenting diverse and potentially competing perspectives about a past event, and the need for buy-in and approval from a range of individuals, the AAR should be circulated to participants for feedback and comments, as well as to key individuals who did not attend the Look-Back. Moreover, goodwill may be generated if the AAR documentation process is coordinated with multiple divisions within the SPHA (e.g., emergency preparedness, infectious disease, health promotion) that are (or should be) involved in routine annual and pandemic influenza planning and preparedness activities. 

An AAR can include brief summaries of the discussion topics, a bulleted list of strengths and successes from the preceding influenza season, implications deriving from qualified successes, and areas for improvement by the topic areas covered. Although routine annual influenza management successes may be overlooked, understanding and building on such successes may provide an important platform for addressing failures. After action reports may provide a useful starting point for an Action, or Improvement, Plan (discussed below).  There is no specific, formal format for an AAR; however, one suggested outline is provided in Appendix C. ⁶  

 Developing an Action Plan or Improvement Plan

Following development and review of the AAR, specific strengths, lessons, and systems-level ideas for improvement identified during the Look-Back can be developed into a detailed, comprehensive plan with actionable and accountable steps and explicit performance benchmarks. This Action Plan, or Improvement Plan, is the means by which lessons learned are translated into concrete, measurable steps that result in improved response capabilities.⁷  It is developed by the SPHA with relevant community stakeholders, and it specifically details what actions will be taken to address each recommendation, who or what division within the SPHA will be responsible for taking the action, and the time line for completion.
 
Although it is unreasonable to expect that a comprehensive Action Plan can be developed during the Look-Back itself, it may be useful to begin the process while all key participants are together.  For one or two areas targeted for improvement, the facilitator should challenge participants to outline explicit improvement goals, identify specific steps required to meet those goals, identify responsible and accountable individuals, and specify time lines for action and follow-up. For some recommendations, the course of action will be clear and can be defined immediately. For other recommendations, additional information will be required before the appropriate course of action can be defined.  Time permitting, the facilitator should help participants explore options for obtaining needed resources or for identifying necessary personnel.   Some suggested general (i.e., not topic-specific) questions to elicit initial action items include the following:  

• What specific steps can the SPHA and community stakeholders take to improve performance next year?
• What are the objectives?
• What has to occur to ensure that these objectives are met?
• • What are the initial tasks?
  • Who is responsible?
  • Who else should be involved?
  • What is the timeline for achieving this?
• How will you evaluate progress for the upcoming influenza season?
• What are plans for documentation?

 Chapter Four

Design Issues and Implementation Challenges

Although we tested the Look-Backs with only three SPHAs and their community stakeholders, we identified four design issues and implementation challenges that were common across sites and that might be useful to SPHAs and other public health agencies in planning and conducting Look-Backs at routine annual influenza.

1.  Advanced planning and investigation allow for customized Look-Backs.

In general, we found that SPHA leadership buy-in to the Look-Back process was critical to the success of the pilot Look-Backs, and that our tailored discussions with the leadership in advance of the Look-Back inspired both interest and confidence in the process.  If the facilitator does not work within the SPHA, it is especially important that he/she collaborate with its leadership and key staff at predetermined intervals leading up to the event to (1) customize the discussion; (2) reflect state (or local) experiences with routine annual influenza; and (3) meet specific needs of the SPHA and its community stakeholders.  Appendix A provides specific ways for a facilitator to tailor a Look-Back to a public health agency’s needs.

2.  Facilitator objectivity and independence are critical.

Look-Back groups ranged from 15 to 25 participants in the three SPHAs, and all groups comprised individuals representing both pubic health agencies and a wide range of community stakeholders. Although in all three Look-Backs it was important to include a variety of community stakeholders, the resulting group size and composition mandated skilled, active, structured, and objective facilitation and mediation.  For example, in large groups, a few confident or opinionated participants may overwhelm others in the room.  To guarantee each individual an appropriate amount of “air time,” the facilitator was required on occasion to directly ask some participants to yield the floor.  At other times, it was important for the facilitator to single out quieter participants to elicit their response.  Facilitation was also required in all three pilot sites to repeatedly encourage participants to consider all options and to challenge one another’s assumptions. However, important debate about past events warranted the need to manage internal disagreement.  Therefore, Look-Back facilitators are required to be adept mediators.⁸  

All three pilot SPHAs recognized that facilitator independence is critical to the success of the Look-Back, and they echoed the need for facilitators who are external to the public health department. Specifically, the SPHAs independently agreed that a public health department employee is unlikely to effectively serve as the kind of facilitator described here as a result of an inherent conflict of interest that would undermine his/her ability to avoid bias and ensure that all sides are adequately heard.

3.  It is a challenge to produce effective and broadly agreed-upon AARs.

By design, a Look-Back (and the resulting Action Plan and AAR) does not seek to capture historical “truth,” but, rather, perceptions of what occurred in the past.  To this end, in a group of diverse individuals, there will necessarily be competing perspectives and sensitivities, and not all relevant players may be present on the day of the discussion to represent their views.  These factors can conspire to create important tension between developing a candid AAR that actually reflects the discussion on the day of the Look-Back and a more politically sensitive document that may be accepted more widely but that may not be as successful in stimulating change. 

Across all three sites, we found that the resulting AAR (which included initial improvement steps, to the degree that participants were willing and able to accept responsibility for a future action item) caused some degree of controversy or debate.  For example, once the AAR had been circulated to Look-Back participants by the SPHA, key staff or community stakeholder participants on at least two occasions challenged some of the reported discussion as being factually inaccurate. On two occasions, participants or their superiors requested background supplements to the AAR that had not been addressed at the discussion, or that the AAR more accurately reflect historical “truth.” 

SPHAs that are planning on conducting their own Look-Backs may want to clarify at the start of the Look-Back that the discussion and the resulting AAR will specifically seek to reveal critical systems-level strengths and weaknesses in past routine annual influenza management, with the express hope of stimulating change.  Moreover, it may be useful to set some guidelines about the documentation of the Look-Back, e.g. that the SPHA will circulate the AAR, and while all panelists will have the opportunity to review it, the AAR will not be changed to accommodate revisions to restore historical accuracy.

4.  AARs can generate valuable dialogue if they are broadly disseminated and reviewed by individuals not typically involved in routine annual influenza activities.

We found that when the AAR was broadly disseminated by the SPHA across agency departments (e.g., public health, emergency preparedness) the contents of the AAR generated important attention from key individuals and departments that had not been involved in the Look-Back.  Specifically, we found in at least two cases that both individual and organizational tensions resulted from the functional separation of annual influenza management and influenza pandemic preparedness. In one particular case, the broad dissemination of the AAR resulted in a new level of engagement in routine annual influenza by the state epidemiologists who were functionally tied to the division of emergency planning and preparedness and who previously viewed routine annual influenza as outside of their purview.  

In general, we found that involving key individuals and functions that were not at the Look-Back and that did not participate in the initial development of the Action Plan and AAR may mitigate potential cross-division or jurisdictional sensitivities and foster increased goodwill and buy-in for changes that may have broad impact and reach.

 Chapter Five

Selected Lessons Learned by State Public Health Agencies

In addition to common overall design and implementation issues, we found that our three pilot SPHAs shared several important lessons and areas for improvement with respect to the 2004-2005 influenza season. That particular influenza season was characterized by influenza vaccine shortage, and as such, many of the lessons learned and areas for improvement focused on the management of vaccine shortages and related events.  Specifically, Look-Backs in all three SPHAs revealed that several areas of planning are particularly important for enhancing routine annual influenza management and pandemic preparedness in the future, including (1) leveraging state emergency preparedness resources and infrastructure; (2) establishing clear lines of communication within the SPHA and between and among SPHAs, LPHAs, healthcare providers, and the public; and (3) facilitating the distribution and administration of vaccine.

1.  Leveraging state emergency management resources and infrastructure may facilitate emergency response by “traditional” state public health agencies.

Despite some concerns about integrating public health and emergency management functions, the benefit of capitalizing on a broader range of existing state emergency preparedness resources and infrastructure to improve public health management of influenza emerged as one lesson from the 2004-2005 influenza season that is particularly relevant for future influenza seasons and pandemic influenza preparedness.  One SPHA commented that having the state’s public health emergency preparedness infrastructure in place greatly improved the health department’s capacity to respond.  Specifically, Health Resources and Services Administration (HRSA) funding in that state provided a link to emergency medical systems and hospitals, and because of that link, SPHA and hospital staff were familiar with the idea of ICS and its contributions to flexible and immediate response.⁹  

Although only one of the three SPHAs we visited formally activated ICS, thus activating emergency response, all three SPHAs explicitly considered using ICS to manage the influenza vaccine shortage during the 2004-2005 season. In the state in which ICS was activated, certain programs, such as radiation and nuclear control, were accustomed to responding to emergencies under a unified command hierarchy, but ICS had not been widely used to coordinate a response across other areas of public health.  When ICS was activated, staff members were reassigned to different functions, and a state call center was established to respond to public demand for information. 

Two SPHAs explicitly decided against ICS activation, and for similar reasons.  Although both states could have activated their Emergency Operations Center, both ultimately decided to handle the vaccine shortage within the state influenza program and the SPHA’s own channels of supervision.  SPHA leaders in both states were wary of ICS and did not want to “militarize” their response or cause public panic.  Broader emergency preparedness assets and systems were therefore not involved in the resulting response, although one local district used the 2004-2005 influenza season as an opportunity to train and practice ICS.  In general, ICS was felt to be too foreign to the SPHAs’ staff responsible for managing routine annual influenza and counter to the more consensus-oriented approach of public health. Despite their concerns, these SPHAs recognized the potential benefit of emergency management resources and specified the following in their action and improvement plans:

• Resolve outstanding internal debate within the SPHA and among key partners about the feasibility and utility of ICS to better manage future influenza vaccine shortages and other unanticipated events.
• Determine whether there are other emergency preparedness and response resources or methods that can be utilized to empower the SPHA in future influenza seasons or a pandemic emergency to mobilize community stakeholders.
• Develop a plan well before influenza season about whether and when ICS should be initiated, who will make that decision, who will be involved, and what their roles will be. 
• Determine whether a modified version of ICS (e.g., ICS with different levels of escalation tailored to public health needs) might assist the SPHA to move vaccine around the state under shortage scenarios, to motivate private providers to pool resources, to clarify organizational communication, and to mitigate individual stress.
• Although there is general consensus that, in a pandemic setting, emergency management functions will assert control and ICS will be implemented, plans and drills should be revisited to identify and test the specific point at which a pandemic will be declared and to ensure that the lines of communication and command and control for a pandemic are clearly designated.
• ICS should be exercised, especially to familiarize public health officials who are historically less accustomed to it.

Two SPHAs used public health emergency authority to issue directives for establishing rules and priorities for distribution and use of influenza vaccine and to manage public anxiety.  In addition, one SPHA used such a directive to require all health care providers, hospitals, pharmacies, etc., to report the number of influenza vaccine doses they had in stock via a Web-based reporting system.  These two SPHAs both found that while the directives were essential to supporting the initial rationing of limited vaccines, they did not prove to be adequately flexible to respond to the evolving situation. Specifically, when the priority groups changed mid-season, and when more vaccine became available, new directives were required, thus hindering the timeliness of the response.  One SPHA observed that, in future years, decisions about how and when to lift a public health emergency directive would be essential.

One state invoked the public health emergency authority for the first time since the early 1940s. Remnants of the old law that attached penalties for refusing to comply with the order were maintained in 2004 to reinforce the significance of the vaccine shortage and also to allow the state government to intervene (e.g., if an employer attempted to administer vaccine to employees in non-high-risk groups).  Backlash ensued from physicians who feared the order would prevent them from providing adequate care to their patients.  In another state, reliance on a public health emergency authority raised concerns that, in a pandemic emergency, a federal response would significantly diminish the authority of state, local, and private entities.  At the same time, however, vaccine administrators in both states seemed relieved that the SPHA had asserted critical leadership and authority. Moreover, we heard that providers in at least one state welcomed the cover of an official legal order for denying vaccine to patients who were not in high-risk groups.

Given the important lessons, resources, and infrastructure that emergency preparedness can offer for the management of routine annual influenza, key individuals from emergency management and preparedness functions should be included in the Look-Backs.  In some cases, emergency management personnel participated.  When representatives from the state’s public health emergency preparedness infrastructure did not participate, it was either because they were not invited or because they were unable to attend.  In one case, key individuals were identified as important after the Look-Back, as a result of the circulation of the AAR.

2.  Communication is of paramount importance.

A second consistent lesson emerging from the 2004-2005 influenza season and one that is applicable to future routine annual and pandemic influenza planning was the importance of clear communications within SPHAs and between and among SPHAs, the CDC, LPHAs, healthcare providers, the media, and the public at large.  Most challenging across all three SPHAs in the 2004-2005 influenza season was communicating with the CDC and managing communication and messaging with the media and healthcare providers.  Improving interdepartmental communication (e.g., between public health and emergency preparedness divisions) also emerged as a consistent theme.

In the Look-Backs, all SPHAs and their community partners expressed frustration about communications to and from the Federal Government.  SPHAs reported frequent calls from the CDC with conflicting messages, requests to survey and resurvey, and a consistent unavailability of informed staff to answer questions.  At least two SPHAs reported that the CDC did not forewarn the SPHA about the shortage, but, instead, communicated directly with the national public.  For example, one SPHA reported that media coverage of the vaccine shortage resulted in a large volume of calls from the community, well before the SPHA was informed of the shortage by CDC.  The SPHA felt pressure to respond to the community’s concerns as well as to those of the providers regarding who would receive vaccine, but when the CDC announced its recommendations about priority groups, the SPHA was forced to change its messages.  This experience was felt to be extremely disempowering at a time when SPHAs were working to maintain credibility.

The importance of partnerships with the media in the 2004-2005 influenza season also emerged as a consistent lesson across all three SPHAs. In all SPHAs historically, there is frustration about how public health messages are delivered to the public, especially regarding routine annual influenza.  The media play a significant role in disseminating messages to the public. But, according to participants at the three Look-Backs, messages are often skewed or counterproductive.   For example, in one state, a primetime news hour presented a fairly grim picture of the influenza clinics (e.g., long lines, people on oxygen having to wait) that was thought to account for low turnout the following day. Further damaging to public opinion, the media could not be summoned back to change the misperception once the clinics were running smoothly.  This experience reinforced the need for consistent and carefully channeled messaging with the media, which would be especially critical in a pandemic.

Communicating with providers proved equally challenging and important.  Generally, there was concern that the provider community tended to minimize the significance of routine annual influenza and, consequently, did not internalize and convey several important messages. Specifically, the SPHAs reported that providers could have done more to communicate with patients about the effectiveness of FluMist in influenza prevention and about the benefit of influenza vaccination even later in the influenza season.  For example, once supplies of vaccine increased in December and January during the 2004-2005 season, providers generally did not encourage patients who became eligible to receive the vaccine to get immunized.  In an effort to step up communication with private providers midway though the season, one SPHA sent alerts via email and blastfax to its database of practicing physicians.  Although the SPHA reached 70 to 80 percent of its database, the database was not current, and it represented only a small portion of the state’s private providers.  Moreover, the messages were felt to be ineffective because of the diminished sense of urgency surrounding influenza in the physician community.  

Regular statewide conference calls between a high-level state official and LPHAs surfaced as a major success story in the management of the previous influenza season.  These regular calls provided a critical sense of leadership and were effective in reassuring local public health officials during the vaccine shortage.  The development of regional coalitions promises to provide additional infrastructure for enhancing coordination and communication between state and local health departments, as well as among locals. 

Discussion at the Look-Backs about communication issues resulted in the following action and improvement items:

• Improve communication with the CDC, address concerns about state versus federal roles in defining target groups during times of vaccine shortages or during a pandemic or other public health emergency situation.   Moreover, encourage CDC to implement a national education campaign about the importance of regular vaccinations and to create messages that the states can tailor for their own use.
• Improve communication with the media so that reporters are a more positive facilitator for communication with the public. For instance, partner with media outlets each August to inform providers and the public about plans for the upcoming influenza season (e.g., respiratory etiquette, hand washing, social-avoidance measures). 
• Explore mechanisms for obtaining and maintaining current contact information for private providers and for improving connection to the Health Alert Network (HAN) in the private provider community. 
• Develop a statewide immunization strategy that focuses on improved communication with the large network of private providers.  An initial action item is to organize a group of representative providers (e.g., in a focus group setting) to convey their importance in the process and to enlist their support and solicit their insight and feedback. 
• Develop and implement an education campaign directed at private providers about the production and distribution of vaccine, the importance of annual immunizations, the value of FluMist, and the role of other public health measures (e.g., respiratory hygiene; hand washing in influenza prevention) to generate their interest in active community participation in influenza planning.  
• Improve interdepartmental (e.g., public health and emergency preparedness) communication, coordination, participation, roles, and responsibilities under vaccine shortage or pandemic conditions.

3.  Broad-based coalitions and public-private partnerships may mitigate challenges SPHAs confront with vaccine distribution and administration.

A large majority of influenza vaccine in the United States is purchased privately and distributed through influenza clinics and hospitals, large healthcare providers, private practitioners, employers, and commercial entities. Consequently, the amount of publicly-held vaccine is typically quite limited, leaving many SPHAs with limited influence over the distribution of influenza vaccine. For example, in two of our pilot states, the SPHA purchases only 5 to 10 percent of influenza vaccine in a typical year.  One of the pilot SPHAs, however, was an outlier in that it typically purchases approximately half of the influenza vaccine.

Irrespective of the amount of vaccine purchased by each SPHA, a real success story of the 2004-2005 influenza season was the role of public-private partnerships in redistributing privately purchased vaccine.  When the influenza vaccine shortage was announced, all three SPHAs worked diligently through a variety of mechanisms to partner with key stakeholders to track vaccine in the state, pool resources, and redistribute vaccine to priority groups.  In one state, the delay in influenza vaccine in 2000 motivated the formation of an influenza vaccine consortium made up of representatives from hospitals, health plans, nursing homes, and provider groups, as well as the SPHA.  The express purpose of the consortium was to provide a mechanism for sharing and distributing vaccine throughout the state as needed, and to promote adult immunizations, particularly against influenza and pneumococcal disease. This consortium proved to be of great assistance in responding to the vaccine shortage in 2004-2005.

Another SPHA administered a survey of private providers to directly estimate vaccine purchases, but low response rates and poor reporting thwarted this effort.  For instance, physicians in that state overinflated their estimates, claimed they did not receive vaccine, and/or refused to give up their supply, even if they reported a surplus.  These barriers notwithstanding, the SPHA persevered, and, through discussions and meetings with community partners and through surveys and a variety of other sources, by early November 2004 had estimated the amount of influenza vaccine doses that had been received by physicians and health care organizations. Armed with this information, the SPHA encouraged private providers to administer vaccine according to the CDC recommended priority groups.  In addition, in conjunction with their community partners, the SPHA developed a plan to distribute publicly held vaccine to the Vaccines for Children (VFC) program and to high-risk indigent adults seen at community health clinics.

Building on the lessons of these public-private partnerships, the Look-Backs identified the following lessons and actions:

• Consider, in advance of influenza season, how best to engage and mobilize community partners and stakeholders in tracking vaccine and pooling resources (e.g., through a community influenza consortium) under future vaccine shortage conditions or during a pandemic. 
• Include the private sector in immunization coalition efforts.
• Encourage private providers to become more willing partners in a future vaccine shortage or pandemic setting by educating them about the production, supply, and distribution of vaccine in the United States, and about the often-limited role of SPHAs in moving vaccine.
• Enlist individual hospitals through the state HAN to help with vaccine redistribution in a future vaccine shortage or pandemic emergency.  Currently, many hospitals view immunizations as a public health function; if there is a role for hospitals to play in routine annual influenza season, they require education.

The above-described lessons and action items emerged from SPHA Look-Backs and demonstrate the utility of Look-Backs as a relatively simple, effective tool that any level of jurisdiction can use to systematically assess actual recent past events to strengthen management and communication systems relevant to future routine annual and pandemic influenza.  Adoption and implementation of Look-Backs with regular frequency (i.e., annually) by public health agencies will capitalize on routine annual influenza to better prepare for pandemic influenza (a rare opportunity in public health); document and formalize learning from successes as well as from problems; encourage follow-through on lessons learned; and reinforce the role of public health during annual and pandemic influenza, as well as other public health emergencies.

 Appendix A.  Look-Back Planning Checklist

This Appendix provides advance-planning and logistical checklists, a list of suggested participants, and sample agenda for SPHAs planning to conduct their own Look-Back.

STAFF

Two individuals are needed to conduct a Look-Back:

• A facilitator, who conducts the discussion and, as necessary, offers probes to the participants. Our limited experience with three SPHAs suggests that an outside, neutral, or professional facilitator is most likely to create an atmosphere that fosters open and honest dialogue among participants.
• A note taker, to record the discussion, keep track of lessons learned and applications for pandemic influenza, and assist in the development of the Action Plan and after action report (AAR).

ADVANCE PLANNING AND PREPARATION

Assuming that the facilitator does not work within the public health agency, he/she should collaborate with the leaders and key staff in advance of the Look-Back to customize the discussion to reflect state (or local) experiences with routine annual influenza and to meet specific needs of the SPHA and its community stakeholders.  The following are specific steps that the facilitator can take to tailor a Look-Back to a public health agency’s needs:

• Once a date and time have been determined for the Look-Back, gather as much background information as possible about the state’s activities during a typical influenza season in a way that minimizes staff burden (e.g., relevant background materials can often be found online).
• Learn about the organizational structure of the SPHA (i.e., various divisions and lines of authority).
• Conduct a short (approximately one hour) meeting with one or two public health agency and/or immunization program officials who were intimately involved in routine annual influenza season activities in advance of the Look-Back. This meeting serves to:
• 1. Clarify any outstanding questions about influenza season management and identify salient topics for the upcoming discussion(see Chapter Two), e.g.:
  2. • In a typical influenza season, who are the key players internally within the public health agency and in the community, and what are their activities, roles, and responsibilities?
     • What were the most critical challenges in the management of the last influenza season (e.g., surveillance, vaccine procurement, communication)?
     • Considering both routine annual influenza and pandemic preparedness activities, what are specific issues that would benefit from in-depth review?
  3. Solicit a list of participants,¹⁰ e.g.: 
  4. • State Public Health Agency officials and staff typically involved in key aspects of managing influenza season:
     • • State health director
       • Emergency management coordinator
       • Immunization program director
       • Pandemic influenza coordinator
       • Communicable disease control/disease investigation director
       • U.S. Centers for Disease Control and Prevention (CDC) Field Officer
       • Quality Improvement Coordinator
       • Public health laboratory representative
       • State epidemiologist
       • Public health nurse
       • Pharmacist
       • Communications specialist/public information officer
       • Emergency management official
  5. Other community partners who assisted with key activities, administered vaccine, and/or played a significant role during the past influenza season:
  6. • District or local public health agency staff
     • Hospital representatives
     • Nursing home and long-term care representatives
     • Representatives from various professional medical organizations (e.g., American Medical Association affiliates, Academy of Pediatrics, Academy of Family Physician affiliates)
     • Managed care organizations (e.g., Kaiser, GroupHealth)
     • Insurers (e.g., Blue Cross)
     • Representatives from commercial enterprises offering influenza vaccine to the public (e.g., grocery stores, Long’s Drugs)
     • Pharmacies
     • Minority community leadership representatives
     • Military officials
     • American Association of Retired Persons (AARP) and other advocacy groups
     • University-based health services
     • Emergency response groups that might have a role to play in influenza planning (e.g., Business Executives for National Security [BENS] affiliates)
     • Business community leaders, including large employers that provide vaccinations for their employees through worksite wellness programs
  7. Identify specific individuals who may be enlisted by the public health agency to provide a very brief (i.e., two-minute) overview of the typical and relevant influenza activities surrounding each topic selected for discussion. Assigning individuals ahead of time to kick off each discussion provides participants with a common ground for each discussion and engages a variety of participants.

AT THE LOOK-BACK

• Introduce Look-Back objectives and agenda and provide participants with assurance about confidentiality and anonymity (see Appendix B for a sample presentation).
• Ensure that detailed notes are taken.
• At the end of the discussion, solicit specific lessons learned and initiate the development of an Action Plan for improvement.
• Close the meeting by asking participants to fill out an evaluation form to solicit feedback about the process and specific suggestions for the next Look-Back.

ADDITIONAL LOGISTICS

• Conference room able to seat 15 to 25 people in an open, U-shaped formation
• Computer with Microsoft (MS) PowerPoint and projector
• Whiteboard, flipcharts, or chalkboard
• Copies of agenda and evaluation form for participants.  

 Figure A.1. Sample Agenda

 Appendix B.  Sample Look-Back Presentation

The following are suggested presentation slides that may be used to guide the Look-Back.  The topics for discussion are illustrative, not prescriptive.

 Appendix C.  Template for an After Action Report

General Information

Date and Time: 
Location: 
Participants: [List of De-Identified Participants, i.e., by title only]

Overview

On [date], [number] individuals from the [SPHA], [other organizations] participated in a facilitated Look-Back at the [year] influenza season.  The objective of the Look-Back is to assist SPHAs to regularly improve routine annual influenza season and pandemic preparedness.  This Look-Back was designed in conjunction with officials at [SPHA] in a pre-Look-Back teleconference held on [date]. The Look-Back took place at [location].

Goals and Objectives

Key goals of the Look-Back were to trace the series of events that unfolded during the [year] influenza season, to identify key decisions that were made by various stakeholders, and to learn how the key decisions were perceived and acted upon by others. Specifically, the Look-Back sought to:

• Conduct a systems-level, no fault analysis of state responses to the [year] influenza season
• Identify lessons applicable to routine annual influenza, as well as to pandemic influenza and other public health emergencies
• Identify opportunities for improvement and an action plan.

Topics and Core Questions

The Look-Back focused on [number] influenza season topics that were selected with [SPHA] officials in the pre-Look-Back teleconference. These broad topics included:  

• [Topic No. 1]
• [Topic No. 2]
• [Topic No. 3] , etc.

For each of the above-listed topics, the discussion focused on the following core questions:

• What are the activities, roles, and responsibilities during the routine annual influenza season?
• What are specific issues that came up this past influenza season? 
• What specific problems emerged? How were they resolved?
• What went well?
• What might have been done differently?
• What should be done differently in the future?
• What are lessons for a pandemic?

Review of Discussion

[Topic No. 1]

Activities, Roles, and Responsibilities During a Typical Influenza Season

Brief Review of What Occurred in [Year]

Lessons Learned from Last Year’s Events and Action Items that Came Up During Discussion

• [Briefly list lessons in bullet form]

Implications for Future Routine Influenza Season

• [Briefly list lessons in bullet form]

Implications for Future Pandemic Influenza Management

• [Briefly list lessons in bullet form]

[Topic No. 2]

Activities, Roles, and Responsibilities during a Typical Influenza Season

Brief Review of what Occurred in [Year]

Lessons Learned from Last Year’s Events and Action Items that Came Up During Discussion

• [List lessons in bullet form]

Implications for Future Routine Influenza Season

• [Briefly list lessons in bullet form]

Implications for Future Pandemic Influenza Management

• [Briefly list lessons in bullet form]

Summary

Action Items Suggested by the Group

• [Briefly list action items in bullet form]

 References

Bhat, N., J. G. Wright, K. R. Broder, E. L. Murray, M. E. Greenberg, et al., “Influenza-Associated Deaths Among Children in the United States, 2003-2004,” New England Journal of Medicine, Vol.353, No. 24, 2005, pp. 2559-2567.

Dausey, D. J., N. Lurie, A. Diamond, B. Meade, et al., Bioterrorism Preparedness Training and Assessment Exercises for Local Public Health Agencies,  Santa Monica, Calif.: RAND Corporation, TR-261-DHHS, 2005.  Available online at http://www.rand.org/publications/TR/TR261/.

Office for Domestic Preparedness (ODP), The Homeland Security Exercise and Evaluation Program. Available online at http://www.ojp.usdoj.gov/odp/docs/hseep.htm.

Schoch-Spana, M., “Hospitals Buckle During Normal Flu Season: Implications for Bioterrorism Response,” Biodefense Quarterly, Vol. 1, No. 4, March 2000.

Thompson, W. W., D. K. Shay, E. Weintraub, et al., “Influenza-Associated Hospitalizations in the United States,” Journal of the American Medical Association, Vol. 292, 2004, pp. 1333-1340.

U.S. Centers for Disease Control and Prevention, Influenza (Flu). Available online at http://www.cdc.gov/flu/.

U.S. Food and Drug Administration. Availability of Influenza Virus Vaccine 2000–2001. Available online at  http://www.fda.gov/CBER/flu/flu2000.htm

U.S. Government Accountability Office (GAO), Influenza Pandemic: Applying Lessons Learned from the 2004-05 Influenza Vaccine Shortage, Washington, D.C.: GAO-06-221T, November 4, 2005.  Available online at www.gao.gov/cgi-bin/getrpt?GAO-06-221T.

 Copyright Information: Rand Corporation

The research described in this report was prepared for the U.S. Department of Health and Human Services. This research was produced within RAND Health’s Center for Domestic and International Health Security. RAND Health is a division of the RAND Corporation.

The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors.

R® is a registered trademark.

A profile of RAND Health, abstracts of its publications, and ordering information can be found on the RAND Health home page at www.rand.org/health.

© Copyright 2006 RAND Corporation

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from RAND.

Published 2006 by the RAND Corporation
1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138
1200 South Hayes Street, Arlington, VA 22202-5050
4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213
RAND URL: http://www.rand.org/
To order RAND documents or to obtain additional information, contact
Distribution Services: Telephone: (310) 451-7002;
Fax: (310) 451-6915; Email: order@rand.org

----------------------------------------------------------------