Abstract

The emergence of pandemic H1N1 influenza in 2009 has prompted public health responses, including production and licensure of new influenza A (H1N1) 2009 monovalent vaccines. Safety monitoring is a critical component of vaccination programs. As proof-of-concept, the authors mimicked near real-time prospective surveillance for prespecified neurologic and allergic adverse events among enrollees in 8 medical care organizations (the Vaccine Safety Datalink Project) who received seasonal trivalent inactivated influenza vaccine during the 2005/06–2007/08 influenza seasons. In self-controlled case series analysis, the risk of adverse events in a prespecified exposure period following vaccination was compared with the risk in 1 control period for the same individual either before or after vaccination. In difference-in-difference analysis, the relative risk in exposed versus control periods each season was compared with the relative risk in previous seasons since 2000/01. The authors used Poisson-based analysis to compare the risk of Guillian-Barré syndrome following vaccination in each season with that in previous seasons. Maximized sequential probability ratio tests were used to adjust for repeated analyses on weekly data. With administration of 1,195,552 doses to children under age 18 years and 4,773,956 doses to adults, no elevated risk of adverse events was identified. Near real-time surveillance for selected adverse events can be implemented prospectively to rapidly assess seasonal and pandemic influenza vaccine safety.

In April 2009, the first cases of pandemic H1N1 influenza were confirmed in several countries (1). The World Health Organization declared a pandemic in June 2009 (2), and influenza A (H1N1) 2009 monovalent vaccines were licensed in September 2009 (3). The Advisory Committee on Immunization Practices recommended initial target groups for these vaccines (4), recommended annual seasonal influenza vaccination for approximately 83% of the US population (5) (255 million children and adults), and may continue to incrementally expand the recommendations (6). Given rapid deployment of 2009 H1N1 vaccines and broad recommendations for seasonal influenza vaccines, monitoring vaccine safety in near real-time is critical (7, 8).

Current systems for evaluating vaccine safety have limitations. First, passive reporting systems, such as the Vaccine Adverse Event Reporting System, allow identification of potential adverse events but tend to underreport numbers of adverse events, lack denominator data, and are not designed to distinguish coincidental associations from potentially true causal relations (9). Second, traditional postlicensure studies can rigorously evaluate potential associations between vaccines and adverse events, but they do not monitor safety in near real-time, since they often require substantial time to design and implement, resulting in delays in the ability to detect possible adverse events. If analyses are conducted after the influenza season is over, potential problems will be identified only after all doses from that season's vaccine have already been administered.

In several recent studies, investigators have addressed this limitation of timeliness by comparing adverse event risks after administration of a vaccine of interest with risks following administration of a comparison vaccine or a preventive care visit in near real-time, using weekly data and a Poisson-based maximized sequential probability ratio test to adjust for repeated analyses (10–13). However, selecting a comparison group can be particularly problematic for influenza vaccine, because of confounding by indication (14); influenza vaccine recipients are often systematically different from nonvaccinees (e.g., by functional status limitations, such as requiring assistance to bathe) in ways that may influence the incidence of adverse events and for which adjustment may not be possible using solely automated data (15). In addition, vaccination recommendations have expanded over time (5), resulting in changing characteristics of vaccinated and unvaccinated populations. Furthermore, vaccinees may be misclassified as unvaccinated in electronic data, particularly if vaccines are administered in nontraditional settings (e.g., workplaces) (16). Self-controlled case series (SCCS) (17, 18) and difference-in-difference (DID) (19) approaches compare risk and control periods within vaccinees, minimizing these potential biases and controlling for known confounders (e.g., age and sex) as well as unmeasured fixed covariates (e.g., race, ethnicity, socioeconomic status, and underlying disease).

In previous postlicensure studies, investigators have compared risk and control periods within persons receiving influenza vaccine (20–22) or have conducted sequential analyses to monitor adverse events following administration of vaccines other than influenza vaccine (10–13). In this report, we unite these methods by applying maximized sequential probability ratio testing to comparisons of risk and control periods within vaccinees. As proof-of-concept in preparation for pandemic and seasonal influenza vaccine safety monitoring, our objective was to mimic prospective surveillance for prespecified adverse events following seasonal influenza vaccine administration, using binomial-based (SCCS and DID) and Poisson-based analytic approaches.

MATERIALS AND METHODS

Study design

We used 3 approaches to conduct safety surveillance for the first seasonal dose of trivalent inactivated influenza vaccine (TIV). The SCCS approach (17, 18) was used to investigate whether a predefined period following TIV administration in a particular season was riskier than a control period. The SCCS approach may be limited by the “healthy vaccinee” effect (23), since some patients may delay or forgo routine influenza vaccination shortly after experiencing an illness classifiable as an adverse event. To minimize this effect, the control period for each adverse event category excluded the 14 days prior to vaccination. Unlike another case-only method, the case-crossover design (24), SCCS does not require the assumption that the underlying probability of vaccination is the same in all intervals (25), which is unlikely to hold for seasonally administered vaccines such as influenza.

The DID approach was used to investigate whether TIV in a particular season was riskier than TIV in cumulative previous seasons, using control periods for both current and previous seasons. It is an extension of the SCCS approach, first applied in economics research (19). The DID approach adjusts for the healthy vaccinee effect, assuming that an adverse event's effect on subsequent vaccination is the same in the current season as in previous seasons. This assumption may not always be valid—for example, if a medical care organization changes its policy for vaccinating after the occurrence of a particular adverse event between seasons. In addition, if the DID approach compares the relative risk of adverse events following pandemic influenza vaccine with the relative risk following TIV, the appropriateness of the comparison may be limited if the populations receiving pandemic vaccine and TIV are substantially different (e.g., according to underlying risk conditions). This approach differs from simply comparing risk periods in the current season with those in previous seasons without control periods; this would adjust for the healthy vaccinee effect but could be biased by secular trends in risk unrelated to vaccination (26).

Finally, a Poisson-based approach was used to compare the risk of Guillian-Barré syndrome following TIV in the current season with that in previous seasons. This analytic approach has been previously described (27) and (unlike SCCS and DID) previously applied for near real-time surveillance within the Vaccine Safety Datalink (VSD) Project (10–13).

We classified instances of adverse events as occurring within a risk period following TIV, when association with vaccination was plausible, or occurring within a control period before vaccination (“control period 1”) or after the risk period following vaccination (“control period 2”), when the adverse event was more plausibly attributable to causes other than vaccination. Only vaccinees with an adverse event were included in analysis, and only if an adverse event occurred in either the risk period or the control period but not both.

Study population

Established in 1990 to monitor and evaluate vaccine safety, the VSD Project collects medical care and vaccination data on over 9 million members annually enrolled in 8 participating medical care organizations (28, 29). Staff at each VSD site prepare data files containing demographic and medical information on the members, such as age, sex, enrollment, vaccinations, and International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes assigned to medical encounters occurring in outpatient, emergency department, and hospital settings.

Children aged 6 months–17 years and adults aged ≥18 years were eligible if they 1) received TIV during the weeks of September 1 through April 30 in the 2005/06, 2006/07, or 2007/08 influenza seasons, 2) experienced a predefined adverse event during either the risk period or the control period in any season, and 3) were enrolled at the time of vaccination in a medical care organization participating in the VSD Project. Uniformly formatted files with immunization and diagnosis data are updated weekly and include information on VSD site, age, sex, week of TIV administration, adverse event type according to ICD-9 codes, number of days between TIV administration and the adverse event, and medical setting in which the adverse event was diagnosed.

The 8 medical care organizations within the VSD Project are: Group Health Cooperative, Seattle, Washington; Harvard Pilgrim Health Care, Boston, Massachusetts; HealthPartners Research Foundation, Minneapolis, Minnesota; Kaiser Permanente of Colorado, Denver, Colorado; Kaiser Permanente of Northern California, Oakland, California; Kaiser Permanente of Southern California, Pasadena, California; Marshfield Clinic Research Foundation; Marshfield, Wisconsin; and Northwest Kaiser Permanente, Portland, Oregon. The Centers for Disease Control and Prevention and the institutional review board at each site provided approval and agreed that informed consent was not required.

Adverse events

The adverse events evaluated included seizures (30), meningoencephalitis (31–33), Bell's palsy (34, 35), other cranial nerve disorders, demyelinating disease (32, 36, 37), disorders of the peripheral nervous system and neuropathy (32, 38), ataxia (39), anaphylaxis (40, 41), allergic reaction other than anaphylaxis (including angioneurotic edema and urticaria) (42), and Guillian-Barré syndrome (43–45) (Table 1). These neurologic (46) and allergic (47) adverse events were selected because they are clinically well-defined, are serious, and either had already been observed in published studies or passive surveillance or were biologically plausible as a consequence of TIV vaccination (e.g., anaphylaxis via immunoglobulin E-mediated hypersensitivity) (48).

Table 1.

Definitions of Adverse Events Used for Influenza Vaccine Safety Surveillance, Vaccine Safety Datalink Project, 2005/06–2007/08

Adverse Event Category ICD-9 Inclusion Code(s)a ICD-9 Exclusion Codes (if Event Occurred ≤7 Days After Diagnosis) Medical Setting Risk Period Postvaccination, days Control Period Pre- or Postvaccination, days First Such Event in What Period and Medical Setting? 
Seizure 345.0*–345.9*, 780.3, 780.31, 780.39 None Inpatient, ED 0–7 8–15 days postvaccination, same season 8 days in outpatient, inpatient, ED 
Meningoencephalitis 047.8, 047.9, 049.9, 321.2, 322*, 323.5–323.9 047.0–047.1, 048, 049.0–049.8, 053*–056*, 320* Inpatient, ED 1–21 15–74 days prevaccination, same season 1 year in inpatient, ED 
Bell's palsy 351.0 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Other cranial nerve disorder 350*, 351.1, 351.8, 351.9, 352* None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Demyelinating disease 340*, 341.0, 341.8, 341.9, 377.30, 377.31, 377.32, 377.34, 377.39 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Disorder of the peripheral nervous system 337.0, 337.9, 354.1–354.9, 355*, 356.4, 356.8, 357.6, 357.7, 357.8*, 357.9 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Ataxia 334.3 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Anaphylaxis 995.0, 999.4 None Inpatient, ED 0–2 7–9 days postvaccination, same season 6 months in inpatient, ED 
Allergic reaction other than anaphylaxis 995.1, 995.3, 708.0, 708.1, 708.9 None Outpatient, inpatient, ED 1–2 8–9 days postvaccination, same season 6 months in outpatient, inpatient, ED 
Guillian-Barré syndrome 357.0 None Outpatient, inpatient, ED 1–42 1–42 days postvaccination, previous seasons 1 year in outpatient, inpatient, ED 
Adverse Event Category ICD-9 Inclusion Code(s)a ICD-9 Exclusion Codes (if Event Occurred ≤7 Days After Diagnosis) Medical Setting Risk Period Postvaccination, days Control Period Pre- or Postvaccination, days First Such Event in What Period and Medical Setting? 
Seizure 345.0*–345.9*, 780.3, 780.31, 780.39 None Inpatient, ED 0–7 8–15 days postvaccination, same season 8 days in outpatient, inpatient, ED 
Meningoencephalitis 047.8, 047.9, 049.9, 321.2, 322*, 323.5–323.9 047.0–047.1, 048, 049.0–049.8, 053*–056*, 320* Inpatient, ED 1–21 15–74 days prevaccination, same season 1 year in inpatient, ED 
Bell's palsy 351.0 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Other cranial nerve disorder 350*, 351.1, 351.8, 351.9, 352* None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Demyelinating disease 340*, 341.0, 341.8, 341.9, 377.30, 377.31, 377.32, 377.34, 377.39 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Disorder of the peripheral nervous system 337.0, 337.9, 354.1–354.9, 355*, 356.4, 356.8, 357.6, 357.7, 357.8*, 357.9 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Ataxia 334.3 None Outpatient, inpatient, ED 1–42 15–74 days prevaccination, same season 1 year in outpatient, inpatient, ED 
Anaphylaxis 995.0, 999.4 None Inpatient, ED 0–2 7–9 days postvaccination, same season 6 months in inpatient, ED 
Allergic reaction other than anaphylaxis 995.1, 995.3, 708.0, 708.1, 708.9 None Outpatient, inpatient, ED 1–2 8–9 days postvaccination, same season 6 months in outpatient, inpatient, ED 
Guillian-Barré syndrome 357.0 None Outpatient, inpatient, ED 1–42 1–42 days postvaccination, previous seasons 1 year in outpatient, inpatient, ED 

Abbreviations: ED, emergency department; ICD-9, International Classification of Diseases, Ninth Revision.

a

Asterisks in ICD-9 codes indicate that any digits after the asterisk were included.

The length of the risk period (Table 1) was adverse event-specific and depended on the timing of adverse events relative to receipt of vaccine in published studies (30, 34, 49) and biologic plausibility. The day of vaccination was included in the risk period only for those adverse events for which a same-day diagnosis was biologically plausible (e.g., anaphylaxis) and, for improved specificity, only for adverse events diagnosed in inpatient or emergency department settings. Certain conditions (e.g., urticaria) that are listed as diagnoses in the outpatient setting are often chronic diagnoses present before vaccination (50).

For most adverse event categories, the control period was selected to occur before the risk period to facilitate timely analyses. However, control periods were selected to occur after the postvaccination risk period (control period 2) for 3 adverse event categories, because 1) a recent prior adverse event might preclude vaccination (i.e., allergic reaction and anaphylaxis) or 2) a person might have an underlying condition that is also an indication for vaccination (i.e., seizure disorder).

For most adverse event categories, an adverse event was included in analysis only if it was the first event of its type to occur in a year, irrespective of vaccination timing (Table 1). This restriction ensured that 1) the specificity of adverse event definitions was enhanced, since follow-up visits for preexisting conditions were not interpreted as new-onset events, and 2) multiple events of the same type could not be counted during a single observation period. For adverse events that could plausibly have more than 1 new-onset episode in a year (e.g., seizure), the event was included in analysis if it was the first event in a shorter time period.

Statistical analysis

In the SCCS approach, the null hypothesis assumes that the risk of an adverse event's occurrence on a day during the exposed risk period following the first TIV dose in a season is the same as the risk of its occurrence on a day during the unexposed control period. For the current season under surveillance, c represents the count of adverse events in the control period, d represents the count of adverse events in the postvaccination risk period (Figure 1, part A), and r represents the ratio of c to d expected under the null hypothesis, based on the ratio of the number of days in the control period to that in the postvaccination period. The estimate of the relative risk is dr/c, which changes as data accumulate weekly. As Kulldorff et al. (27) previously described, the likelihood under the null hypothesis is L0 = (1/[r + 1])d × (r/[r + 1])c, and the likelihood under the alternative hypothesis is L1 = (d/[c + d])d × (c/[c + d])c. The log-likelihood ratio is then ln(L1/L0) = d ln(d/[c + d]) + c ln(c/[c + d]) – d ln(1/[r + 1]) – c ln(r/[r + 1]).

Figure 1.

Risk and control periods for an example potential adverse event proximal to vaccination in the 2007/08 influenza season. The cumulative number of events occurring during each period, summed across previous seasons, is represented by a and b. The number of events occurring during the current season is represented by c and d. A) In the self-controlled case series approach, d is compared with c. B) In the difference-in-difference approach, d is compared with c, adjusting for the comparison of b with a.

Figure 1.

Risk and control periods for an example potential adverse event proximal to vaccination in the 2007/08 influenza season. The cumulative number of events occurring during each period, summed across previous seasons, is represented by a and b. The number of events occurring during the current season is represented by c and d. A) In the self-controlled case series approach, d is compared with c. B) In the difference-in-difference approach, d is compared with c, adjusting for the comparison of b with a.

In the DID approach, the null hypothesis is that the relative risk of an adverse event's occurrence in the risk period versus the control period during the current influenza season is no greater than the same relative risk across cumulative previous seasons. The vaccine in previous seasons was assumed to have an acceptable risk profile (5). The estimate of the relative risk from previous seasons was based on at least 5 seasons and was treated as known without error (51); however, DID analyses were conducted only for adverse events with 25 or more cases in the risk and control periods in previous seasons. Accumulated from vaccinated cases across previous seasons, a represents the number of adverse events in the control period and b represents the number of adverse events in the postvaccination risk period (Figure 1, part B). For DID, the estimate of the relative risk ratio is (d/c)/(b/a). The likelihood is L0 = (b/[a + b])d × (a/[a + b])c, while L1 is identical to that of the SCCS approach. Therefore, the log-likelihood ratio is ln(L1/L0) = d ln(d/[c + d]) + c ln(c/[c + d]) – d ln(b/[a + b]) – c ln(a/[a + b]).

For Guillian-Barré syndrome, we used a Poisson-based approach (27) to compare the risk following TIV in the current season with that in previous seasons (relative risk = [d/doses in current season]/[b/doses in previous seasons]). SCCS and DID analyses were not conducted for this adverse event; since the occurrence of Guillian-Barré syndrome within 6 weeks of TIV is a precaution for receiving future TIV (5), fewer cases of Guillian-Barré syndrome may occur among vaccinees during a control period prior to vaccination. This can bias SCCS analysis, and there may be too few events to adjust for this bias using DID analysis. In contrast, a postvaccination control period matching the length of a 42-day risk period would not elapse until 84 days after vaccination, long after most vaccinations are administered.

Prospective weekly surveillance was mimicked using sequential analysis (10), with the strong assumption of complete data accrual without delay. To account for repeated weekly looks at the data, we applied the maximized sequential probability ratio test, using a binomial probability model for all adverse events except Guillian-Barré syndrome (27). The null hypothesis was rejected if the log-likelihood ratio reached an upper bound (the “critical value” of the log-likelihood ratio). The null hypothesis was not rejected if the total number of adverse events reached a prespecified “upper limit,” or if surveillance ended without reaching this upper limit and without the log-likelihood ratio reaching the critical value.

For each adverse event, specific to each age group and season, the critical value of the log-likelihood ratio was dictated by the user-specified upper limit of expected adverse events and α-level. Upper limits were selected on the basis of the approximate number of adverse events expected under the null hypothesis, assuming that the risk in the current season was no greater than the risk in cumulative previous seasons. For example, upper limits for 2005/06 were determined using the risk of adverse events in the VSD population during risk and control periods around influenza vaccination, accumulated across the 2000/01–2004/05 influenza seasons; upper limits for the following season, 2006/07, were determined using data accumulated across 2000/01–2005/06, etc. If upper limits were reached before all cases were accumulated for the season, the power would be lower than what would have been possible with a higher upper limit. Therefore, upper limits were chosen such that they would not usually be reached. The upper limits were identical between the SCCS and DID analyses.

For very rare adverse events (e.g., anaphylaxis), surveillance was conducted for all ages together. For less rare adverse events (e.g., seizures), surveillance was conducted for children and adults separately. Because many adverse events and age groups were considered, α = 0.01 was selected. We used 1-tailed tests, looking only for risks of vaccination rather than protective effects. Surveillance ended after the risk and control periods passed for April vaccinees; if the upper limit had not been reached by that point, the cumulative α value remained below 0.01, indicating a conservative test.

Statistical power for adverse events in the 2007/08 season was calculated using previously described simulations (27) and was determined by the ratio of days in the control period to days in the risk period, the stringent selection of α = 0.01, and predefined upper limits. Since surveillance could end before upper limits were reached, statistical power could only be approximated, assuming that surveillance lasted at least halfway to the upper limit (Table 2). For the Poisson-based Guillian-Barré syndrome analysis in 2007/08, the background incidence following TIV was assumed from previous seasons to be 8.5 per million doses. With the upper limit equal to 20 and α = 0.05, the power was 45% to detect a relative risk of 1.5 and 92% to detect a relative risk of 2.0 (27).

Table 2.

Statistical Power of the Binomial-based Maximized Sequential Probability Ratio Test for Different Adverse Events and True Relative Risks, Vaccine Safety Datalink Project, 2007/08

Adverse Event Category No. of Days in Control Period No. of Days in Risk Period Age Groupa Prespecified Upper Limit of No. of Events Rangeb of Statistical Power for α = 0.01, %
 
To Detect RR = 1.5 To Detect RR = 2.0 To Detect RR = 5.0 
Seizure Children 200 18–45 64–96 >99–100 
   Adults 500 57–92 99–>99 100 
Meningoencephalitis 60 21 All ages 150 12–28 47–84 >99–100 
Bell's palsy 60 42 Children 50 4–10 14–34 78–>99 
   Adults 1,000 92–>99 100 100 
Other cranial nerve disorder 60 42 Children 25 2–5 6–15 35–77 
   Adults 1,200 96–>99 100 100 
Demyelinating disease 60 42 Children 15 2–4 4–11 22–62 
   Adults 800 83–>99 >99–100 100 
Disorder of the peripheral nervous system 60 42 Children 80 6–15 23–56 97–100 
   Adults 2,000 >99–100 100 100 
Ataxia 60 42 All ages 80 6–15 23–56 97–100 
Allergic reaction other than anaphylaxis Children 200 18–45 64–96 >99–100 
   Adults 400 44–84 96–>99 100 
Adverse Event Category No. of Days in Control Period No. of Days in Risk Period Age Groupa Prespecified Upper Limit of No. of Events Rangeb of Statistical Power for α = 0.01, %
 
To Detect RR = 1.5 To Detect RR = 2.0 To Detect RR = 5.0 
Seizure Children 200 18–45 64–96 >99–100 
   Adults 500 57–92 99–>99 100 
Meningoencephalitis 60 21 All ages 150 12–28 47–84 >99–100 
Bell's palsy 60 42 Children 50 4–10 14–34 78–>99 
   Adults 1,000 92–>99 100 100 
Other cranial nerve disorder 60 42 Children 25 2–5 6–15 35–77 
   Adults 1,200 96–>99 100 100 
Demyelinating disease 60 42 Children 15 2–4 4–11 22–62 
   Adults 800 83–>99 >99–100 100 
Disorder of the peripheral nervous system 60 42 Children 80 6–15 23–56 97–100 
   Adults 2,000 >99–100 100 100 
Ataxia 60 42 All ages 80 6–15 23–56 97–100 
Allergic reaction other than anaphylaxis Children 200 18–45 64–96 >99–100 
   Adults 400 44–84 96–>99 100 

Abbreviation: RR, relative risk.

a

Age groups: all ages, ≥6 months; children, 6 months–17 years; adults, ≥18 years.

b

Because the upper limit is not always reached, the exact power is unknown. Assuming that at least half of the upper limit is reached, the statistical power to detect a signal is within the ranges shown.

Critical values of the log-likelihood ratio, above which the null hypothesis would be rejected, were obtained for Poisson and SCCS analyses for r = 1 from published tables (27). Critical values for SCCS analysis for r ≠ 1 and for DID analysis (accounting for the ratio of cases in historical control periods to cases in historical risk periods) were obtained from the same program used to generate the published tables (27).

Analyses were conducted using SAS, version 9.1.3 (SAS Institute Inc., Cary, North Carolina). Sample input data, SAS code, and output are provided in the Web Appendix, which is posted on the Journal’s Web site (http://aje.oxfordjournals.org/).

RESULTS

The number of TIV doses administered to children and adults in the VSD population increased each season, consistent with increased utilization following expanded recommendations for TIV use (52–54). In 2005/06, 317,108 first TIV doses were administered to children aged 6 months–17 years, and 1,432,218 were administered to adults aged ≥18 years. In 2006/07, the first dose counts in these age groups were 415,446 and 1,598,880, respectively, and in 2007/08, the dose counts were 462,998 and 1,742,858. Vaccine administration was temporally concentrated; each season, more than 85% of vaccines were administered in October and November.

Fifteen age-specific adverse event categories were analyzed for each of 3 seasons, for a total of 45 sequential analyses each using SCCS and DID approaches and 3 additional sequential analyses using the Poisson-based approach for Guillian-Barré syndrome. Of these, the number of adverse events reached the prespecified upper limit in 1 analysis in all ages (Guillian-Barré syndrome in 2006/07) and in 5 analyses in adults (seizure in 2005/06, other cranial nerve disorder in 2005/06, and disorder of the peripheral nervous system in all 3 seasons).

The critical value of the log-likelihood ratio was not reached in any analysis, so no associations between vaccination and adverse events warranting further investigation were detected. Summarized results as of each season's final surveillance week are shown in Table 3. The DID analyses, which adjusted for the relative risk in previous seasons, yielded relative risk ratio estimates that did not always correspond to the SCCS results in terms of magnitude and direction of effect, as would be expected. The relative risk and relative risk ratio estimates obtained using both approaches were not statistically significant for any adverse event. The Poisson-based approach for Guillian-Barré syndrome showed that the risk after vaccination each season was not significantly greater than in previous seasons.

Table 3.

Results of Sequential Testing for Influenza Vaccine Safety as of Each Season's Final Surveillance Week, Vaccine Safety Datalink Project, 2005/06–2007/08

Adverse Event Category and Age Groupa Influenza Season No. of Vaccine Doses Administered No. of Cases
 
Upper Limit of No. of Events No. of Cases
 
Self-Controlled Case Series Analysis
 
Difference-in-Difference Analysis
 
In Current Risk Period In Current Control Period In Historical Risk Periods In Historical Control Periods RR LLRb Critical Value of LLR RRR LLRb Critical Value of LLR 
Seizure              
    Children 2005/06 317,108 58 43 120 133 149 1.35 1.12 5.37 1.51 2.13 5.26 
 2006/07 415,446 51 64 200 191 192 0.80  5.51 0.80  5.51 
 2007/08 462,998 55 56 200 243 257 0.98  5.51 1.04 0.02 5.40 
    Adults 2005/06c 1,429,974 199 201 400 605 724 0.99  5.55 1.18 1.44 5.54 
 2006/07 1,598,880 153 160 500 808 933 0.96  5.56 1.10 0.38 5.62 
 2007/08 1,742,858 168 154 500 965 1,104 1.09 0.30 5.56 1.25 1.98 5.59 
Meningoencephalitis, all ages 2005/06 1,749,326 23 87 120 71 319 0.76  5.40 1.19 0.26 5.11 
 2006/07 2,014,326 21 53 150 94 406 1.13 0.11 5.40 1.71 1.99 5.06 
 2007/08 2,205,856 22 71 150 115 459 0.89  5.40 1.24 0.37 5.17 
Bell's palsy              
    Children 2005/06 317,108 17 40 28 39 0.67  5.01 0.66  5.14 
 2006/07 415,446 19 15 50 36 56 1.81 1.49 5.24 1.97 1.94 4.92 
 2007/08 462,998 16 18 50 55 71 1.27 0.24 5.24 1.15 0.08 4.97 
    Adults 2005/06 1,432,218 271 365 800 1,025 1,473 1.06 0.27 5.72 1.07 0.33 5.70 
 2006/07 1,598,880 300 399 800 1,296 1,838 1.07 0.44 5.72 1.07 0.35 5.72 
 2007/08 1,742,858 311 447 1,000 1,596 2,237 0.99  5.77 0.98  5.81 
Other cranial nerve disorder              
    Children 2005/06 317,108 20 11 23 1.07 0.00 4.53 1.57 0.17 4.51 
 2006/07 415,446 25 14 27 0.95  4.72 1.29 0.11 4.68 
 2007/08 462,998 11 25 20 36 0.39  4.72 0.49  4.94 
    Adults 2005/06c 1,416,883 337 466 800 1,172 1,582 1.03 0.10 5.72 0.97  5.75 
 2006/07 1,598,880 391 532 1,000 1,512 2,051 1.05 0.27 5.77 1.00  5.81 
 2007/08 1,742,858 374 544 1,200 1,903 2,583 0.98  5.81 0.93  5.84 
Demyelinating disease              
    Children 2005/06 317,108 15 15 11 2.14 0.36 4.49 1.10 0.01 4.41 
 2006/07 415,446 20 18 13 0.48  4.53 0.24  4.81 
 2007/08 462,998 15 19 16 0.86  4.49 0.51  4.28 
    Adults 2005/06 1,432,218 171 298 600 753 1,189 0.82  5.65 0.91  5.68 
 2006/07 1,598,880 201 325 600 924 1,487 0.88  5.65 1.00  5.75 
 2007/08 1,742,858 194 347 800 1,125 1,812 0.80  5.72 0.90  5.76 
Disorder of the peripheral nervous system              
    Children 2005/06 317,108 22 50 46 56 0.58  5.24 0.50  5.01 
 2006/07 415,446 10 20 60 55 78 0.71  5.24 0.71  5.23 
 2007/08 462,998 19 27 80 65 98 1.01  5.32 1.06 0.02 5.26 
    Adults 2005/06c 569,039 938 1,524 2,000 7,893 11,642 0.88  5.95 0.91  5.89 
 2006/07c 601,861 1,007 1,482 2,000 10,212 14,918 0.97  5.95 0.99  5.90 
 2007/08c 581,324 887 1,355 2,000 12,792 18,542 0.94  5.95 0.95  5.93 
Ataxia, all ages 2005/06 1,749,326 13 27 80 86 105 0.69  5.32 0.59  5.20 
2006/07 2,014,326 27 38 80 99 132 1.02 0.00 5.32 0.95  5.08 
2007/08 2,205,856 35 36 80 126 170 1.39 0.95 5.32 1.31 0.65 5.12 
Anaphylaxisd, all ages 2005/06 1,749,326 12 3.00 0.52 4.27  N/A  
 2006/07 2,014,326 12 4.00 0.96 4.27  N/A  
 2007/08 2,205,856 12 1.00 0.00 4.27  N/A  
Allergic reaction other than anaphylaxise              
    Children 2005/06 319,203 49 52 150 162 122 0.94  5.45 0.71  5.35 
 2006/07 418,081 96 70 200 211 174 1.37 2.04 5.51 1.13 0.31 5.41 
 2007/08 459,263 104 70 200 307 244 1.49 3.34 5.51 1.18 0.58 5.37 
    Adults 2005/06 1,466,842 137 98 300 459 376 1.40 3.25 5.55 1.15 0.53 5.44 
 2006/07 1,635,562 156 125 300 596 474 1.25 1.71 5.55 0.99  5.51 
 2007/08 1,746,619 196 164 400 752 599 1.20 1.42 5.55 0.95  5.61 
      Expected No. of Cases Based on Historical Risk Periods Poisson-based Analysis
 
   
      RR LLRb Critical Value of LLR    
Guillian-Barré syndromee, all ages 2005/06 1,788,108 12  15 14.4 0.83  3.57    
 2006/07c 1,960,430 17  15 15.1 1.13 0.12 3.57    
 2007/08 2,214,335 23  20 16.7 1.37 1.05 3.63    
Adverse Event Category and Age Groupa Influenza Season No. of Vaccine Doses Administered No. of Cases
 
Upper Limit of No. of Events No. of Cases
 
Self-Controlled Case Series Analysis
 
Difference-in-Difference Analysis
 
In Current Risk Period In Current Control Period In Historical Risk Periods In Historical Control Periods RR LLRb Critical Value of LLR RRR LLRb Critical Value of LLR 
Seizure              
    Children 2005/06 317,108 58 43 120 133 149 1.35 1.12 5.37 1.51 2.13 5.26 
 2006/07 415,446 51 64 200 191 192 0.80  5.51 0.80  5.51 
 2007/08 462,998 55 56 200 243 257 0.98  5.51 1.04 0.02 5.40 
    Adults 2005/06c 1,429,974 199 201 400 605 724 0.99  5.55 1.18 1.44 5.54 
 2006/07 1,598,880 153 160 500 808 933 0.96  5.56 1.10 0.38 5.62 
 2007/08 1,742,858 168 154 500 965 1,104 1.09 0.30 5.56 1.25 1.98 5.59 
Meningoencephalitis, all ages 2005/06 1,749,326 23 87 120 71 319 0.76  5.40 1.19 0.26 5.11 
 2006/07 2,014,326 21 53 150 94 406 1.13 0.11 5.40 1.71 1.99 5.06 
 2007/08 2,205,856 22 71 150 115 459 0.89  5.40 1.24 0.37 5.17 
Bell's palsy              
    Children 2005/06 317,108 17 40 28 39 0.67  5.01 0.66  5.14 
 2006/07 415,446 19 15 50 36 56 1.81 1.49 5.24 1.97 1.94 4.92 
 2007/08 462,998 16 18 50 55 71 1.27 0.24 5.24 1.15 0.08 4.97 
    Adults 2005/06 1,432,218 271 365 800 1,025 1,473 1.06 0.27 5.72 1.07 0.33 5.70 
 2006/07 1,598,880 300 399 800 1,296 1,838 1.07 0.44 5.72 1.07 0.35 5.72 
 2007/08 1,742,858 311 447 1,000 1,596 2,237 0.99  5.77 0.98  5.81 
Other cranial nerve disorder              
    Children 2005/06 317,108 20 11 23 1.07 0.00 4.53 1.57 0.17 4.51 
 2006/07 415,446 25 14 27 0.95  4.72 1.29 0.11 4.68 
 2007/08 462,998 11 25 20 36 0.39  4.72 0.49  4.94 
    Adults 2005/06c 1,416,883 337 466 800 1,172 1,582 1.03 0.10 5.72 0.97  5.75 
 2006/07 1,598,880 391 532 1,000 1,512 2,051 1.05 0.27 5.77 1.00  5.81 
 2007/08 1,742,858 374 544 1,200 1,903 2,583 0.98  5.81 0.93  5.84 
Demyelinating disease              
    Children 2005/06 317,108 15 15 11 2.14 0.36 4.49 1.10 0.01 4.41 
 2006/07 415,446 20 18 13 0.48  4.53 0.24  4.81 
 2007/08 462,998 15 19 16 0.86  4.49 0.51  4.28 
    Adults 2005/06 1,432,218 171 298 600 753 1,189 0.82  5.65 0.91  5.68 
 2006/07 1,598,880 201 325 600 924 1,487 0.88  5.65 1.00  5.75 
 2007/08 1,742,858 194 347 800 1,125 1,812 0.80  5.72 0.90  5.76 
Disorder of the peripheral nervous system              
    Children 2005/06 317,108 22 50 46 56 0.58  5.24 0.50  5.01 
 2006/07 415,446 10 20 60 55 78 0.71  5.24 0.71  5.23 
 2007/08 462,998 19 27 80 65 98 1.01  5.32 1.06 0.02 5.26 
    Adults 2005/06c 569,039 938 1,524 2,000 7,893 11,642 0.88  5.95 0.91  5.89 
 2006/07c 601,861 1,007 1,482 2,000 10,212 14,918 0.97  5.95 0.99  5.90 
 2007/08c 581,324 887 1,355 2,000 12,792 18,542 0.94  5.95 0.95  5.93 
Ataxia, all ages 2005/06 1,749,326 13 27 80 86 105 0.69  5.32 0.59  5.20 
2006/07 2,014,326 27 38 80 99 132 1.02 0.00 5.32 0.95  5.08 
2007/08 2,205,856 35 36 80 126 170 1.39 0.95 5.32 1.31 0.65 5.12 
Anaphylaxisd, all ages 2005/06 1,749,326 12 3.00 0.52 4.27  N/A  
 2006/07 2,014,326 12 4.00 0.96 4.27  N/A  
 2007/08 2,205,856 12 1.00 0.00 4.27  N/A  
Allergic reaction other than anaphylaxise              
    Children 2005/06 319,203 49 52 150 162 122 0.94  5.45 0.71  5.35 
 2006/07 418,081 96 70 200 211 174 1.37 2.04 5.51 1.13 0.31 5.41 
 2007/08 459,263 104 70 200 307 244 1.49 3.34 5.51 1.18 0.58 5.37 
    Adults 2005/06 1,466,842 137 98 300 459 376 1.40 3.25 5.55 1.15 0.53 5.44 
 2006/07 1,635,562 156 125 300 596 474 1.25 1.71 5.55 0.99  5.51 
 2007/08 1,746,619 196 164 400 752 599 1.20 1.42 5.55 0.95  5.61 
      Expected No. of Cases Based on Historical Risk Periods Poisson-based Analysis
 
   
      RR LLRb Critical Value of LLR    
Guillian-Barré syndromee, all ages 2005/06 1,788,108 12  15 14.4 0.83  3.57    
 2006/07c 1,960,430 17  15 15.1 1.13 0.12 3.57    
 2007/08 2,214,335 23  20 16.7 1.37 1.05 3.63    

Abbreviations: LLR, log-likelihood ratio; N/A, not applicable; RR, relative risk; RRR, relative risk ratio.

a

Age groups: all ages, ≥6 months; children, 6 months–17 years; adults, ≥18 years.

b

Since the statistical tests were 1-sided, the LLR was not calculated when the RR was less than 1.0.

c

Current season results for these adverse events are presented as of the week the upper limit was reached, not as of the end of surveillance.

d

Difference-in-difference analyses were not conducted for anaphylaxis, since there were fewer than 25 cases in the risk and control periods in previous seasons.

e

Analyses for allergic reactions and Guillian-Barré syndrome were conducted after the analyses for the other categories. Because of the dynamic nature of the data source, the dose counts for these categories fluctuated in comparison with those for the other categories.

Weekly sequential analysis results for 1 representative outcome, seizure in adults, are highlighted for both the SCCS (Figure 2, part A) and DID (Figure 2, part B) approaches. The relative risk or relative risk ratio had a rapid increase early in the season with few doses administered, but as the season progressed and more doses were given, the relative risk or relative risk ratio stabilized and quickly trended towards 1. During surveillance, the log-likelihood ratio peaked at only 12% of the critical value for SCCS and at only 35% of the critical value for DID. In other words, the SCCS approach showed that vaccinated cases had no statistically significant elevated risk of seizure in the risk period versus the control period in 2005/06, 2006/07, or 2007/08. The DID approach showed that this relative risk was not significantly greater in 2005/06 than in 2000/01–2004/05, not significantly greater in 2006/07 than in 2000/01–2005/06, and not significantly greater in 2007/08 than in 2000/01–2006/07.

Figure 2.

Weekly sequential analysis results for seizures among trivalent inactivated influenza vaccine recipients aged ≥18 years in the 2007/08 influenza season, Vaccine Safety Datalink Project. A) Self-controlled case series approach; B) difference-in-difference approach.

Figure 2.

Weekly sequential analysis results for seizures among trivalent inactivated influenza vaccine recipients aged ≥18 years in the 2007/08 influenza season, Vaccine Safety Datalink Project. A) Self-controlled case series approach; B) difference-in-difference approach.

DISCUSSION

Near real-time surveillance for selected adverse events following seasonal or pandemic influenza vaccine is possible in systems such as the VSD Project, where data are updated at least weekly. Implementing SCCS and DID approaches in sequential testing allows for a timely alternative to more traditional systems for evaluating influenza vaccine safety, while adjusting for important confounders. Previous randomized controlled trials (49, 55, 56), passive reporting systems (30, 57), and postlicensure safety studies (20, 21) have supported the overall safety of TIV. This multisite exercise was consistent with previous studies in showing no evidence of elevated risk following TIV for any of 10 predefined adverse event categories. Poisson-based analyses indicated that in each of 3 recent seasons, the risk of Guillian-Barré syndrome following vaccination was not significantly different from that in previous seasons. SCCS analyses indicated that the risk of adverse events during the risk period was not significantly different from that during the control period. DID analyses indicated that the relative risk in each season was not significantly different from the relative risk across cumulative previous seasons. SCCS and DID analyses compared risk periods with control periods within the same vaccinees each season, thus controlling for potential biases. Using binomial-based maximized sequential probability ratio testing, power was adequate for most adverse event categories to detect relative risks in the range of 2.0 and very good to detect high relative risks in the range of 5.0; more modest risk elevations may not have been detectable using these methods. However, if a modest elevation in risk were undetected for a very rare adverse event, the population attributable risk for vaccination would be very small. The lack of any safety signals can increase public confidence in influenza vaccine safety.

Using more than 1 analytic approach for surveillance may enhance the rigor of a safety assessment. A signal based on either method would warrant additional investigation for assessment of its validity. If both approaches were to produce a signal for the same adverse event, evidence for a safety concern requiring further evaluation would be strengthened. The relative risks and relative risk ratios produced by SCCS and DID were different, which was expected because they are calculated differently, but neither approach generated a safety signal. The application of repeated confidence intervals to maximized sequential probability ratio testing is under development (58).

At least 4 potential limitations affected the analyses. First, timely detection of a safety problem depends partially on the lag between adverse event occurrence and electronic reporting. This lag is typically 2 weeks for outpatient and emergency department data, but it ranges from 1 day to several weeks for hospitalizations, depending on the data source. The delay in ascertaining hospitalizations could be reduced with additional resources by using health plans that ascertain hospitalizations via electronic records (not claims) or by monitoring preadjudicated claims. Near real-time surveillance is most useful for adverse event categories requiring observation of only a brief risk period of less than 2–3 weeks, so that a signal could occur within a month after vaccines are given. In such a situation, surveillance may identify a safety problem before most vaccines for a season are administered, and warnings may be issued to prevent additional exposure. In contrast, using sequential analyses for adverse events with a lengthy risk window or a postvaccination control period sacrifices both power and simplicity with little potential gain in timeliness compared with a final analysis each spring.

Second, the VSD Project uses electronic data collected largely for administrative purposes. Some vaccinations are missed (16), as are telephone encounters and adverse events that are not medically attended. The sensitivity, specificity, and positive and negative predictive values of ICD-9 codes vary by adverse event and setting (59–61) and have not been thoroughly characterized. Third, binomial-based maximized sequential probability ratio testing had limited power for timely detection of a modest elevation in risk for very rare adverse events (e.g., anaphylaxis). To minimize false-negative findings, power could be improved as the VSD population expands and by investing in larger surveillance systems—for example, the US Food and Drug Administration's Sentinel Initiative, which aims to use electronic data from 100 million people by 2012 (62). Using a historical comparison group and a Poisson probability model is another alternative for rare adverse events and may have better statistical power (27). Finally, although the SCCS and DID approaches adjusted for some confounders, they did not adjust for seasonality of vaccination and adverse event risk. A case-centered approach (63) accounting for the timing of vaccination in the overall population could be used to adjust for seasonal effects.

To our knowledge, this is the first application of both SCCS and DID approaches to the sequential monitoring of adverse events. In addition to influenza vaccines, these approaches may be applicable to monitoring the safety of other vaccines for which finding an appropriate comparison group is challenging and the healthy vaccinee effect may be substantial. These approaches also may be useful in drug safety surveillance (e.g., for influenza antiviral medications) and for other data sources (e.g., Medicare or other insurers) (62, 64, 65). New adverse events of potential concern (e.g., those detected through the Vaccine Adverse Event Reporting System) can be quickly added to surveillance, and subgroup analyses (e.g., for manufacturer-specific vaccine products) can be readily undertaken, although power may be limited. If a safety signal is generated in the future, additional investigations will distinguish true positives from false positives, including evaluation of the temporal association between vaccination and adverse events, high-risk subgroup analyses, and medical chart review (12, 66).

Near real-time safety surveillance for seasonal and pandemic influenza vaccines is possible, and prospective analyses for seasonal influenza vaccines are being conducted, starting with the 2008/09 season. With a well-defined population and the ability to access medical records for review, the VSD Project will monitor the safety of influenza A (H1N1) 2009 monovalent vaccines (67), using binomial- (SCCS and DID) and Poisson-based approaches.

Abbreviations

    Abbreviations
  • DID

    difference-in-difference

  • ICD-9

    International Classification of Diseases, Ninth Revision

  • SCCS

    self-controlled case series

  • TIV

    trivalent inactivated influenza vaccine

  • VSD

    Vaccine Safety Datalink

Author affiliations: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts (Sharon K. Greene, Martin Kulldorff, Rong Li, Ruihua Yin, Tracy A. Lieu, Grace M. Lee); Kaiser Permanente Vaccine Study Center, Oakland, California (Edwin M. Lewis, Bruce H. Fireman, Roger Baxter); Immunization Safety Office, Centers for Disease Control and Prevention, Atlanta, Georgia (Eric S. Weintraub, Karen R. Broder); HealthPartners Research Foundation, Minneapolis, Minnesota (James D. Nordin); Kaiser Permanente of Colorado, Denver, Colorado (Jason M. Glanz); Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, California (Steven J. Jacobsen); and Division of Infectious Diseases, Department of Medicine and Department of Laboratory Medicine, Children's Hospital Boston, Boston, Massachusetts (Grace M. Lee).

This work was supported by a subcontract with America's Health Insurance Plans under contract 200-2002-00732 from the Centers for Disease Control and Prevention and by the Agency for Healthcare Research and Quality, US Department of Health and Human Services (grant 5 K08 HS013908-04 to G. M. L.).

For their valuable contributions to conception, data acquisition, and technical support of this work, the authors acknowledge Irene M. Shui and Drs. Richard Platt, Jeffrey S. Brown, Edward Belongia, Allison L. Naleway, Lisa A. Jackson, James G. Donahue, Robert L. Davis, and Margarette Kolczak (deceased).

Preliminary results of this work were presented as a platform session (abstract G-769) at the 48th Annual Interscience Conference on Antimicrobial Agents and Chemotherapy and the Infectious Diseases Society of America 46th Annual Meeting in Washington, DC, on October 25, 2008.

The findings and conclusions in this report are those of the authors and do not necessarily represent the views or policies of the Centers for Disease Control and Prevention.

Edwin M. Lewis reports having received grant support from Sanofi Pasteur, Chiron, MedImmune, Merck, and Wyeth. Dr. James D. Nordin reports having received grant support from Sanofi Pasteur. Dr. Roger Baxter reports having received grant support from Sanofi Pasteur, Novartis, GSK, and MedImmune. No other authors have any potential or real conflicts of interest to declare.

References

1.
Update: novel influenza A (H1N1) virus infections—worldwide, May 6, 2009
MMWR Morb Mortal Wkly Rep
 , 
2009
, vol. 
58
 
17
(pg. 
453
-
458
)
2.
Collin
N
de Radiguès
X
Vaccine production capacity for seasonal and pandemic (H1N1) 2009 influenza. World Health Organization H1N1 Vaccine Task Force
Vaccine
 , 
2009
, vol. 
27
 
38
(pg. 
5184
-
5186
)
3.
Center for Biologics Evaluation and Research, US Food and Drug Administration
Influenza A (H1N1) 2009 Monovalent
 , 
2009
Rockville, MD
Center for Biologics Evaluation and Research
 
4.
Use of influenza A (H1N1) 2009 monovalent vaccine: recommendations of the Advisory Committee on Immunization Practices (ACIP), 2009
MMWR Recomm Rep
 , 
2009
, vol. 
58
 
RR-10
(pg. 
1
-
8
)
5.
Fiore
AE
Shay
DK
Broder
K
, et al.  . 
Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP), 2009
MMWR Recomm Rep
 , 
2009
, vol. 
58
 
RR-8
(pg. 
1
-
52
)
6.
Schwartz
B
Hinman
A
Abramson
J
, et al.  . 
Universal influenza vaccination in the United States: are we ready? Report of a meeting
J Infect Dis
 , 
2006
, vol. 
194
 
suppl 2
(pg. 
S147
-
S154
)
7.
Iskander
J
Broder
K
Monitoring the safety of annual and pandemic influenza vaccines: lessons from the US experience
Expert Rev Vaccines
 , 
2008
, vol. 
7
 
1
(pg. 
75
-
82
)
8.
US Department of Health and Human Services
Vaccine monitoring and data collection. (Section S6-III.B3)
HHS Pandemic Influenza Plan. Supplement 6. Vaccine Distribution and Use
 , 
2005
Washington, DC
US Department of Health and Human Services
 
9.
Zhou
W
Pool
V
Iskander
JK
, et al.  . 
Surveillance for safety after immunization: Vaccine Adverse Event Reporting System (VAERS)—United States, 1991–2001
MMWR Surveill Summ
 , 
2003
, vol. 
52
 
1
(pg. 
1
-
24
)
10.
Lieu
TA
Kulldorff
M
Davis
RL
, et al.  . 
Real-time vaccine safety surveillance for the early detection of adverse events
Med Care
 , 
2007
, vol. 
45
 
10 suppl 2
(pg. 
S89
-
S95
)
11.
Yih
WK
Nordin
JD
Kulldorff
M
, et al.  . 
An assessment of the safety of adolescent and adult tetanus-diphtheria-acellular pertussis (Tdap) vaccine, using active surveillance for adverse events in the Vaccine Safety Datalink
Vaccine
 , 
2009
, vol. 
27
 
32
(pg. 
4257
-
4262
)
12.
Update: recommendations from the Advisory Committee on Immunization Practices (ACIP) regarding administration of combination MMRV vaccine
MMWR Morb Mortal Wkly Rep
 , 
2008
, vol. 
57
 
10
(pg. 
258
-
260
)
13.
Belongia
EA
Irving
S
Shui
IM
, et al.  . 
Real-time surveillance to assess risk of intussusception and other adverse events after pentavalent rotavirus vaccine (RotaTeq)
Pediatr Infect Dis J
 , 
2009
 
Nov 10 [Epub ahead of print]
14.
Hak
E
Verheij
TJ
Grobbee
DE
, et al.  . 
Confounding by indication in non-experimental evaluation of vaccine effectiveness: the example of prevention of influenza complications
J Epidemiol Community Health
 , 
2002
, vol. 
56
 
12
(pg. 
951
-
955
)
15.
Jackson
ML
Nelson
JC
Weiss
NS
, et al.  . 
Influenza vaccination and risk of community-acquired pneumonia in immunocompetent elderly people: a population-based, nested case-control study
Lancet
 , 
2008
, vol. 
372
 
9636
(pg. 
398
-
405
)
16.
Greene
SK
Shi
P
Dutta-Linn
MM
, et al.  . 
Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites
Am J Prev Med
 , 
2009
, vol. 
37
 
6
(pg. 
552
-
555
)
17.
Farrington
CP
Relative incidence estimation from case series for vaccine safety evaluation
Biometrics
 , 
1995
, vol. 
51
 
1
(pg. 
228
-
235
)
18.
Whitaker
HJ
Farrington
CP
Spiessens
B
, et al.  . 
Tutorial in biostatistics: the self-controlled case series method
Stat Med
 , 
2006
, vol. 
25
 
10
(pg. 
1768
-
1797
)
19.
Meyer
BD
Natural and quasi-experiments in economics
J Bus Econ Stat
 , 
1995
, vol. 
13
 
2
(pg. 
151
-
161
)
20.
France
EK
Glanz
JM
Xu
S
, et al.  . 
Safety of the trivalent inactivated influenza vaccine among children: a population-based study
Arch Pediatr Adolesc Med
 , 
2004
, vol. 
158
 
11
(pg. 
1031
-
1036
)
21.
Hambidge
SJ
Glanz
JM
France
EK
, et al.  . 
Safety of trivalent inactivated influenza vaccine in children 6 to 23 months old
JAMA
 , 
2006
, vol. 
296
 
16
(pg. 
1990
-
1997
)
22.
Stowe
J
Andrews
N
Wise
L
, et al.  . 
Investigation of the temporal association of Guillain-Barré syndrome with influenza vaccine and influenzalike illness using the United Kingdom General Practice Research Database
Am J Epidemiol
 , 
2009
, vol. 
169
 
3
(pg. 
382
-
388
)
23.
Virtanen
M
Peltola
H
Paunio
M
, et al.  . 
Day-to-day reactogenicity and the healthy vaccinee effect of measles-mumps-rubella vaccination [electronic article]
Pediatrics
 , 
2000
, vol. 
106
 
5
pg. 
E62
 
24.
Maclure
M
The case-crossover design: a method for studying transient effects on the risk of acute events
Am J Epidemiol
 , 
1991
, vol. 
133
 
2
(pg. 
144
-
153
)
25.
Farrington
CP
Control without separate controls: evaluation of vaccine safety using case-only methods
Vaccine
 , 
2004
, vol. 
22
 
15-16
(pg. 
2064
-
2070
)
26.
Davis
R
Lewis
N
Kulldorff
M
, et al.  . 
Rapid cycle analysis of vaccine safety data [abstract]
 
Presented at the Vaccine Safety Datalink Annual Meeting, Redondo Beach, California, April 20, 2005
27.
Kulldorff
M
Davis
RL
Kolczak
M
, et al.  . 
A Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance. (Working paper)
 , 
2009
Boston, MA
Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute
 
28.
Chen
RT
DeStefano
F
Davis
RL
, et al.  . 
The Vaccine Safety Datalink: immunization research in health maintenance organizations in the USA
Bull World Health Organ
 , 
2000
, vol. 
78
 
2
(pg. 
186
-
194
)
29.
DeStefano
F
The Vaccine Safety Datalink Project
Pharmacoepidemiol Drug Saf
 , 
2001
, vol. 
10
 
5
(pg. 
403
-
406
)
30.
McMahon
AW
Iskander
J
Haber
P
, et al.  . 
Adverse events after inactivated influenza vaccination among children less than 2 years of age: analysis of reports from the Vaccine Adverse Event Reporting System, 1990–2003
Pediatrics
 , 
2005
, vol. 
115
 
2
(pg. 
453
-
460
)
31.
Gross
WL
Ravens
KG
Hansen
HW
Meningoencephalitis syndrome following influenza vaccination
J Neurol
 , 
1978
, vol. 
217
 
3
(pg. 
219
-
222
)
32.
Haber
P
Influenza vaccine and neurological adverse events: VAERS 7/1990–1/2003 [abstract]. Presented at the meeting
“Immunization Safety Review: Influenza Vaccine and Possible Neurological Complications,”
  
Washington, DC, March 13, 2003. Washington, DC: Institute of Medicine; 2003. (http://www.iom.edu/CMS/3793/4705/4755/7054.aspx). (Accessed October 10, 2008)
33.
Nakamura
N
Nokura
K
Zettsu
T
, et al.  . 
Neurologic complications associated with influenza vaccination: two adult cases
Intern Med
 , 
2003
, vol. 
42
 
2
(pg. 
191
-
194
)
34.
Zhou
W
Pool
V
DeStefano
F
, et al.  . 
A potential signal of Bell's palsy after parenteral inactivated influenza vaccines: reports to the Vaccine Adverse Event Reporting System (VAERS)—United States, 1991–2001
Pharmacoepidemiol Drug Saf
 , 
2004
, vol. 
13
 
8
(pg. 
505
-
510
)
35.
Chou
CH
Liou
WP
Hu
KI
, et al.  . 
Bell's palsy associated with influenza vaccination: two case reports
Vaccine
 , 
2007
, vol. 
25
 
15
(pg. 
2839
-
2841
)
36.
Brostoff
JM
Beitverda
Y
Birns
J
Post-influenza vaccine chronic inflammatory demyelinating polyneuropathy
Age Ageing
 , 
2008
, vol. 
37
 
2
(pg. 
229
-
230
)
37.
Hull
TP
Bates
JH
Optic neuritis after influenza vaccination
Am J Ophthalmol
 , 
1997
, vol. 
124
 
5
(pg. 
703
-
704
)
38.
Hull
JH
Mead
SH
Foster
OJ
, et al.  . 
Severe vasculitic neuropathy following influenza vaccination
J Neurol Neurosurg Psychiatry
 , 
2004
, vol. 
75
 
10
(pg. 
1507
-
1508
)
39.
Saito
H
Yanagisawa
T
Acute cerebellar ataxia after influenza vaccination with recurrence and marked cerebellar atrophy
Tohoku J Exp Med
 , 
1989
, vol. 
158
 
1
(pg. 
95
-
103
)
40.
Bohlke
K
Davis
RL
Marcy
SM
, et al.  . 
Risk of anaphylaxis after vaccination of children and adolescents
Pediatrics
 , 
2003
, vol. 
112
 
4
(pg. 
815
-
820
)
41.
Coop
CA
Balanon
SK
White
KM
, et al.  . 
Anaphylaxis from the influenza virus vaccine
Int Arch Allergy Immunol
 , 
2008
, vol. 
146
 
1
(pg. 
85
-
88
)
42.
Horowitz
L
Urticaria subsequent to administration of influenza vaccine
South Med J
 , 
1979
, vol. 
72
 
10
(pg. 
1334
-
1335
)
43.
Juurlink
DN
Stukel
TA
Kwong
J
, et al.  . 
Guillain-Barré syndrome after influenza vaccination in adults: a population-based study
Arch Intern Med
 , 
2006
, vol. 
166
 
20
(pg. 
2217
-
2221
)
44.
Schonberger
LB
Bregman
DJ
Sullivan-Bolyai
JZ
, et al.  . 
Guillain-Barré syndrome following vaccination in the National Influenza Immunization Program, United States, 1976–1977
Am J Epidemiol
 , 
1979
, vol. 
110
 
2
(pg. 
105
-
123
)
45.
Haber
P
DeStefano
F
Angulo
FJ
, et al.  . 
Guillain-Barré syndrome following influenza vaccination
JAMA
 , 
2004
, vol. 
292
 
20
(pg. 
2478
-
2481
)
46.
Piyasirisilp
S
Hemachudha
T
Neurological adverse events associated with vaccination
Curr Opin Neurol
 , 
2002
, vol. 
15
 
3
(pg. 
333
-
338
)
47.
Kalaboka
S
Annesi-Maesano
I
The complex link between immunization against childhood diseases and allergy
Expert Rev Vaccines
 , 
2007
, vol. 
6
 
4
(pg. 
635
-
643
)
48.
Moylett
EH
Hanson
IC
Mechanistic actions of the risks and adverse events associated with vaccine administration
J Allergy Clin Immunol
 , 
2004
, vol. 
114
 
5
(pg. 
1010
-
1020
)
49.
Nichol
KL
Margolis
KL
Lind
A
, et al.  . 
Side effects associated with influenza vaccination in healthy working adults. A randomized, placebo-controlled trial
Arch Intern Med
 , 
1996
, vol. 
156
 
14
(pg. 
1546
-
1550
)
50.
Moore
KM
Duddy
A
Lee
GM
, et al.  . 
Outpatient urticaria diagnosis codes have limited predictive value for same-day influenza vaccine adverse event detection
J Clin Epidemiol.
 , 
2009
 
Nov 2 [Epub ahead of print]
51.
Li
L
Kulldorff
M
A conditional maximized sequential probability ratio test for pharmacovigilance
Stat Med
  
In press
52.
Harper
SA
Fukuda
K
Uyeki
TM
, et al.  . 
Prevention and control of influenza. Recommendations of the Advisory Committee on Immunization Practices (ACIP)
MMWR Recomm Rep
 , 
2005
, vol. 
54
 
RR-8
(pg. 
1
-
40
)
53.
Smith
NM
Bresee
JS
Shay
DK
, et al.  . 
Prevention and control of influenza. Recommendations of the Advisory Committee on Immunization Practices (ACIP)
MMWR Recomm Rep
 , 
2006
, vol. 
55
 
RR-10
(pg. 
1
-
42
)
54.
Fiore
AE
Shay
DK
Haber
P
, et al.  . 
Prevention and control of influenza. Recommendations of the Advisory Committee on Immunization Practices (ACIP)
MMWR Recomm Rep
 , 
2007
, vol. 
56
 
RR-6
(pg. 
1
-
54
)
55.
Barry
DW
Mayner
RE
Hochstein
HD
, et al.  . 
Comparative trial of influenza vaccines. II. Adverse reactions in children and adults
Am J Epidemiol
 , 
1976
, vol. 
104
 
1
(pg. 
47
-
59
)
56.
Govaert
TM
Dinant
GJ
Aretz
K
, et al.  . 
Adverse reactions to influenza vaccine in elderly people: randomised double blind placebo controlled trial
BMJ
 , 
1993
, vol. 
307
 
6910
(pg. 
988
-
990
)
57.
Vellozzi
C
Burwen
DR
Dobardzic
A
, et al.  . 
Safety of trivalent inactivated influenza vaccines in adults: background for pandemic influenza vaccine safety monitoring
Vaccine
 , 
2009
, vol. 
27
 
15
(pg. 
2114
-
2120
)
58.
Mullooly
J
Crane
B
MaxSPRT-based repeated interval estimates of event rate ratios for independent concurrent comparison populations [abstract]
 
Presented at the Vaccine Safety Datalink Annual Meeting, Atlanta, Georgia, March 18–19, 2009
59.
Mullooly
J
Drew
L
DeStefano
F
, et al.  . 
Quality assessments of HMO diagnosis databases used to monitor childhood vaccine safety
Methods Inf Med
 , 
2004
, vol. 
43
 
2
(pg. 
163
-
170
)
60.
Mullooly
JP
Donahue
JG
DeStefano
F
, et al.  . 
Predictive value of ICD-9-CM codes used in vaccine safety research
Methods Inf Med
 , 
2008
, vol. 
47
 
4
(pg. 
328
-
335
)
61.
Shui
IM
Shi
P
Dutta-Linn
MM
, et al.  . 
Predictive value of seizure ICD-9 codes for vaccine safety research
Vaccine
 , 
2009
, vol. 
27
 
39
(pg. 
5307
-
5312
)
62.
Platt
R
Wilson
M
Chan
KA
, et al.  . 
The new Sentinel Network—improving the evidence of medical-product safety
N Engl J Med
 , 
2009
, vol. 
361
 
7
(pg. 
645
-
647
)
63.
Fireman
B
Lee
J
Lewis
N
, et al.  . 
Influenza vaccination and mortality: differentiating vaccine effects from bias
Am J Epidemiol
 , 
2009
, vol. 
170
 
5
(pg. 
650
-
656
)
64.
Burwen
DR
La Voie
L
Braun
MM
, et al.  . 
Evaluating adverse events after vaccination in the Medicare population
Pharmacoepidemiol Drug Saf
 , 
2007
, vol. 
16
 
7
(pg. 
753
-
761
)
65.
Brown
JS
Moore
KM
Braun
MM
, et al.  . 
Active influenza vaccine safety surveillance: potential within a healthcare claims environment
Med Care
 , 
2009
 
Sep 25. [Epub ahead of print]
66.
Yih
K
Kulldorff
M
Three years of rapid cycle analysis—what we've learned and what remains to be done [abstract]
 
Presented at the Vaccine Safety Datalink Annual Meeting, Atlanta, Georgia, March 18–19, 2009
67.
DeStefano
F
Tokars
J
H1N1 vaccine safety monitoring: beyond background rates
Lancet
 , 
2009
 
Oct 31 [Epub ahead of print]