ARTICLE
Changes in Clinic Vaccination Coverage After Institution of Measurement and Feedback
in 4 States and 2 Cities
Charles W. LeBaron, MD; J. Todd Mercer, MPH; Mehran S. Massoudi, PhD, MPH; Eugene Dini, MPA; John Stevenson, MA; Wayne M. Fischer, MPH; Herbert Loy; Lori Stonehocker Quick, RN, MSN;
John C. Warming; Pat Tormey, RN, MPH; M. DesVignes-Kendrick, MD, MPH
Background: Since 1995, states and jurisdictions re- ceiving federal immunization funds have been required to perform annual measurements of vaccination cover- age in their public clinics, based on data from Georgia where clinic coverage increased after the institution of a measurement and feedback intervention.

Objective: To determine if clinic vaccination coverage improved in localities that used the Georgia interven- tion model.

Design: Retrospective examination of clinic vaccina- tion coverage data.

Participants: Children aged 19 to 35 months enrolled in clinics in localities that had applied the intervention for 4 years or longer.

Intervention: The Georgia intervention model: assess- ment of clinic vaccination coverage, feedback of the in- formation to the clinic, incentives to clinics, and promo- tion of exchange of information among clinics (AFIX).

Main Outcome Measure: Change in median clinic cov- erage rates, based on the primary (4-3-1) vaccine series,
Editor’s Note: I guess we can now AFIX vaccine coverage problems.
Catherine D. DeAngelis, MD
VACCINATION RATES among 2-year-old children en- rolled in Georgia public clinics rose from 53% to 89% during a period of 7 years, following implementation of an intervention involving annual mea- surement of coverage in each clinic, feed- back of the data to the clinic, nonmon- etary awards to high-performing clinics, and facilitation of exchange of in-
with comparison to results of the National Immuniza- tion Survey.

Results: Four states and 2 cities that had applied the AFIX intervention for 4 years or longer were identified. The number of clinic records reviewed annually was 4639 to 18 000 in 73 to 116 clinics for states, and 714 to 5276 in 8 to 25 clinics for cities. Median clinic coverage rose in all localities: Missouri, 44% (1992) to 93% (1997); Loui- siana, 61% (1992) to 83% (1997); Colorado, 55% (1993) to 75% (1997); Iowa, 71% (1994) to 89% (1997); Bos- ton, Mass, 41% (1994) to 79% (1997); and Houston, Tex, 28% (1994) to 84% (1997). The increase in clinic coverage exceeded that of the general population in 5 lo- calities and was identical in the sixth. The average an- nual coverage rise attributable to the intervention was +5 percentage points per year (Georgia, +6 per year). The average crude direct program cost was $49 533 per lo- cality per year.

Conclusion: The Georgia intervention model (AFIX) can be reproduced elsewhere and is associated with improve- ments in clinic vaccination coverage.

Arch Pediatr Adolesc Med. 1999;153:879-886
formation among the clinics through regular district and statewide meetings.1 This intervention model has been given the acronym of AFIX (assessment, feed- back, incentives, and exchange), and in Georgia was associated with relatively low program costs (25% of the work time of 8 district health workers and 1 supervisor).2
Since 1995, all states and jurisdic- tions receiving federal immunization funds have been required to perform annual as- sessments of vaccination coverage in all public clinics (Senate appropriation lan- guage, 1995-1997), although the other components of the AFIX strategy have not been mandated. State immunization pro-
The affiliations of the authors appear in the acknowledgment section at the end of the article.

ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 879
©1999 American Medical Association. All rights reserved.

MATERIALS AND METHODS
SELECTION CRITERIA FOR INCLUSION
A state or city was required to meet 3 criteria: (1) to have performed annual vaccination coverage measurements in its clinics at least since 1994, a duration that permitted com- parison of clinic data with published results of the Na- tional Immunization Survey (NIS); (2) to have used the re- sulting information for feedback, incentives, and exchange of information; and (3) to provide us with clinic-specific coverage data.

IDENTIFICATION PROCESS
The 50 states and 14 jurisdictions in receipt of federal immunization funds are required to submit to the Cen- ters for Disease Control and Prevention annual reports that verify that they have met grant requirements, including measurement of public clinic coverage. We inspected these reports to identify candidate states and cities, then contacted program staff to determine if crite- ria were met.

ESTIMATION OF CLINIC COVERAGE
The Centers for Disease Control and Prevention provides a standard software package for entering and analyzing clinic vaccination data (Clinic Assessment Software Appli- cation, or CASA), and the methods by which it estimates coverage are described in detail elsewhere.3 In brief, the records of children aged 19 to 35 months who had been seen in the clinic are randomly sampled and, for those who had not been documented to have moved or gone elsewhere for care, dates of vaccination are entered into the program. Coverage rates are based on documented receipt of the 4-3-1 set of antigens: 4 doses of the diphthe- ria toxoid, tetanus toxoid, and pertussis vaccine; 3 doses of polio-containing vaccine; and 1 dose of the measles, mumps, and rubella vaccine.
ANALYSIS OF TRENDS IN CLINIC COVERAGE
We analyzed coverage data from each state in the same man- ner as had been previously used for Georgia.1 For each cycle of measurement, we created a distribution plot by rank or- dering all clinics in a locality by coverage rate, plotting this distribution in deciles, and determining the median. Change over time in these indicators of clinic coverage was the pri- mary study outcome.

INTERVENTION ACTIVITIES AND COSTS
We administered a questionnaire to determine the clinic measurement activities, how the data were used, the esti- mated proportion of the birth cohort served by the clinics for 1997 (the last year under analysis), and crude program costs in full-time equivalent employees (FTEs) and cur- rent annual expenses. We did not attempt to estimate in- direct costs.

POPULATION-BASED COVERAGE CHANGES
Since 1994, the NIS has provided population-based esti- mates of series-completion rates for children aged 19 to 35 months living in each of the 50 states and in 27 selected urban areas, with methods that have been described elsewhere.4 For each locality, we compared clinic measurements with published NIS estimates for 2 years: the first full year for which NIS provided data (July 1994 to June 1995)5 and the most recently avail- able comparable full year of data (July 1996 to June
1997).6

gram managers recently reported that coverage had been measured in approximately 75% of the nation’s estimated 4400 public clinics in 1996 (Centers for Disease Control and Prevention, unpublished data,
1998).
Only 1 or 2 years of data are available from states that began public clinic assessments in response to the federal requirement. The improvements seen in Geor- gia were slow and incremental (+6 percentage points per year), so it will be some time before the national impact of this initiative can be reliably assessed. How- ever, several states and cities began to implement com- ponents of the AFIX intervention in their localities before the congressional requirement. We examined data from these sources to help address the question of whether the Georgia methods and results could be reproduced elsewhere. This narrow question of repro- ducibility has large implications—if clinic coverage did not rise in localities that attempted to apply the Georgia model vigorously for several years, does the nationwide effort of clinic vaccination assessments need to be reevaluated?
RESULTS

IDENTIFICATION OF STATES AND CITIES MEETING CRITERIA

From the 64 reports, we identified 14 candidate states and cities that had conducted clinic coverage measure- ments at least since 1994. On further investigation, 3 were excluded for not having used the measurement data in a way that met study criteria. One had mea- sured clinic coverage but had not shared the informa- tion with clinics (coverage rates were reported to be static in the 60% range). A second had measured cov- erage in its public clinics and provided feedback, but had not offered incentives or opportunities for exchange of information (coverage rates were reported to be static in the 60% range). A third had used a pub- lic sector vaccination registry to make clinic-specific coverage data available to clinics and to provide reminder-recall to the patient population, but had not offered incentives or opportunities to exchange infor- mation (coverage rates were reported to have risen

ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 880
©1999 American Medical Association. All rights reserved.

over time to the 80% range, where they were stable). Five others reported having used the data in a way that met study criteria (all reported coverage improve- ments), but we were unable to obtain interpretable data: 2 because data were missing for certain years, 1 because the state’s software records could not be read by the Centers for Disease Control and Prevention software, and 2 because no data were supplied. Four states and 2 cities were included in the analysis.

INTERVENTION ACTIVITIES, COSTS, AND VACCINATION COVERAGE CHANGES

Localities are presented in order of the duration and size of their clinic assessment program. All programs are on- going. The total number of clinics in 4 localities varied from year to year, and the range is shown.

Missouri
The data for Missouri are shown in Figure 1. Cover- age has been measured annually since 1992 in each of the state’s 113 to 114 clinics (which provide vaccina- tions to an estimated 32% of the state birth cohort of 72 804), with 9273 to 16 368 records reviewed each year. At the time of each measurement, state staff assessed each clinic’s compliance with the Standards for Pediatric Immunization Practices.7 In a face-to-face feedback session conducted immediately after the mea- surement, clinic staff were provided with the coverage results and a set of recommendations of how coverage could be improved based on the standards. Rank-order lists of clinics and maps were published yearly in the state publication, Missouri Epidemiologist, with descrip- tions of the strategies of successful clinics. Nine state employees (with an annual salary of $33 636) devoted 10% of their work time to these activities, for a total of
0.9 FTE. The costs of supervision, clerical support, travel, and supplies were about $10 616, for a total esti- mated annual program cost of $40 888. Median clinic coverage rose from 44% in 1992 to 93% in 1997, or +49 percentage points over 5 years (+10 percentage points per year).

Louisiana
The data for Louisiana are shown in Figure 2. Cover- age has been measured annually since 1992 in each of the state’s 103 to 107 clinics (which provide vaccina- tions to an estimated 60% of the state birth cohort of 65 574), with approximately 16 000 to 18 000 records reviewed each year. Feedback was conducted immedi- ately after measurement of coverage, with subsequent faxing of a formal report. Rank-order lists of clinics and health districts were distributed during the annual state immunization conference, at which time awards were given for achievement. Ten immunization program spe- cialists (with an annual salary of $32 077) devoted 17% (2 of 12 months) of their time to these activities, for a total of 1.7 FTEs. The costs of supervision, clerical sup- port, travel, and supplies were about $25 000, for a total estimated annual program cost of $78 462. Median clinic coverage rose from 61% in 1992 to 83% in 1997, or +22 percentage points over 5 years (+4 percentage points per year).
Colorado
The data for Colorado are shown in Figure 3. Cover- age has been measured annually since 1993 in 73 to 91 of the state’s 91 clinics (which provide vaccinations to an estimated 30% of the state birth cohort of 54 050), with 4639 to 14 485 records reviewed each year. Feedback was provided in written form and mailed to the clinic subsequent to the measurement. A detailed annual compilation of every clinic’s perfor- mance was distributed to all clinics and health units. A
100 1992 1993
1994
1995
1996
1997
33 59 44
Figure 1. Distribution of clinics in Missouri by vaccination coverage. The percentage of children aged 19 to 35 months who were series complete based on the 4-3-1 set of antigens (as described in the “Estimation of Clinic Coverage” subsection of the “Materials and Methods” section) is shown on the x-axis. The percentage of clinics among the total number of clinics measured is shown on the y-axis. Each bar represents the proportion of clinics falling into each coverage increment. The arrow indicates the median clinic coverage; the bracketing numbers at the ends of the horizontal line, the 25th (left) and 75th (right) percentiles; and the length of the bracket, the interquartile distance around the median.
50
0
100
50
0
100
50
0
100
50
0
100
50
0
100
50
0
38 66 51

52 80 68

% of Clinics
71 87 80

83 92 90

90 95 93

0-9 10-19 30-39 50-59 70-79
20-29 40-49 60-69 80-89 90-100
% of Series Complete at Age 19 to 35 mo

ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 881
©1999 American Medical Association. All rights reserved.

100 1992 1993
1994
1995
1996
1997
100 1993 1994
1995
1996
1997
51 69 61
Figure 2. Distribution of clinics in Louisiana by vaccination coverage. See legend to Figure 1 for further details.
42 67 55
Figure 3. Distribution of clinics in Colorado by vaccination coverage. See legend to Figure 1 for further details.
50
0
100
50
0
100
50
0
100
50
0
100
50
0
100
50
0
50
0
100
50
0
100
50
0
100
50
0
100
50
0
56 75 65
51 78 64

61 81 70
57 86 73
% of Clinics
% of Clinics
67 87 79
63 87 79

70 88 81
60 87 75

0-9 10-19 30-39 50-59 70-79
20-29 40-49 60-69 80-89 90-100
% of Series Complete at Age 19 to 35 mo
73 91 83

0-9 10-19 30-39 50-59 70-79
20-29 40-49 60-69 80-89 90-100
% of Series Complete at Age 19 to 35 mo

quarterly state immunization publication highlighted several individual clinics’ achievements. At the annual statewide immunization conference, awards were given and clinics shared successful strategies. One public health nurse (with an annual salary of $63 833) devoted 60% and one administrative assistant (with an annual salary of $28 750) devoted 40% of their work time to these activities. The program is converting to registry-based measurements, involving 10% of the work time of a programmer (with an annual salary of $42 000), 30% of the work time of a data analyst (with an annual salary of $71 333), and 20% of the work time of a customer service representative (with an annual salary of $37 500), for a total of 1.6 FTEs. The costs of supervision, clerical support, travel, and sup- plies were about $15 000, for an estimated total annual program cost of $97 900. (This excludes other costs associated with the registry, for which the total yearly budget is approximately $250 000.) Median clinic cov- erage rose from 55% in 1993 to 75% in 1997, or +20 percentage points over 4 years (+5 percentage points per year).
Iowa

The data for Iowa are shown in Figure 4. Coverage has been measured annually since 1994 in each of the state’s 84 to 116 clinics (which provide vaccinations to an es- timated 32% of the state birth cohort of 35 233), with 11 119 to 13 660 records reviewed each year. Feedback was conducted immediately after measurement of cov- erage, using numerical and graphical output. A yearly re- port summarizing the results for every clinic in the state was mailed to each clinic and each county board of health, simultaneous with a state media release. At the annual statewide immunization conference, awards were pre- sented to clinics meeting 90% coverage goals, and suc- cessful clinics made presentations on their approaches. Six state employees (with an annual salary of $39 396) devoted 12% of their work time (1.5 of 12 months) to these activities, for a total of 0.7 FTE. The costs of su- pervision, clerical support, travel, and supplies were about $10 000, for an estimated total annual program cost of $38 685. Median clinic coverage rose from 71% in 1994 to 89% in 1997, or +18 percentage points over 3 years (+6 percentage points per year).

Boston, Mass

The data for Boston are shown in Figure 5. Coverage has been measured semiannually since 1994 in each of

ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 882
©1999 American Medical Association. All rights reserved.

100 1994 1995
1996
1997
100 1994 1995
1995
1996
1996
1997
60 80 71
Figure 4. Distribution of clinics in Iowa by vaccination coverage. See legend to Figure 1 for further details.
30 54 41

50
0
100
50
0
100
50
0
100
50
0
50
0
100
50
0
100
50
0
100
50
0
100
50
0
100
50
0
100
50
0
65 89 81
45 77 62
(Jun) (Oct) (Mar) (Sep) (Mar) (Sep)
0-9 10-19 30-39 50-59 70-79
20-29 40-49 60-69 80-89 90-100
% of Series Complete at Age 19 to 35 mo
% of Clinics
81 93 89
54 86 67

82 94 89
45 85 65
% of Clinics

0-9 10-19 30-39 50-59 70-79
20-29 40-49 60-69 80-89 90-100
% of Series Complete at Age 19 to 35 mo

57 87 77

the city’s 20 to 25 clinics (which provide vaccinations to an estimated 60% of the city birth cohort of 8020), with 3203 to 5276 records reviewed each measurement. Bos- ton has no clinics under the direct control of a health de- partment, and the clinics assessed were mainly commu- nity health centers and hospital-based clinics, as well as one private practice. The source of data was a citywide vaccination registry. The semiannual clinic report included a histogram showing the individual clinic’s position in the distribution of other clinics anony- mously rank ordered by coverage. A city health depart- ment nurse visited each site each month to monitor participation in the registry, to provide in-person feed- back on coverage data, to evaluate vaccination practices and strategies, and to share successful practices and strategies from other clinics. An annual ceremony at the Massachusetts Medical Society honored clinics with high coverage. A health department nurse and epidemi- ologist were dedicated full-time to the registry. The nurse devoted approximately 20% of her time to feedback-related activities, and the epidemiologist devoted approximately 5% of his time to generating coverage reports, for a total of 0.25 FTE. The costs of supervision, clerical support, travel, and supplies were about $8000, for a total annual program cost of $17 400. (This excludes other costs associated with the registry, for which the total yearly budget is approxi- mately $350 000.) Median clinic coverage rose from 41% in 1994 to 79% in 1997, or +38 percentage points over 3 years (+13 percentage points per year).

Houston, Tex

The data for Houston are shown in Figure 6. Coverage has been measured annually or semiannually since 1994 in each of the city’s 8 clinics (which provide vacci- nations to an estimated 40% of the city birth cohort of 33 145), with 714 to 1277 records reviewed each mea- surement. Immediately after measurement, one-on-one feedback was conducted with the immunization coordi- nator, chief nurse, and health care manager for each clinic. Certificates of recognition were awarded to clin- ics with the most improved coverage and the highest coverage, and these clinics’ practices and successes were described in the health department newsletter. Clinics attaining coverage of greater than 80% and greater than 90% were recognized at the state immunization confer- ence. The city immunization bureau paid for managers and immunization nurses from each recognized clinic to attend the state or federal immunization conference. Two registered nurses (with an annual salary of $32 618) and 3 licensed vocational nurses (with an
60 90 80

1997
64 90 79
Figure 5. Distribution of clinics in Boston, Mass, by vaccination coverage. See legend to Figure 1 for further details.

ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 883
©1999 American Medical Association. All rights reserved.

100 1994 1995
1995
1996
1997
1997
14 34 28
Figure 6. Distribution of clinics in Houston, Tex, by vaccination coverage. See legend to Figure 1 for further details.
ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 884
©1999 American Medical Association. All rights reserved.
50
0
100
50
0
100
50
0
100
50
0
100
50
0
100
50
0
14 32 21
(May)
(Dec) (Feb)
(Nov)
0-9 10-19 30-39 50-59 70-79
20-29 40-49 60-69 80-89 90-100
% of Series Complete at Age 19 to 35 mo
27 53 33

% of Clinics
46 83 62

59 90 80

75 96 84

annual salary of $24 465) devoted 10% of their work time to these activities, for a total of 0.5 FTE. The costs of su- pervision, clerical support, travel, and supplies were about $10 000, for an estimated total annual program cost of $23 863. Median clinic coverage rose from 28% in 1994 to 84% in 1997, or +56 percentage points over 3 years (+19 percentage points per year).

CHANGES IN MEASURED CLINIC COVERAGE COMPARED WITH THE NIS

In 1994, median clinic coverage was lower than the population-based coverage estimate in every locality. Between 1994 and 1996, the increase in clinic-based coverage was greater than the change in population- based coverage in 5 localities, and in the sixth it was identical. During the 2 years under examination, the average annual rate of rise of clinic coverage was +11 percentage points, compared with +1.2 for the popula- tions of the 6 localities and +1.5 for the nation as a whole (Table).
COMMENT

We report that the Georgia measurement and feedback immunization intervention system (AFIX) was repro- duced successfully in 4 states and 2 cities, at relatively low crude direct program cost, and was associated with improvements in clinic vaccination rates that exceeded population-based trends.
Part of the initial rise in clinic coverage is attribut- able to improvements in clinic record keeping. In 1994, the discrepancy between clinic and population-based coverage was large in localities that had just started measuring clinic coverage that year, eg, Boston (41% vs 86%) and Houston (28% vs 64%). Previous studies8-13 have suggested that vaccination coverage of children enrolled in health department clinics may be lower than that of the general population but not dramatically so (1%-10%). Since large fractions of the birth cohorts of each locality were vaccinated in the surveyed clinics (Boston, 60%; Houston, 40%), such initial clinic mea- surements are implausibly low. After 2 years of clinic assessments (1996), the discrepancies between clinic and population coverage had been markedly reduced: Boston (77% vs 85%) and Houston (62% vs 65%). These dramatic 2-year percentage point increases in clinic coverage (Boston, +36; Houston, +34) do not resemble the modest yearly effects of measurement and feedback observed elsewhere and strongly suggest documentation artifact.
The sensitivity of clinic-based coverage estimates to record-keeping accuracy was confirmed by studies conducted in 2 of the localities. A 1995 study14 of a large Boston clinic participating in the intervention demonstrated that coverage measured by the registry (which was the source of measurement data in the Boston intervention) would double if all administered vaccinations had been accurately recorded. In Colo- rado, registry-based measurements were substituted for chart audits in 24 of the 91 clinics during 1996- 1997, and median coverage in these clinics fell from 70% to 49%, producing a -4 percentage point decline in overall clinic coverage. A study15 conducted by the state suggested that inaccuracies in data entered into the registry (which had not previously been subject to the influence of the intervention) were responsible. These data are consistent with other studies16,17 of the accuracy of provider-based immunization coverage measurements.
Documentation artifact is unlikely to account for the coverage improvements seen after the first few years of clinic assessments. In 1994, the discrepancy between clinic and population-based coverage was minimal for localities that had more than 1 year of auditing clinic records, eg, Missouri (68% vs 71%) and Louisiana (70% vs 72%). Two years later (1996), clinic coverage in these 2 states had increased to relatively high levels (Missouri, 90%; Louisiana, 81%), and these clinic improvements were associated with population-based increases. In Louisiana, where most children were vac- cinated in public clinics, the coverage improvements for clinics and the general population were measured to be the same.

Changes in Immunization Coverage in Clinics Compared With a Population-Based Survey of Each Locality Locality
Birth Cohort*
Program Start Date
Vaccination Rates†

1994 1996 Change
Total
% Vaccinated by Clinic NIS‡ Clinics§ Difference NIS‡ Clinics§ Difference NIS‡ Clinics
Missouri 72 804 32 1992 71 ± 6 68 -3 75 ± 5 90 +15 +4 +22 Louisiana 65 574 60 1992 72 ± 5 70 -2 83 ± 4 81 -2 +11 +11 Colorado 54 050 30 1993 74 ± 6 64 -10 74 ± 5 79 +5 0 +15 Iowa 35 233 42 1994 82 ± 4 71 -11 81 ± 4 89 +8 -1 +18 Boston, Mass 8020 60 1994 86 ± 5 41 -45 85 ± 4 77 -8 -1 +36 Houston, Tex 33 145 40 1994 64 ± 8 28 -36 65 ± 6 62 -3 +1 +34 National\ ... ... ... 75±1 ... ... 78±1 ... ... +3 ...

*Based on 1996 annual immunization program reports to the Centers for Disease Control and Prevention from each locality. The birth cohort is based on provisional 1995 vital statistics data. The proportion vaccinated in clinics is the locality’s estimate based on doses administered data and other information. †Series-completion rates, based on the 4-3-1 set of antigens (described in the “Estimation of Clinic Coverage” subsection of the “Materials and Methods” section), for children aged 19 to 35 months.
‡NIS indicates National Immunization Survey. Results, with 95% confidence intervals, based the weighted, provider-verified immunization status of an average of 284 to 322 children in each survey area each year.4-6
§Median clinic coverage. The number of clinic records reviewed annually was 4635 to 18 000 in 73 to 116 clinics for states and 714 to 5276 in 8 to 25 clinics for cities. \Ellipses indicate that data are not applicable.
ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 885
©1999 American Medical Association. All rights reserved.
If we exclude as documentation artifact the first 2 years of clinic coverage improvements in each of the 6 localities, the average annual rate of improvement was +5 percentage points. Intensive investigation in Georgia suggested that the +6 percentage point annual coverage rise there was due to increased vaccination, a finding that is consistent with other studies17-21 of the impact of measurement and feedback. As modest as a +5 percent- age point annual rise may seem, it is considerably greater than the national population-based increase of +1.5 a year. During a decade, persistent +5 percentage point annual increments can take low immunization rates to high levels.
Estimated average annual costs of implementing the program were reported to be relatively low: 1 FTE (range,
0.3-1.7) implemented the program at an expense of $49 533 (range, $17 400-$97 900). These cost figures as- sume the existence of a public health infrastructure and do not include expenses incurred by the individual clin- ics in efforts to raise coverage. Such estimates, while crude, are comparable with those of Georgia (2.3 FTEs; ex- pense, $80 000).
The 4 states and 2 cities that provided data for this study are not representative of the rest of the United States: they began measuring clinic coverage before it was mandated and aggressively applied the other elements of the AFIX strategy in the absence of any requirement. Federal grant language requires only measurement of public clinic coverage, and evidence is lacking that measurement alone will raise immuni- zation rates.
Even fully implemented AFIX may fail in certain con- texts, or stall at certain levels. In several of the localities included in this analysis, a number of clinics had poor rates (,50%) despite years of exposure to the interven- tion. Median vaccination coverage at the end of the study showed considerable variability among the 6 localities (75%-93%), which was not readily explicable by differ- ences in program intensity or fiscal investment. Factors associated with individual clinic response to AFIX have been examined in Georgia22; and, as more data from more localities become available, it may be possible to iden- tify overall ecological factors associated with program- matic success.
In this study, we were explicitly attempting to an- swer a narrow question: Can the Georgia methods and results be reproduced elsewhere? The answer appears to be yes. The next question is one of generalizability: To what extent can the Georgia experience be reproduced throughout the United States?

Accepted for publication January 4, 1999.
From the National Immunization Program, Centers for Disease Control and Prevention, Atlanta, Ga (Drs LeBaron and Massoudi and Messrs Mercer, Dini, and Stevenson); Missouri Department of Health, Jefferson City (Mr Fischer); Louisiana Department of Health and Hospitals, New Orleans (Mr Loy); Colorado Department of Public Health and Environment, Denver (Ms Quick); Iowa Department of Public Health, Des Moines (Mr Warming); Boston Public Health Commission, Boston, Mass (Ms Tormey); and Houston Department of Health and Human Services, Houston, Tex (Dr DesVignes- Kendrick).
We thank Lance Rodewald, MD, Dennis O’Mara, Igor Bulim, Maureen Dempsey, Marilyn Kemna, Pamela Rice Walker, Ruben A. Tapia, and M. Anita Barry, MD, without whom this article would not have been possible. From Sep- tember 1997 through March 1998, Mr Mercer was as- signed to the National Immunization Program as a preven- tion specialist in the Public Health Prevention Service by the Division of Applied Public Health Training, Epidemiology Program Office, Centers for Disease Control and Preven- tion, Atlanta.
Reprints: Charles W. LeBaron, MD, MS E-61, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA 30333 (e-mail: cel3@cdc.gov).

REFERENCES

1. LeBaron CW, Chaney M, Baughman AL, et al. Impact of measurement and feed- back on vaccination coverage in public clinics, 1988-1994. JAMA.1997;277: 631-635.
2. Dini EF, Chaney M, Moolenaar RL, LeBaron CW. Information as intervention: how Georgia used vaccination coverage data to double public sector vaccination cov- erage in seven years. J Public Health Manage Pract.1996;2:45-49.
3. Centers for Disease Control and Prevention. Clinic Assessment Software Appli- cation (CASA): User’s Guide.Atlanta, Ga: Centers for Disease Control and Pre- vention; 1994.
4. Centers for Disease Control and Prevention. State and national vaccination cov- erage levels among children aged 19-35 months: United States, April- December 1994. MMWR Morb Mortal Wkly Rep.1995;44:146-150.
5. Centers for Disease Control and Prevention. National, state, and urban area vac- cination coverage levels among children aged 19-35 months: United States, July 1994-June 1995. MMWR Morb Mortal Wkly Rep.1996;45:100-104.
6. Centers for Disease Control and Prevention. National, state, and urban area vac- cination coverage levels among children aged 19-35 months: United States, July 1996-June 1997. MMWR Morb Mortal Wkly Rep.1998;47:108-116.
7. Ad Hoc Working Group for the Development of Standards for Pediatric Immu- nization Practices. Standards for Pediatric Immunization Practices.JAMA.1993; 269:1817-1822.
8. Bobo J, Gale J, Thapa P, Wassilak S. Risk factors for delayed immunization in a random sample of 1163 children from Oregon and Washington. Pediatrics.1993; 31:308-314.
9. Morrow A, Rosenthal J, Lakkis H. A population-based study of access to immu- nization among urban Virginia children served by public, private, and military health care systems. Pediatrics.1998;101:1-10.
10. Rodewald L, Peak R, Ezzati-Rice T, Zell E, Thompson K. Who are the immuni- zation providers for US children: findings from the 1994 National Health Inter- view Survey (NHIS) provider record check (PRC) [abstract]. Ambulatory Child Health.1997;3:168.
11. Massoudi M, LeBaron CW, Stokley S, Dini E, Belmont L, Schultz R. Vaccination levels and access to health care during a pertussis outbreak in a rural popula- tion. In: Program and abstracts of the Ambulatory Pediatric Association Meet- ing; May 3, 1998; New Orleans, La. Abstract 109.
12. LeBaron CW, Birkhead GS, Parsons P, et al. Measles vaccination levels of chil- dren enrolled in WIC during the 1991 measles epidemic in New York City. Am J Public Health.1996;86:1551-1556.
13. Maes E, Rodewald LE, Coronado V, Battaglia M, Izrael D, Ezzati-Rice T. Immu- nization providers for impoverished preschool children: results from the 1997 National Immunization Survey. In: Program and abstracts of the Ambulatory Pe- diatric Association Meeting; May 5, 1998; New Orleans, La. Abstract 365.
14. Samuels RC, Appel L, Reddy S, Tilson M. Tracking of immunizations: comput- ers may not be the magic bullet. In: Program and abstracts of the Ambulatory Pediatric Association Meeting; May 8, 1996; Washington, DC. Abstract 160.
15. Quick LS, Rotharmel P, Webb W, Meagher G, Hoekstra EJ. How valid is registry data for AFIX? Paper presented at: 31st Annual National Immunization Confer- ence; May 20, 1997; Detroit, Mich.
16. Murphy TV, Pastor P, Medley FB. Factors associated with unnecessary immu- nizations given to children. Pediatr Infect Dis J.1997;16:47-52.
17. Yawn BP, Edmonson L, Huber L, Poland GA, Jacobson RM, Jacobsen SJ. The impact of a simulated immunization registry on perceived childhood vaccina- tion status. Am J Manage Care.1998;4:185-192.
18. Buffington J, Bell KM, LaForce FM, the Genesee Hospital Medical Staff. A target- based model for increasing influenza immunizations in private practice. J Gen Intern Med.1991;6:204-209.
19. Kouides RW, Lewis B, Bennett NM, et al. A performance-based incentive pro- gram for influenza immunization in the elderly. Am J Prev Med.1993;9:250-
254.
20. Morrow RW, Gooding AD, Clark C. Improving physicians’ preventive health care behavior through peer review and financial incentives. Arch Fam Med.1995;4: 165-169.
21. Thompson RS, Taplin SH, McAfee TA, Mandelson MT, Smith AE. Primary and secondary prevention services in clinical practice: twenty years’ experience in development, implementation, and evaluation. JAMA.1995;273:1130-1135.
22. Dietz VJ, Baughman AL, Dini EF, Stevenson JM, Pierce BK, Hersey JC. Vacci- nation practices, policies, and management factors associated with high vac- cination coverage levels in Georgia public clinics. Arch Pediatr Adolesc Med. In press.

ARCH PEDIATR ADOLESC MED/VOL 153, AUG 1999 886
©1999 American Medical Association. All rights reserved.