Assessing Immunization Performance of Private Practitioners in Maine: Impact of the Assessment, Feedback, Incentives, and Exchange Strategy
Mehran S. Massoudi, PhD, MPH*; Jude Walsh‡; Shannon Stokley, MPH*; Jorge Rosenthal, PhD, MPH*; John Stevenson, MA*; Biljana Miljanovic, MD, MPH‡; Jack Mann, MD§; and Eugene Dini, MPA*

ABSTRACT. Introduction. A provider-based vaccina- tion strategy that has strong supportive evidence of effi- cacy at raising immunization coverage level is known as Assessment, Feedback, Incentives, and Exchange. The Maine Immunization Program, and the Maine Chapter of the American Academy of Pediatrics collaborated on the implementation and evaluation of this strategy among private providers.
Methods. Between November 1994 and June 1996, the Maine Immunization Program conducted baseline im- munization assessments of all private practices adminis- tering childhood vaccines to children 24 to 35 months of age. Coverage level assessments were conducted using the Clinic Assessment Software Application. Follow-up assessments were among the largest practices, delivering 80% of all vaccines.
Results. Of the 231 practices, 58 were pediatric and 149 were family practices. The median up-to-date vac- cination coverages among all providers for 3 doses of diphtheria-tetanus-pertussis vaccine and 2 doses of oral polio vaccine, and 4 doses of diphtheria-tetanus- pertussis vaccine, 3 doses of oral polio vaccine, and 1 dose of measles-mumps-rubella vaccine at age 12 and 24 months were 90% and 78%, respectively, and did not vary by number of providers in a practice or by spe- cialty. Urban practices had higher coverage than rural practices at 12 months (92% vs 88%). The median up- to-date coverage for 4 doses of diphtheria-tetanus- pertussis vaccine, 3 doses of oral polio vaccine, and 1 dose of measles-mumps-rubella vaccine at 24 months of age improved significantly among those practices assessed 1 year later (from 78% at baseline to 87% at the second assessment). On average, the assessments required 21/2 person-days of effort.
Conclusions. We document the feasibility and impact of a public/private partnership to improve immunization delivery on a statewide basis.
Implications. Other states should consider using public/private partnerships to conduct private practice assessments. More cost-effective methods of assessing immunization coverage levels in private practices are needed. Pediatrics 1999;103:1218–1223; immunizations, assessments, private providers, CASA, health services, per- formance measurement, managed care.
ABBREVIATIONS. DTP, diphtheria-tetanus-pertussis (vaccine); CDC, Centers for Disease Control and Prevention; MIP, Maine Immunization Program; AAP, American Academy of Pediatrics; AAFP, American Academy of Family Physicians; CASA, Clinic Assessment Software Application; 3:2, 3 diphtheria-tetanus- pertussis vaccine:2 oral polio vaccine; 4:3:1, 4 diphtheria-tetanus- pertussis vaccine:3 oral polio vaccine:1 measles-mumps-rubella vaccine; HEDIS, Healthplan Employer Data and Information Set.

Although the 1996 Childhood Immunization Initiative goals for disease incidence and vac- cination coverage levels were met,1 a signifi- cant gap exists between the current coverage levels and the Healthy People 2000 goals.2 For example, the coverage levels of diphtheria-tetanus-pertussis (DTP) and of hepatitis B vaccine are below the ob- jective of 90% for both vaccines. Currently, coverage for the fourth dose of DTP is 81% and for the third dose of hepatitis B is 84%,3 indicating that ;760 000 2-year-old children are incompletely protected against diphtheria, tetanus, and pertussis, and 640 000 2-year-old children are incompletely pro- tected against hepatitis B.
Strong evidence exists demonstrating the effective- ness of measuring the performance of immunization providers to improve the immunization coverage levels of their patients.4–6 Because of this proven effectiveness, the Advisory Committee for Immuni- zation Practices7 recommended, and the Centers for Disease Control and Prevention (CDC) required, health departments to assess the coverage levels of children served in their immunization clinics. The Standards for Pediatric Immunization Practices8 rec- ommends that providers conduct assessments of their immunization coverage levels on a routine basis.
However, during the 1990s, the delivery of vacci- nations to preschool children has been significantly privatized.9–12 Health department-operated clinics currently serve only 20% of preschool children,12 down from ;50% at the start of the decade.13 There- fore, assessment of only health department clinics will reach an insufficient number of children to achieve the year 2000 objectives; assessments must also reach children served by private providers if national objectives are to be met.
We report an evaluation of performance measure- ment with feedback conducted through a public/ private partnership of the Maine Immunization Pro- gram (MIP) and the Maine chapters of the American Academy of Pediatrics (AAP) and the American
From the *Centers for Disease Control and Prevention, National Immuni- zation Program, Atlanta, Georgia; the ‡Maine Department of Human Ser- vices, Bureau of Health, Immunization Program, Augusta, Maine; and the §Maine Chapter of the American Academy of Pediatrics, Augusta, Maine. Received for publication Aug 13, 1998; accepted Dec 21, 1998.
Reprint requests to (M.S.M.) CDC/NIP, 1600 Clifton Rd, MS E-52, Atlanta, GA 30333. E-mail: mrm8@cdc.gov
PEDIATRICS (ISSN 0031 4005). Copyright © 1999 by the American Acad- emy of Pediatrics.
1218 PEDIATRICS Vol. 103 No. 6 June 1999

Academy of Family Physicians (AAFP). The purpose of the intervention was to improve immunization coverage in the state by assessing immunization lev- els in all Maine immunization provider practices, private and public. The purpose of the evaluation was to determine the feasibility and impact of the assessment intervention.
METHODS
Setting and Study Participants
The intervention was conducted in 1994 through 1996 in Maine, which provides all Advisory Committee for Immunization Prac- tices recommended vaccines free of charge to all immunization providers, public and private.
Maine’s population in 1994 was 1 239 448; the annual birth cohort is ;14 000. Ninety percent of the vaccines administered to preschool children in Maine are given by private clinicians; the remaining 10% receive their vaccinations from 3 health depart- ment clinics and 30 community health centers (J. Walsh, MIP, personal communication, 1997). Maine is a high-performing state with regard to immunizations. The state ranked number 1 or 2 throughout the time period for the study, according to the CDC’s National Immunization Survey.1,3,14 Maine’s most recent 4 doses of DTP, 3 doses of oral polio vaccine, and 1 dose of measles-mumps- rubella (4:3:1) vaccine coverage level for children 19 through 35 months of age was 87%.3
Unit of analysis was private practice sites receiving vaccine from MIP (n 5 231 practice sites; 25% pediatricians, 65% family physician, 10% other). To receive vaccine from the state, providers sign an Immunization Provider Vaccine Agreement, which allows MIP staff to make site visits to evaluate vaccine handling and storage, verify record keeping, and conduct immunization cover- age assessments.
Intervention
Between November 1994 and June 1996, MIP assessed all pri- vate practices providing childhood vaccines. Assessment results on the immunization coverage levels and office practices associ- ated with immunization delivery problems, such as missed op- portunities for simultaneous vaccinations and delayed starting of immunizations, were reported to the office staff and clinicians. Between September 1996 and May 1997, follow-up assessments were conducted on 63 of the 231 practices. The practices selected for year 2 assessments were the largest practices, delivering 80% of all vaccines to Maine’s preschool children.
Assessments were conducted using procedures outlined in the CDC Clinic Assessment Manual using Clinic Assessment Software Application (CASA).15 CASA calculated practice immunization coverage levels and immunization practices for each participating site. For practices with 200 or more eligible patients, a systematic random sample of age-appropriate children seen in the practice was selected by state health department assessment staff, and immunization information was abstracted from the medical charts. For practices with fewer than 200 eligible patients, all records of age-appropriate children were abstracted. Eligibility criteria for inclusion of a child in an assessment were: 1) age between 24 and 35 months at time of assessment; 2) visited the practice at least 3 times; and 3) no documentation in the chart of having moved or gone elsewhere before the second birthday.
Once an assessment was completed, results were shared ver- bally with the providers and office staff. MIP then sent a written report within 3 weeks of the review. A letter discussing ways to improve vaccination practices and coverage levels in their practice accompanied the reports. Practices with coverage levels ,85% were visited by a MIP representative to discuss the assessments and diagnostic feedback to help improve immunization service delivery. The names of those practices with 4:3:1 coverage rates of 90% or better were published in the quarterly MIP newsletter, “Immunews,” and were officially recognized at an award lun- cheon sponsored by MIP.
Evaluation
The unit of analysis was private practices and representative sam- ples of their patients aged 24 to 35 months of age. The evaluation was descriptive and analytic, and included all practice sites eligible for the initial round of assessments. Practices were described in terms of the
1) type of provider (pediatrician, family physician, and other); 2) practice size; 3) practice location (rural versus urban); 4) number of providers in the practice; and 5) resulting immunization coverage levels at 12 and 24 months of age.
For the 24- to 35-month-old children selected for the study, the outcome measures were coverage levels, using the harmonized immunization schedule, at 12 months of age for 3 doses of DTP, 2 doses of oral polio vaccine (3:2), and at 24 months of age for 4:3:1. The analyses measured the change in immunization coverage levels between the first round and the second round of assess- ments among the sites selected for the second round of assess- ments. To measure the impact on the up-to-date status of 2 year olds, each practice’s 24-month round two coverage was subtracted from its round one coverage; the median change among all prac- tices was determined. Analysis included using Wilcoxon rank sum test to test for within-group differences and Wilcoxon signed rank test to detect differences between groups.
RESULTS Baseline
All 231 practices had an immunization assessment at baseline and 63 practices had a repeat assessment. All practices were given feedback about their performance. Of the 231 practices that were assessed at baseline, 126 (55%) practices were solo; 159 (69%) practices were rural practices. A total of 9076 patients were assessed; the median number and mean number of charts as- sessed in each practice were 21 and 39, respectively (Table 1). The average cost of these assessments to MIP was 21/2 person-days per assessment.
The median coverage for 3:2 vaccine for all practices for children 12 months of age was 90%; the median coverage for the 4:3:1 for children 24 months of age was 78% (Table 2). Table 2 also shows coverage levels by number of providers within the practice, specialty, geo- graphic location, and specialty by geographic location. Coverage levels at 12 months of age were higher for urban children than rural children, but this difference was not seen at 24 months of age. No relation was seen between immunization coverage and the number of providers in the practice.
Figure 1 displays the percent of all private prac- tices by coverage decile for children at 24 months of age. The median coverage was 78%; the 25th and 75th percentiles of 4:3:1 coverage were 64% and 88%, respectively. Eighty-one percent of practices had coverage levels .60%. Twenty-four percent of all private practices were at or greater than the year 2000 objective of 90% coverage for 4:3:1 series.

TABLE 1. Provider Characteristics
All Pediatrics Family
Practice Other*
Number of practices 231 58 149 24 Median sample size 21 88 13 16 Sample size range 1–222 3–222 1–136 1–41 Number of providers
in practice
1 126 29 81 16 2–3 83 22 54 7
.3 22 7 14 1 Geographic location
Urban 72 17 46 9 Rural 159 41 103 15
* Other includes nurse practitioner, general practitioner, obstet- rics/gynecology, and internal medicine.
ARTICLES 1219

TABLE 2. Median Up-to-date Coverage at 12* and 24† Months of Age
No. Practice 12 Months* 24 Months†

Median Coverage
P Value‡
Median Coverage
P Value‡
Up-to-date 231 90 78 Number of providers in practice
1 126 91 .30 78 .86 2–3 83 90 78
.3 22 8681 Specialty
Pediatrics 58 90 .97 82 .61 Family practice 149 90 76
Other 24 88 78
Practice sample size§
Below median 117 92 .06 77 .68 Above median 114 89 79
Geographic location
Urban 72 93 .04 79 .91 Rural 159 88 78
Specialty by residence type
Pediatrics:
Urban 17 93 .15 86 .10 Rural 41 89 81
Family practice
Urban 46 93 .07 78 .93 Rural 103 88 75
Other
Urban 9 88 .90 64 .34 Rural 15 88 83
* Up-to-date coverage at 12 months is defined as receiving 3 doses of DTP and 2 doses of polio vaccine. † Up-to-date coverage at 24 months is defined as receiving 4 doses of DTP, 3 doses of polio vaccine, and 1 dose of MMR. ‡ P values are results from Wilcoxon rank sum test.
§ Median sample size of practices is 21.
1220 IMMUNIZATION ASSESSMENTS AND FEEDBACK IN MAINE
 
Fig 1. Coverage levels for 4 diphtheria-tetanus-pertussis:3 oral polio vaccine:1 measles-mumps-rubella by 24 months of age (n 5 231 practices, 9076 children).

Follow-up
For the 63 largest practices, the median 4:3:1 cov- erage level increased from 78% at baseline to 87% at follow-up (P , .001) (Table 3). Significant improve- ments in coverage levels between baseline and fol- low-up were observed in all subgroups by specialty, number of physicians per practice, and residence type. Of the 63 large practices, the increase in cover- age level was observed for the largest practices (those greater than the median for practice size), as well as the smaller practices (those below the median for practice size).
The percent of practices with coverage level for 4:3:1 .90% improved from 5% at baseline to 36% at follow-up. The percent of practices with coverage level for 4:3:1 ,80% decreased from 54% at baseline to 26% at follow-up (Fig 2).
DISCUSSION
The partnership between the MIP and the Maine chapters of the AAP and AAFP demonstrated the feasibility of assessing all immunization practices for preschool children in 1 state. The intervention pro- tocol was conducted in all practices and then repli-

TABLE 3. Median Up-to-date Coverage at 24* Months of Age at Baseline and Follow-up Assessment
No. Practice Baseline Median
Coverage
Follow-up Median Coverage
Fig 2. Baseline and follow-up up-to-date coverage for 4 diphtheria-tetanus-pertussis:3 oral polio vaccine:1 measles-mumps-rubella at 24 months of age for 63 of the largest practices.
P Value†

Up-to-date coverage 63 78 87 ,.001 Number of providers
1 29 76 84 ,.001 2–3 24 80 90 .001
.3 10 72 84 .05 Specialty
Pediatrics 38 81 91 ,.001 Family practice 23 74 84 .07 Other‡ 2 69 67 ¶ Practice sample size§
Below median 32 75 85 .01 Above median 31 80 90 ,.001 Geographic location
Urban 15 75 92 .01 Rural 48 78 85 ,.001 Specialty by geographic location\
Pediatrics
Urban 9 83 94 .02 Rural 29 81 89 ,.001 Family practice
Urban 5 75 86 .13 Rural 18 73 83 .32
* Up-to-date coverage at 24 months is defined as receiving 4 doses of DTP, 3 doses of polio vaccine, and 1 dose of MMR. † P values are results from Wilcoxon signed rank test.
‡ Other includes nurse practitioner, general practitioner, obstetrics/gynecology, and internal medicine. § Median sample size of practices is 74.
\ Excludes practices classified as Other. ¶ Sample size too small for comparison.
ARTICLES 1221
 
cated among the practices serving the largest num- bers of children, and it included both pediatricians and family physicians. As a result of the interven- tion, measured immunization coverage levels among the larger practices increased from 78% in 1996 to 87% in 1997.
It is not possible to determine how much of the improvement in coverage was attributable to im- provement in the delivery of immunizations and how much was attributable to improved recording. Both are critical in achieving high immunization cov- erage levels because accurate documentation is a prerequisite for optimal immunization practices. Studies have examined the contribution of record scattering to undervaccination.16,17 They demonstrate the importance of accurate records and good com- munication of immunization histories among pro- viders. The high baseline coverage was reassuring that the practice documentation was quite good, be- cause poor documentation results in underestima- tion of coverage as measured by an assessment.
Assessment of coverage is a critical first step toward

improving performance. Because most providers over- estimate performance, often by .30 percentage points,18–21 assessments provide objective evidence to a practice that a problem exists. Without this recognition, a practice would not likely be motivated to improve. The overestimation of performance hinders public health from conducting assessments because providers frequently feel no problem exists, and thus see no need to have their records assessed. For Maine, the method used to overcome reticence to assess coverage came from the leadership of the AAP working with the state health department leadership to promote universal coverage assessments.
The impact of the Maine intervention is consistent with other impact studies of immunization perfor- mance measurement. Failure to administer the fourth dose of DTP vaccine during the 12- to 18- month age period was the most common reason for low series coverage at 24 months of age. The effec- tiveness of assessment with feedback has been dem- onstrated in a variety of settings, including health department clinics for routine childhood vaccina- tion4,22 and private practices of internists for influ- enza vaccine for adults .65 years of age.5,6,23–25 In North Carolina, an intensive academic detailing model was used to improve immunization coverage among private providers in three counties by 6 per- centage points. The intervention also resulted in im- proved lead screening and anemia screening.26
This evaluation advances this line of research be- cause Maine is the first state in the United States to assess the performance of all its childhood immuni- zation providers, both public and private. Currently, all state and urban-area immunization programs that are funded by the CDC are required to assess the performance of health department immunization clinics. The CDC recommends that immunization programs also assess performance of private provid- ers and other immunization providers not under the control of the health department. However, few pro- grams have made significant progress in the assess- ment of private providers; nationally, ,5% of private practices have had assessments (CDC unpublished data available at www.cdc.gov/nip/afix).
With the increasing privatization of preschool im- munization delivery, assessment of coverage levels in health department clinics, although important for the population they serve, will cover a smaller pro- portion of children in the United States. Thus, assess- ment of private providers will be critical, although more challenging, for several reasons: 1) there are many more private practices than there are health department clinics, and the average cost of 21/2 per- son-days per assessment makes this a labor-intensive intervention. For example, the Vaccines for Children program currently has 33 706 private practices en- rolled (October 1997, Vaccines for Children Program Implementation Survey). Thus, the number of assess- ments will require substantial resources. 2) Because medical homes for primary care deliver many ser- vices in addition to vaccinations, the performance of routine immunization delivery may be overwhelmed by other priorities in the practice. Assessments are already being conducted by managed care organiza- tions and peer-review organizations in many offices. An immunization performance assessment is often viewed as an additional burden. 3) Few states re- quire the assessment of private sector immunization performance. Some states, including Maine, do re- quire performance measurement; however, such an arrangement is unusual. Thus, the vast majority of states will have to rely on a partnership between private providers and immunization programs to accomplish large-scale performance measurement. 4) Although acceptable methods for private provider assessments exist, optimal methods have yet to be developed.27 The CDC’s CASA software, which was used for the Maine assessments, was developed ini- tially to assess health department clinics. Because CASA requires a simple random sample of patients in a specified date-of-birth range, the ability to pro- duce a computer listing sample frame is helpful. However, many offices do not have that capability and have to use a hand-search method to identify CASA-eligible subjects.
Immunization performance measurement is also be- ing accomplished for children enrolled in managed care organizations through the National Committee for Quality Assurance’s Healthplan Employer Data and Information Set (HEDIS). One intent of HEDIS is to provide purchasers of health care with the ability to compare the performance of different managed care plans. However, for the individual provider, HEDIS is likely to sample only a few of his or her patients, making it impossible for HEDIS to provide any mean- ingful diagnostic capability for the individual provider to increase his or her immunization coverage levels.
Although assessment of private providers is chal- lenging, the Maine experience demonstrates that it is feasible and effective on a statewide level. Given that Maine’s coverage levels were among the highest in the nation, implementation of the Assessment, Feed- back, Incentives, and Exchange strategy in states with lower coverages should yield similar or larger increases. Key features of the Maine intervention included close collaboration between the health de- partment and the Maine Chapters of the AAP and AAFP through routine attendance of immunization program staff at AAP meetings, attendance and par- ticipation of many private providers in the statewide immunization conferences, membership of AAP and AAFP representatives on the advisory committee of MIP, and regularly scheduled conference calls.

Limitations
The percentage-point increase shown by this inter- vention may not generalize to other states given that Maine has consistently been a top performing state with regard to preschool immunization coverage levels as reported by the National Immunization Survey. Lower-performing states have more room to improve and may find percentage improvements different from those of Maine. We reported provider-validated immu- nizations; however, we did not link records across pro- viders. Thus, our results may underestimate the true practice coverage. Maine has also had limited penetra- tion of managed care and thus these results should be interpreted appropriately.

1222 IMMUNIZATION ASSESSMENTS AND FEEDBACK IN MAINE

Ideally, the evaluation of assessment as an inter- vention would have been based on comparisons to a true control group. Because Maine chose to assess all private practices at baseline, this remains as a limi- tation in interpreting the results of this study.
For the purposes of calculating practice coverage levels, Maine defined an active patient as a child mak- ing three or more visits to the practice. This is different from the criterion used by health departments for their clinic assessments, which include any child making one or more visits as an active patient. The decision to use three or more visits was made through discussions of the MIP staff, the local AAP and AAFP, and other providers, and represented a consensus agreeable to all parties. Future assessments will evaluate children who have made at least one visit to the practice. This will allow Maine to track the children who may be at greater risk for underimmunization.

Recommendations
This evaluation supports the recommendation that public health departments collaborate with their pro- vider organizations, such as the AAP and AAFP, to encourage conducting assessments in the private sector as part of the Assessment, Feedback, Incentives, and Exchange implementation plan. The experience of Maine may be used as a model of a successful public/ private partnership because it resulted in assessing all private and public provider sites. However, the cost of the assessment is a barrier to the successful implemen- tation of universal assessment of immunization provid- ers, and more cost-effective methods of assessing im- munization coverage levels in private practices are needed. Specific questions to address include: 1) the use of smaller sample sizes for assessments; 2) the validity and utility of self-assessments; 3) the use of more convenient sample frames in a practice than sim- ple random samples (eg, sequential patient visits;27) and 4) linkage of the assessment of other clinical pre- ventive services, such as lead and anemia screening, into immunization assessments.

ACKNOWLEDGMENTS
We would like to thank the following people for providing helpful comments on the manuscript: Lance Rodewald, MD; Ed- ward Brink, MD; Rafael Harpaz, MD; and Charles LeBaron, MD. We also thank Joseph Beaver for his helpful comments in support of Maine’s immunization program. MIP field staff who conducted the many assessments at the provider offices were: Barbara Ray- mond, Martin Sabol, Chriss Hayden, Jeremy Black, Gilman Beau- lieu, and Agnes Jankoski. We also acknowledge the support of the Maine Chapter of the American Academy of Pediatrics; the Maine Chapter of the AAFP; and Dora Anne Mills, MD, MPH, Director, Maine Bureau of Health.
REFERENCES
1. Centers for Disease Control and Prevention. National, state, and urban area vaccination coverage levels among children aged 19–35 months— United States, 1996. MMWR Morb Mortal Wkly Rep. 1996;46:657–671
2. US Department of Health and Human Services. Healthy People 2000. Public Health Service, DHHS Publication No. 91–50213. Washington, DC: US Government Printing Office; 1990
3. Centers for Disease Control and Prevention. National, state, and urban area vaccination coverage levels among children aged 19–35 months— United States, 1997. MMWR Morb Mortal Wkly Rep. 1997;47:547–554
4. LeBaron CW, Chaney M, Baughman AL, et al. Impact of measurement and feedback on vaccination coverage in public clinics, 1988–1994. JAMA. 1997;277:631–635
5. Buffington J, Bell KM, LaForce FM, the Genesee Hospital Medical Staff. A target-based model for increasing influenza immunizations in private practice. J Gen Intern Med. 1991;6:204–209
6. Kouides RW, Lewis B, Bennett NM, et al. A performance-based incen- tive program for influenza immunization in the elderly. Am J Prev Med. 1993;9:250–255
7. Advisory Committee on Immunization Practices, Centers for Disease Control and Prevention. Programmatic strategies to increase vaccina- tion rates—assessment and feedback of provider-based vaccination cov- erage information. MMWR Morb Mortal Wkly Rep. 1996;45:219–220
8. National Vaccine Advisory Committee. Standards for Pediatric Immuni- zation Practices. JAMA. 1993;269:1817–1822
9. Freed GL, Clark SJ, Pathman DE, Schectman RM. Differential effects of a state universal purchase program on children’s immunization rates according to insurance status. Abstract. Presented at the Pediatric Ac- ademic Societies Annual Meeting; May 1–5, 1998; New Orleans, LA
10. Szilagyi PG, Humiston SG, Shone LP, Barth R, Kolasa MS, Rodewald LE. Reduction in vaccinations delivered by health department clinics: impact of vaccine financing? Abstract. Presented at the Pediatric Aca- demic Societies Annual Meeting; May 1–5, 1998; New Orleans, LA
11. Rodewald L, Peak R, Ezzati-Rice T, Zell E, Thompson K. Who are the immunization providers for US children: findings from the 1994 Na- tional Health Interview Survey (NHIS) Provider Record Check (PRC). Ambulatory Child Health. 1997;3:168
12. Maes EF, Rodewald L, Coronado VG, et al. Who is immunizing children in the US: public or private providers? Abstract. Presented at the Pediatric Academic Societies Annual Meeting; May 1–5, 1998; New Orleans, LA
13. The National Vaccine Advisory Committee. The measles epidemic—the problems, barriers, and recommendations. JAMA. 1991;266:1547–1552
14. Centers for Disease Control and Prevention. National, state, and urban area vaccination coverage levels among children aged 19–35 months— United States, January-December 1995. MMWR Morb Mortal Wkly Rep. 1995;46:176–182
15. Centers for Disease Control and Prevention. Clinic Assessment Software Application (CASA): User’s Guide. Atlanta, GA: Centers for Disease Con- trol and Prevention; 1994
16. Hamlin J, Wood D, Pereyra M, Grabowski M. Inappropriately timed immunizations: types, causes, and their relationship to record keeping. Am J Public Health. 1996;86:1812–1814
17. Yawn BP, Edmonson L, Huber L, Poland GA, Jacobson RM, Jacobsen SJ. The impact of a simulated immunization registry on perceived child- hood immunization status. Am J Managed Care. 1998;4:186–192
18. Watt J, Kahane S, Smith N, et al. The difference between measured and estimated vaccination coverage among private physicians in California. Abstract. Presented at the Pediatric Academic Societies Annual Meeting; May 1–5, 1998; New Orleans, LA
19. Bordley WC, Margolis PA, Lannon CM. The delivery of immunizations and other preventive services in private practices. Pediatrics. 1996;97: 467–473
20. Szilagyi PG, Roghmann KJ, Campbell JR, et al. Immunization practices of primary care practitioners and their relation to immunization levels. Arch Pediatr Adolesc Med. 1994;148:158–166
21. Bushnell CJ. The ABC’s of Practice-based Immunization Assessments. Pro- ceedings of the 28th National Immunization Conference. Public Health Service. Washington, DC: US Department of Health and Human Services; 1994:207–209
22. Dini EF, Chaney M, Moolenaar RL, LeBaron CW. Information as intervention: how Georgia used vaccination coverage data to double public sector vaccination coverage in seven years. J Public Management Pract. 1996;2:45–49
23. Barton MB, Schoenbaum SC. Improving influenza vaccination perfor- mance in an HMO setting: the use of computer-generated reminders and peer comparison feedback. Am J Public Health. 1990;80:534–536
24. Karuza J, Calkins E, Feather J, Hershey CO, Katz L, Majeroni B. En- hancing physician adoption of practice guidelines. Dissemination of influenza vaccination guideline using a small-group consensus process. Arch Intern Med. 1995;155:625–632
25. Korn JE, Schlossberg LA, Rich EC. Improved preventive care following an intervention during an ambulatory care rotation: carryover to a second setting. J Gen Intern Med. 1988;3:156–160
26. Bordley WC, Margolis PM, Jayne S, Lannon CM, Keyes LL. Office-based assistance to enhance preventive services delivery. Abstract. Presented at the Pediatric Academic Societies Annual Meeting; May 1–5, 1998; New Orleans, LA
27. Darden PM, Taylor JA, Slora EJ, et al. Methodological issues in deter- mining rates of childhood immunizations in office practice. A study from the pediatric research in office setting (PROS) network. Arch Pediatr Adolesc Med. 1996;150:1027–1031
ARTICLES 1223