Detailed Information on the
Childhood Immunization Program Assessment

Explicitly tie budget requests to the accomplishment of annual and long-term goals, and present resource needs in a complete and transparent manner.

Will review administrative functions to determine whether improvements in program operations and efficiency can be made.

Improve mechanisms linking the program's budget for state immunization program operations activities to program performance.

Will continue a comprehensive evaluation of the program and will work with grantees to better measure outcomes and allocate resources based on more clear criteria.

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by cases of polio, rubella, measles, congenital rubella, mumps and tetanus.

Explanation:Please refer to Annual Measures for individual disease cases targets and actuals. Target:Goal is 0: Polio (from 0), Rubella (from 181 in 1997), Measles (From 81 in 1997), Diphtheria (from 3 in 1997), Congenital Rubella (from 5 in 1997), Mumps (from 683 in 1997), Tetanus (From 50 in 1997) Actual Progress achieved toward goal: 2001 Data: Polio: 0; Rubella: 19; Measles: 61; Hib: 183; Diphtheria: 2; Congenital Rubella: 2; Tetanus: 27, Mumps: 231.

Measure: Percentage of children 19-35 months of age who receive recommended vaccines every year.

Explanation:The ACIP Recommended Childhood and Adolescent Immunization Schedule recommends routine vaccination of children for the above diseases. As childhood immunization coverage rates increase, cases of vaccine-preventable diseases decline significantly. To sustain current high coverage rates and increase coverage rates for vaccines that have not yet reached the 90 percent target, CDC provides funding, guidance, and technical assistance to state and local immunization programs so that they may conduct provider assessments, develop and utilize immunization information systems, utilize coverage assessment information from the National Immunization Survey, and provide education and training to both public and private immunization providers.

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by cases of measles.

Explanation:Performance Target: FY 04: measles 50; Actual Performance: FY 2001: measles: 61

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by indigenous cases of paralytic polio.

Explanation:Performance Target: FY 04: polio: 0; Actual Performance: FY 2001: polio: 0;

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by cases of rubella.

Explanation:

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by cases of congenital rubella syndrome.

Explanation:

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by cases of Tetanus.

Explanation:

Measure: Number of cases of vaccine-preventable diseases in the United States as measured by cases of mumps.

Explanation:This measure tracks the number of mumps cases in the country. A large national mumps outbreak occurred in December 2005 and continued through FY 2006. Although the outbreak was concentrated in the Midwest, most states reported some increase in number of mumps cases. As a result of this outbreak, vaccination recommendations were modified in 2006 to better define evidence of immunity, ensure routine two-dose vaccination for high risk adult groups including college students and healthcare workers, and address additional vaccination needs for persons in outbreak settings. CDC is working to ensure that lessons learned from the 2005-2006 outbreak and specific enhancements in mumps prevention and control are fully applied to reverse the increase in disease cases. As a result of the outbreak, CDC will work with state health departments, and with laboratory, immunology and mumps disease subject matter experts to reassess the current HP 2010 target and determine if changes need to be made for the HP 2020 targets.

Measure: Make vaccine distribution more efficient and improve availability of vaccine inventory by reducing the number of vaccine inventory depots in the U.S.

Explanation:VMBIP (Vaccine Management Business Improvement Project) is a CDC initiative aimed at increasing the efficiency, visibility, and management of publicly purchased vaccines. A key aspect of VMBIP is the centralization and consolidation of vaccine inventory and distribution. Currently, publicly purchased vaccine, including vaccine purchased by the Section 317 program, is held at various third party distribution depots or in state run depots. It is estimated that 400 storage locations exist. Even though the current system works, it is inefficient. A high number of depots results in redundancy of distribution resources, loss of distribution efficiency, and impedes the program's ability to track vaccine. These factors increase the likelihood vaccines could be lost. CDC, through VMBIP, aims to centralize distribution under a federal contract that will utilize third party commercial distributors to consolidate national inventory in significantly fewer inventory depots than currently exists. The expected efficiencies gained from consolidation of vaccine depots include improved management of vaccine inventory through use of distribution best practices and increased visibility of the location of vaccines throughout the public vaccine supply chain. As VMBIP implementation progresses through a phased approach, the number of locations holding vaccines will decrease. Full implementation of this new vaccine purchase and distribution operating model is anticipated to gain additional efficiencies by reducing vaccine wastage and reducing inventory holding costs.

Is the program purpose clear?

Explanation: The goal of the National Immunization Program (NIP) is to prevent disease, disability and death in children (and increasingly) adults through vaccination. NIP is comprised of two primary grant programs to states - 1) the discretionary 317 program; and 2) the mandatory Vaccines for Children (VFC) program. The 317 grant program provides some vaccines for those who are not eligible to receive vaccines under any other insurance program, but primarily focuses on assuring vaccines for the entire population through: 1) public information and outreach; 2) quality assurance within the medical community; 3) assessment of immunizations within the population; 4) surveillance of disease and vaccine safety; 5) immunization registries; 6) vaccine management. CDC also supports global efforts such as eradicating polio and eliminating measles because to eliminate/eradicate diseases in the U.S. completely it is necessary to eliminate/eradicate them internationally.

Evidence: Cited in the NIP Strategic Plan mission and GPRA plan. The 317 program is authorized through the Public Health Service Act Section 317j, to provide vaccines for individuals (later specified as children, adolescents and adults) free of charge and to provide preventive health services related to the delivery of immunizations. With the establishment of VFC in 1994, the 317 program shifted more of its efforts towards vaccine assurance rather than direct provision of vaccines. For global activities, Congress authorizes NIP's global activities through appropriations language and NIP's strategic plan includes a goal to eliminate and eradicate diseases globally as well as domestically. However, there is no clear guiding principle for how CDC prioritizes its global activities other than that CDC works closely with WHO and its priorities to determine what international activities to undertake.

Does the program address a specific interest, problem or need?

Explanation: CDC focuses on activities (including service delivery and supportive services) to ensure that children domestically (and increasingly adults) and internationally receive the appropriate and recommended vaccines. CDC is also using the 317 program to try and reach "pockets-of-need," or specific populations where immunization rates are much lower than the national average.

Evidence: In the U.S., 11,000 babies are born each day that must be vaccinated (approximately 4 million per year), and need to receive 12-16 doses of vaccine by 18 months, and 16-20 doses through childhood. The immunization rates for newer vaccines such as varicella and Hep. B have not yet reached 90 percent coverage. 317 also serves as a gap-filler for those children who are not receiving vaccines from any other provider.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: Although there are no good estimates for how much states contribute to vaccine purchase/infrastructure activities, NIP estimates that it provides the majority of the public funding for vaccine purchase and assurance activities. For vaccine purchase, the Federal contribution (both 317 and VFC) represents a majority of the funds (a 2000 IOM report estimates the state contribution to vaccines on the Federal contract ranges from less than 10 to 30 percent) so that increases and decreases in Federal vaccine purchase funds will have an impact on coverage levels.

Evidence: For vaccine purchase, in FY 2001, CDC estimates that states provided $116 million in purchases through the Federal contract (excluding how much states spent independently purchasing vaccines), while CDC spent $201 million in 317 funds. NIP has helped increase overall childhood immunization rates from 55 percent in 1992 to an all-time high of approximately 80% in 2000.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: The 317 program provides vaccines for those that do not receive vaccines through other private or public insurance programs (largely the underinsured with large copayments), and also supports outreach, education, and quality assurance activities.

Evidence:

Is the program optimally designed to address the interest, problem or need?

Explanation: CDC provides direct financial assistance to grantees for infrastructure activities and a line of credit for vaccine purchase since it is from a single contract.

Evidence:

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: CDC's overall outcome goal is to reduce the number of indigenous cases of vaccine preventable diseases in the U.S. to 0 by 2010. NIP uses Healthy People 2010, its strategic plan and GPRA to guide and measure its activities. The five-year strategic plan (2000-2005) is more qualitative and process-oriented, and is more of a vision document to help guide CDC's overall activities, while GPRA is used to measure progress on achieving specified Healthy People 2010 goals.

Evidence: Strategic Plan examples: 1) Eradicate/eliminate/control all vaccine-preventable disease disability and death in the U.S. and globally ; 2) Raise and sustain vaccine coverage levels in all populations for all recommended vaccines.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: The GPRA plan includes several goals to help measure progress on this long-term goal annually including vaccine coverage levels, annual targets for specific diseases, and global polio eradication efforts.

Evidence: Examples: 1) The number of indigenous cases of: a) measles will go from 63 in FY 2000 to 60 in FY 2002 to 50 in FY 2004; b) rubella will go from 176 in FY 2000 to 20 in FY 2002 to 15 in FY 2004; c) Hib from 183 in FY 2001 to 175 in FY 2002 to 150 in FY 2004; c) polio will remain at 0; 2) achieve or sustain immunization coverage of at least 90% in children 19-35 months of age for recommended vaccines each year; 3) achieve and sustain zero cases of polio by 2005.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: In the FY 2003 grant announcement, NIP will require grantees to develop measurable outcomes in relation to five of its GPRA goals. Previously, NIP included 15 HP 2010 goals as the objectives that grantees should be working towards and reporting progress on in their applications.

Evidence: In FY 2003, grantees will be required to develop measurable objectives in relation to the following GPRA goals: 1) Reduce the number of indigenous cases of vaccine-preventable disease; 2) ensure that 2 year-olds are appropriately vaccinated; 3) improve vaccine safety surveillance; 4) increase routine vaccination coverage levels for adolescents; 5) increase the proportion of adults who are vaccinated annually against influenza and ever vaccinated against pneumoccocal diseases. Previously, grantees were required to develop and measure progress on their own objectives that were in support of CDC's overarching goals.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: CDC leverages the National Vaccine Program Office to coordinate activities among different HHS agencies. CDC collaborates closely with NIH on IOM vaccine trials and CMS on the development of GPRA goals, reimbursement rates, and administration fees.

Evidence:

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: In 2003, the program drafted a proposal and has entered into a contract to have an independent party conduct a comprehensive evaluation. The first phase of the evaluation will focus on the 317 program and will be paid for in FY 2003 and completed in one year. The evaluation will provide information about the interaction with the Vaccines for Children program. The program is also planning internal reviews to improve strategic planning, management, cost controls and efficiency. While NIP has undertaken several management evaluations over the past few years to see if certain aspects of the program can be improved, there have previously been no comprehensive evaluations looking at how well the program is structured/managed to achieve its overall goals. A 2000 IOM report, while comprehensive in scope, focused more on how the Federal government could improve its ability to address childhood immunizations rather than evaluating how well the 317 and VFC programs, as currently structured and operated, were improving immunization rates among children.

Evidence: Evidence includes the program revised submission and outline of focus areas for the new evaluation. Two divisions of the program have had an independent review of their management structure and operations within the last few years; NIP recently undertook an evaluation of its NIP-wide IT systems, which will have recommendations in the Fall; an independent contractor was brought in to review and help develop the NIP strategic plan; NIP brought in an independent contractor to review its indirect cost rates.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: For the vaccine purchase activities, yes, for state infrastructure, no. For the infrastructure activities, there are a lot of different activities that comprise infrastructure (education, outreach, administration of vaccines), so it's unclear exactly how funding/policy/legislative changes will affect performance. The program is able to show after the fact the impact of changes in funding levels.

Evidence: There is no specific mechanism or measurement that links NIP's infrastructure budget and activities to its performance goals.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The planned evaluation described in Question 6 of this section is to provide guidance on improving the alignment of the program's budget with performance measures and information. The program anticipates this evaluation will help the program determine how budget alignment can be improved. The program is also working to develop logic models of 317 outputs. The program has made additional progress on the strategic plan and refinement of performance measures.

Evidence: Evidence includes the program revised submission and outline of focus areas for the new evaluation.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: CDC collects grantee information from a variety of sources including annual progress reports from states, a financial status report, and at least one site visit per year. CDC also receives information quarterly from the National Immunization Survey (NIS) on immunization coverage across all 50 states, and disease surveillance information. CDC is moving towards a more formula-based grant in FY 2003 that will take into account more objective criteria, including performance. NIP's project officers have constant contact with grantees to determine if a change in program direction is warranted. NIP also conducts quality assurance reviews of private providers to make sure that they are administering the vaccines properly, and storing/rotating them.

Evidence: Disease rates from surveillance and the National Immunization Survey have helped CDC determine internal priorities (e.g., what diseases/populations scientists should be looking at), and their activities in collaboration with states, as well as how well their grantees are achieving immunization coverage levels. For grantees, if CDC sees that there are low immunization levels within a jurisdiction, CDC may provide technical assistance or direct additional funds to this area.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: NIP's Federal program managers, while responsible for cost and schedule, do not have performance-based contracts that integrate program performance into their personnel evaluations. Within CDC, only SES have performance-based contracts and NIP has no SES. For grantees, while NIP reviews grantees' vaccine coverage levels and progress reports to determine if they are meeting their stated objectives, NIP doesn't reallocate funds as a result of grantees not meeting their objectives, and tends to provide technical assistance instead. CDC is in the process of initiating performance contracts for center and division directors, but has not gone through all of the steps to put them in place at this time. The program also is updating the AFIX and Provider Quality Assurances to improve physician practices. A new review panel is planned to improve accountability of grantees.

Evidence: Evidence includes the agency submissions.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NIP generally obligates almost all of its funds by the end of the year, and has many mechanisms to make sure that grantees spend their funding for the intended purpose.

Evidence: Grantees tend to have less than 10% of their obligations carried over to the following year (approx. $1,000-$100,000) and have to use their carry-over in lieu of new funds. NIP also conducts site visits to assess grantee obligation patterns and how funds are spent, and interacts frequently with grantees through conference calls to monitor activities and progress. Grantees are required to provide a detailed budget by object class, so if they want to move funds around they have to notify CDC. CDC's central program and grants office has also started site visits to focus on management/funding issues.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program hired a contractor to do a baseline assessment of IT activities and is consolidating all IT into the office of the director. The change realigns branches and eliminates a division. A second phase of the effort will examine administrative staff to determine available efficiencies and savings. The operations manual includes efficiency measures on vaccine wastage that grantees report on to CDC. Improvements in efficiency is also a focus of a new evaluation being contracted by the program. The program has committed to additional efficiency measures and further steps to put procedures in place to regularly review potential efficiencies and cost-effectiveness in administering the program are warranted. Additional steps to improve the efficiency of vaccine distribution should be examined.

Evidence: Reorganization plans were announced in March 2003. Efficiencies: NIP is converting to some electronic processing, including its disease reporting system, vaccine ordering system, and collecting records from providers to improve efficiency, and is undertaking a comprehensive review of its IT positions/activities. While CDC centrally cost-competes for certain procurement and other administrative activities, the program doesn't cost-compete for services. Cost-Effectiveness: There are no dollars per unit service. CDC has achieved some cost savings in vaccine purchase through having a single Federal contract, contracts with multiple manufacturers and re-competing vaccine bids every four years. NIP also contracts with GSA to help states establish vaccine registries.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: While CDC includes the full cost of its activities including overhead, program performance cannot be readily identified with changes in funding levels.

Evidence: Evidence based on GPRA plans and reports and budget justifications.

Does the program use strong financial management practices?

Explanation: The FY 2002 report noted reportable conditions relating to information systems; the internal controls over preparation, analysis and monitoring of financial information, including manually intensive procedures; reimbursable agreements; and grants accounting and oversight. None of the reportable conditions are considered material internal control weaknesses. CDC has actively addressed key areas. CDC automated reimbursable billings, enhanced year end closing transactions and implemented a new indirect cost methodology. CDC is also addressing staffing needs, including core accounting competencies, professional staff recruitment, financial systems, training and customer service.

Evidence: Evidence includes the FY 2002 Chief Financial Officers annual report, including summary of reportable conditions, summary documents on end of year balances, OIG reports (e.g., CIN-A-04-98-04220). Four areas of findings were also documented the prior year. CDC has received five consecutive unqualified opinions on the agency's financial statements. Additional data include that CDC issued 64 duplicate or erroneous payments in FY 2002, or 0.042% of all payments and has a 97% compliance rate for prompt payments.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: As noted above, the agency is actively addressing financial management. In its FY 2003 application, NIP is trying to formalize its application criteria, requiring grantees to provide more quantifiable objective information in its application and annual progress reports, and developing more clear evaluation criteria. NIP has also contracted with a firm to review its IT organizational structure and develop a 5-year plan to help improve the efficiency of NIP. As noted above, the program is also planning performance contracts for federal managers once the CDC executive team performance plans are in place. A review panel is being established for fall grantee reviews to improve consistency of awards and oversight of grantees.

Evidence: Grantee applications will be ranked based on: 1) plan; 2) objectives; 3) methods; 4) evaluation; previously, grantees were primarily funded based on population and need.

Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process?

Explanation: NIP assigns project officers to review the applications and determine how much funding each state should receive. Before FY 2003, the funding decisions were based upon the information included in individual grantee applications, taking into account historical funding levels and factors like state need/population/poverty levels. In FY 2003, CDC is formalizing this process to include clear criteria for allocating resources.

Evidence: In FY 2003, NIP will use the following criteria to rank applications: 1) plan; 2) objectives; 3) methods; 4) evaluation.

Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process?

Explanation: NIP provides funding to all 50 states.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: CDC collects information from a variety of sources, including disease surveillance reports, annual progress reports, and site visits. States also conduct annual program reviews of local health departments and intensive reviews of immunization clinics.

Evidence:

Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NIP makes both aggregate and state performance information on coverage levels and disease burden available through its website and Morbidity and Mortality Weekly reports.

Evidence:

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: CDC has made significant progress in achieving its long-term goals.

Evidence:

Does the program (including program partners) achieve its annual performance goals?

Explanation: CDC has largely achieved its annual goals.

Evidence:

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: While NIP has achieved some cost savings through negotiating a single Federal contract, the program does not have a stated efficiency or cost-effectiveness goal to measure progress in this area.

Evidence:

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: While VFC is similar to the 317 program, VFC serves a distinct population and focuses primarily on vaccine purchase. The 317 program does some vaccine purchase but also provides a lot of support for activities that cover the entire population including education, outreach, and surveillance.

Evidence:

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: While the more comprehensive IOM report indicated that childhood immunization levels are at an all-time high and the program has helped contribute to this outcome, this report focused more on the appropriate role of the Federal government rather than evaluating whether the 317 program, as currently structured/managed was effective at improving immunization rates among children.

Evidence: