Detailed Information on the
AmeriCorps Volunteers in Service to America Assessment

VISTA has awarded a contract for an in-depth program evaluation designed to assess effectiveness of the VISTA program in helping organizations to develop and sustain anti poverty programs. The evaluation will assess factors that lead to project success and sustainability and will develop a predictive model that VISTA can use to improve project effectiveness. Final results are expected in March 2009. As of June 16, The VISTA Program Evaluation is under way with instruments currently being piloted with a group of four test projects. Once the instruments are final, a 30 day notice will be posted and the final clearance package sent to OMB (approximately late July.) As of January 2009 the assessment is ongoing with analysis under way of paper records that are readily available. Surveys of former VISTA sponsors and selected site visits will begin upon approval of Paperwork Reduction Act clearance. Final completion date is pending the start of these activities.

Continuing to upgrade and enhance electronic data systems that will enable the program to accurately establish baselines and measure program performance.

Measure: Number of Community Volunteers recruited by AmeriCorps*VISTA members.

Explanation:This measure tracks the number of community members recruited and placed as volunteers by AmeriCorps*VISTA members to support programs that serve low-income communities. This measure relates directly to AmeriCorps*VISTA's stated program purpose of encouraging volunteer service.

Measure: Amount of cash and inkind resources generated by AmeriCorps*VISTA members.

Explanation:This measure tracks the level of private sector resources generated by AmeriCorps*VISTA members to support programs in low-income communities. This measure relates directly to AmeriCorps*VISTA's stated program purpose of generating the commitment of private sector resources.

Measure: Percentage of organizations reporting that the services of AmeriCorps*VISTA members enabled them to increase the number of persons served by their program.

Explanation:This measure tracks program performance towards helping sponsoring organizations increase capacity to serve low-income communities. This measure relates directly to AmeriCorps*VISTA's stated program purpose of strengthening local organizations that serve low income communities.

Measure: Percentage of member service years supported by cost share agreements with sponsoring organizations.

Explanation:This measure tracks the degree to which the program can offset the federal cost per participant through cost share agreements with sponsoring organizations.

Measure: Average number of community volunteers recruited per member service year.

Explanation:This is an intermediate outcome that tracks progress towards achieving the long term outcome goals of mobilizing community volunteers to support programs that serve low income communities.

Measure: Average amount of cash and inkind private sector resources generated per member service year.

Explanation:This is an intermediate outcome that tracks progress toward achieving the long term outcome goals of generating the commitment of cash and inkind private sector resources to support programs serving low income communities.

Is the program purpose clear?

Explanation: Created in 1964 under the Economic Opportunity Act, the Volunteers in Service to America (VISTA) program provides full-time volunteers to nonprofit community and faith-based organizations to build sustainable anti-poverty programs. The program was reauthorized in 1981 and charged with: (1) supporting low-income communities by strengthening the capacity of local agencies and organizations that serve those communities; (2) encouraging volunteer service; and (3) generating the commitment of private sector resources. In 1993, VISTA became part of the new AmeriCorps program managed by the Corporation for National and Community Service (CNCS) and is now known as the AmeriCorps VISTA program.

Evidence: VISTA In Service to America: Fighting Poverty for 40 Years (May, 2006) National and Community Service Trust Act of 1993; September 21, 1993 (Public Law 103-82)

Does the program address a specific and existing problem, interest, or need?

Explanation: 40 years after its creation, VISTA's anti-poverty mission continues to fill a critical need in American communities. While the demographics and nature of poverty have changed over the years, the need for help in addressing conditions that cause poverty remains current. VISTA serves two specific needs: (1) the need to engage more Americans in volunteer service in the United States; (2) the need to strengthen the ability of faith based and community organizations to fulfill their missions of serving low income communities.

Evidence: Building Capacity in Nonprofit Organizations; Urban Institute (2001). This study cited capacity building as an existing need, specifically describing a lack of capacity and technical expertise among community based groups to keep up with rapid change, increasingly complex social and federal networks, and a growing demand for community based services.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Although the AmeriCorps' VISTA design is unique among the other two AmeriCorps programs (AmeriCorps State and National Grants; AmeriCorps National Civilian Community Corps) the goals of encouraging volunteerism and serving low income communities are duplicative of other volunteer efforts and federal programs. For example, the Points of Light Foundation (POLF) works solely to engage volunteers and resources more effectively in service to help solve serious social problems. The POLF has partnerships with more than 79,000 organizations and helps connect more than 2.5 million people with over a half million opportunities to serve. Within the federal government, the Compassion Capital Fund was created to enhance the efforts of faith-based and community organizations serving low-income individuals and families. The FY 2007 Budget requests $100 million for the fund to provide grassroots organizations with training and technical assistance, as well as targeted capacity building to meet the needs of the poor by improving their delivery of social services.

Evidence: U.S. Department of Labor, Bureau of Labor Statistics. Current Population Survey: Volunteering in the United States (2005). http://www.bls.gov/cps/ U.S. Department of Health and Human Services, Administration for Children and Families: Compassion Capital Fund. www.acf.hhs.gov

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There are no known major flaws in the current system. For over 40 years the AmeriCorps VISTA program has fulfilled its mission to support anti-poverty programs by placing full-time volunteers in local communities throughout the country. AmeriCorps VISTA members have helped individuals and communities implement grassroots solutions to alleviate poverty. Since its inception, more than 177,000 individuals have served as VISTA volunteers. VISTA members have helped establish health clinics, literacy programs and computer training programs. VISTA knows of no evidence that a different approach or mechanism would be more efficient or effective at this time.

Evidence: VISTA In Service to America: Fighting Poverty for 40 Years, May 2006. FY 2007 Congressional Justification Domestic Volunteer Service Act of 1993

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: AmeriCorps VISTA resources are targeted and managed through state level CNCS program staff. The selection process includes a pre-application screening process from which only those organizations that most closely serve the program purpose are selected to submit a full application. Once VISTA resources are awarded, monitoring protocols are enforced to ensure that the resources are used appropriately by the intended beneficiaries. Projects that do not use resources appropriately or effectively are closed down prior to the anticipated project end date so that the resources can be reallocated to other organizations. VISTA's program model does not lend itself to unintended subsidies.

Evidence: FY 2007 Congressional Budget Justification

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: AmeriCorps VISTA has established three long-term performance measures that link to the program's purpose of helping low-income communities address poverty related problems through three objectives: (1) encouraging volunteer service; (2) generating the commitment of private sector resources; and (3) strengthening non profit and public agencies that serve those communities.

Evidence: AmeriCorps VISTA measures are included in: (1) FY2007 Congressional Budget Justification and (2) 2005 Performance and Accountability Report to Congress.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: AmeriCorps VISTA has established reasonable targets for its long term outcome measures. The program has set targets through 2010 that reflect a 20% increase over five years in the number of community volunteers recruited and a 15% increase over five years in private sector resources generated by VISTA members. These increases reflect the program's commitment towards strengthening the ability of individual members to carry out the program purpose.

Evidence: In FY 2005, AmeriCorps*VISTA members recruited 510,000 community volunteers and generated $157M in private sector resources to support local programs. By 2010, AmeriCorps*VISTA projects that VISTA members will mobilize 650,000 community volunteers and generate $200M in private sector resources.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: AmeriCorps VISTA has developed two annual measures that assess the program's progress in reaching its long term goals. As intermediate outcomes, AmeriCorps VISTA measures the average number of community volunteers recruited and the amount of private sector resources generated per volunteer. This allows the program to monitor progress towards the long term goals, and to adjust targets more effectively should the projected number of volunteers change.

Evidence: Annual measures have been included in the program's FY 2007 Congressional Budget Justification, 2005 Performance and Accountability Report to Congress.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: VISTA will continue to work on setting ambitious yet attainable annual targets. In FY 2006, VISTA set an annual target of 96 volunteers recruited per member service year, a slight increase above 2005. VISTA is committed towards setting ambitious annual targets that will get the program to its long-term goal of reaching 650,000 volunteers by 2010. The program does not have reliable trend or historical data, so progress against the former targets cannot be assessed. Further, baseline data for the annual measures was established in 2004.

Evidence: Annual measures have been included in the program's FY 2007 Congressional Budget Justification, 2005 Performance and Accountability Report to Congress.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: In fiscal year 2005, AmeriCorps VISTA strengthened its requirements for project performance measurement. Currently all AmeriCorps VISTA sponsoring organizations are required to develop annual outcome measures that show progress toward long term goals. All programs, regardless of locally determined project outcomes must report on progress toward such national goals as the amount of private sector resources generated through the efforts of VISTA members, and the number of community volunteers recruited.

Evidence: FY 2006 AmeriCorps VISTA Program Guidance, December 2005

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: While the AmeriCorps VISTA program is subject to Inspector General evaluations, none have been conducted to date. Recent and ongoing evaluations of national service activities conducted by such independent groups as the Rockefeller Institute, the Urban Institute, and Abt Associates have focused mainly on the AmeriCorps State and National program. The internal VISTA evaluations are not sufficiently rigourous to assess the program's effectiveness.

Evidence: VISTA in Service to Amerca: 40 years of fighting poverty in America, May 2006; The Volunteer Management Capacity Study (2003); The Assessment of Faith Based and Community Organizations (2003- 2006); and the Customer Satisfaction Survey (2004 - 2006).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: AmeriCorps VISTA's budget requests are developed through a logic model that ties financial and human resource inputs to program activities, annual and long term performance targets and goals. However, the calculations used to determine cost per participant are not easily transparent in the VISTA budget request. While VISTA notes a cost per-participant of $17,103 in their Congressional Justification, they do not display $6,000 in costs associated with the education award and overhead needs used to fully fund a VISTA volunteer. The VISTA calculations are inconsistent with data the Corporation has released to Congress on the full costs associated with supporting a VISTA volunteer.

Evidence: FY 2007 Congressional Budget Justification

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: During the spring of 2004 AmeriCorps VISTA incorporated a new strategic planning and budget framework based on a "logic model" methodology. In conjunction with all agency programs AmeriCorps VISTA defined its strategic goals in terms of quantifiable output and outcome measures. Performance measures and targets are linked to each section of the Corporation's strategic plan. Strategic plans and logic models are reviewed and revised annually, and often more frequently, to help ensure that activities are reflective of the current resource levels, and are aligned within the agency's overall goals. The program monitors performance data against the established targets and uses the information to continually adjust the plan and make sure that resources are targeted effectively.

Evidence: Examples of the AmeriCorps VISTA strategic planning process can be found in the CNCS 2007 Congressional Budget Justification, the AmeriCorps VISTA program logic models, and the Corporation's Strategic Plan.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AmeriCorps VISTA collects data from sponsoring organizations through semi-annual progress reports. These progress reports include national performance data as well as project-specific performance measures. CNCS state program staff review these reports within 30 days of submission and use the information to regularly monitor program performance. Training plans are developed based on this information and interventions conducted when performance is lacking. On a national level, AmeriCorps VISTA regularly collects data to assess overall progress towards program goals. This information is used to determine how well sponsoring organizations are meeting their own performance objectives as well as national performance goals. AmeriCorps VISTA also collects performance data through external surveys and uses the information to make programmatic adjustments to enhance performance.

Evidence: VISTA Program Evaluations and Surveys, Program Year 2004 and 2005.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: AmeriCorps VISTA managers and employees, are held accountable for operations and results of the AmeriCorps VISTA program through the Corporation's newly strengthened appraisal system. Performance plans include criteria that measure the performance of each manager and employee on the completion and quality of essential duties and responsibilities, including oversight of program partners and achievement of key program milestones on schedule and within budget. Project sponsors sign a Memorandum of Agreement and are held accountable for complying with its provisions, and meeting the objectives specified in their project plan within an established timeline. Private sector contractors who deliver service on behalf of the AmeriCorps VISTA program are held accountable to providing services on schedule and in a cost effective manner.

Evidence: In FY 2005, due to demonstrated performance and cost effectiveness, AmeriCorps VISTA extended an agreement with the International Rescue Committee, an intermediary project sponsor. IRC had demonstrated strong programmatic performance, and were able to cost share 100% of their VISTA resources. AmeriCorps VISTA has started termination proceedings for another large intermediary project, the Latino Pastoral Action Council, due to lack of demonstrated results, non-compliance with performance requirements and poor fiscal management. In FY2005 an independent audit was conducted on one of AmeriCorps VISTA's largest private sector contractors, North West Regional Educational Laboratories (NWREL), that conducts formal pre-service training for VISTA members and sponsors. The audit indicated that NWREL was performing at an excellent level, and operating within cost and schedule commitments. This information was an important factor in the decision to select NWREL when the time came to re-bid this contract.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are obligated throughout the fiscal year according to the program's annual operating budget. Formal budget reviews are conducted three times a year for each of five geographic clusters and for AmeriCorps VISTA headquarters. These reviews include progress reviews against established objectives to ensure that funds are spent for the planned purposes. Obligation and commitment reports are reviewed monthly to ensure obligations are made in a timely manner. Budgeted Member Service Years (MSYs), allocated through the cluster budgets to individual state offices, are monitored through Management Information System (MIS) charts that track actual and forecasted member placements throughout the year. This data is updated monthly, reconciled against an electronic data base, and monitored by headquarters to ensure that funding to support VISTA members is being used on schedule and as projected or is re-allocated in the case of projected underproduction.

Evidence: Management Information System Charts, 2005 Program Year FY 2007 Questions for the Record (Chairman Ralph Regula); AmeriCorps VISTA data on unexpended balances (FY 2001-2005)

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The VISTA program has developed an efficiency measure, "Percentage of member service years supported by cost share agreements with sponsoring organizations." Increasing the number of cost share agreements has historically been a challenge for VISTA. In FY 2004, VISTA took active steps to increase the level of cost share agreements with its grantees to help them sustain anti-poverty programs and allow VISTA to place more members across the country. This effort resulted in an increase in the percentage of member service year supported by cost share agreeement from 11% to 12%, the first such increase in several years. VISTA has set targets for 2006 at 13% and 2007 at 15%. VISTA has also taken meaningful steps to restructure participant training programs to reduce travel and lodging costs.

Evidence: AmeriCorps VISTA is currently undertaking a study through the Rockefeller Institute to examine cost-sharing practices among current and former sponsoring organizations. The information from this study will be used to enhance the program strategy for increasing cost share partnerships over time.

Does the program collaborate and coordinate effectively with related programs?

Explanation: VISTA has collaborated with related federal programs in the past. For example, in FY 2005, AmeriCorps VISTA received $200,000 in resources from the Department of Justice to directly support an additional 16 VISTA members plus a $300,000 investment in training and technical assistance to support the Weed and Seed program. The Community Capacity Development Office, which houses the Weed & Seed program, reported that this effort resulted in a leveraging of resources and development of local capacity that they could not have achieved without the VISTA partnership. Other agencies with which AmeriCorps VISTA has collaborated include the Department of Labor, Housing and Urban Development and Health and Human Services.

Evidence: Weed and Seed Interagency Agreement, Department of Justice (2005)

Does the program use strong financial management practices?

Explanation: No material weaknesses or internal control weaknesses related to the AmeriCorps VISTA program have been reported by auditors. VISTA has procedures in place to ensure that payments are made properly and for the intended purpose. Each of five geographic clusters, and VISTA Headquarters, develop an annual budget that is tied to the program goals and to the strategic plan. Budgets are based on projected programming and are monitored through monthly obligations reports and three formal budget reviews per year. Budgets are adjusted as needed, based on actual spending and programming, to ensure that funding supports program operations as intended by the integrated budget and performance plan.

Evidence: FY 2005 Performance and Accountability Report; 2006 Operating Plan

Has the program taken meaningful steps to address its management deficiencies?

Explanation: AmeriCorps VISTA has decentralized its operations over the years to better support local programming. However, this structure presents challenges in effectively monitoring and reporting on program performance. To overcome these challenges, VISTA developed controls to ensure increased consistency in management and oversight between state offices, and enhanced systems and structures for collecting and managing information on program performance. VISTA also restructured staff resources to include an Information Management Unit to strengthen, and better integrate, data management, performance measurement and budget systems.

Evidence: Examples of steps the program has taken to address management deficiencies include: (1) the development of a rigorous system for assessing risk as part of a strengthened compliance monitoring system to be used by all state program staff. This system includes a single set of risk criteria against which all local VISTA projects are assessed; (2) a uniform process for the development of national demonstration projects to ensure proper oversight at the local level; (3) transitioning to an electronic project management system (a component of the agency's eGrants system) that will allow performance data to be directly managed online, increasing efficiencies in information management and data integrity.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: AmeriCorps VISTA has demonstrated some progress towards its long term performance goals. The program does not have reliable trend or historical data, making it difficult to fully assess the programs overall progress. VISTA has taken steps to improve the reliablity and accuracy of its data collection methods. In 2005 the program met all of its program targets. The number of community volunteers recruited by AmeriCorps*VISTA members increased by 5% from FY 2004 to FY 2005 and the amount of private sector resources generated has increased over 40% since 2003.

Evidence: Project Progress Report data base, Performance Benchmarking Initiative

Does the program (including program partners) achieve its annual performance goals?

Explanation: AmeriCorps VISTA has made some progress towards its annual performance goals. In FY 2005, the program met its goal of supporting 12% of total member service years through cost share agreements. The program also met its targets for increasing the number of community volunteers recruited per volunteer and the total amount of private sector resources generated per volunteer. In FY 2006, VISTA will continue to work on setting ambitious targets and improving the data collection methods for its measures.

Evidence: 2007 Congressional Budget Justification, 2006 Performance and Accountability Report to Congress

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The VISTA program has made some improvements to operate the program more efficiently. Specifically, VISTA developed an efficiency measure, "Percentage of member service years supported by cost share agreements with sponsoring organizations." As noted in Question 3.4, VISTA increased the percentage of cost share agreeements from 11% to 12% in FY 2005, the first such increase in several years. VISTA also made operational improvements to restructure how the program delivers training and technical assistance. The program is being given "small extent" based on these improvements. Additional work is needed to improve the transparency of the cost per participant.

Evidence: 2007 Congressional Budget Justification, 2006 Performance and Accountability Report to Congress

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Created three years after the Peace Corps, VISTA has long been considered its domestic counterpart. Although the two programs serve different populations, and have different specific objectives, they share a common purpose of recruiting full time volunteers to help meet local community needs. VISTA compares favorably to the Peace Corps in the diversity of the volunteers recruited into the program. In FY 2005, nearly 40% of AmeriCorps VISTA members were from minority ethnic groups, while Peace Corps reports that 16% of Peace Corps trainees were minorities in that same year.

Evidence: Peace Corps 2005 Performance and Accountability Report to Congress and 2007 Congressional Budget Justification

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The VISTA program has never been evaluated in terms of its outcomes: i.e., the effect of the VISTA program in alleviating poverty and the sustainability of VISTA projects. While VISTA has been the subject of internal evaluations and annual customer satisfaction surveys, these activities do not gauge program effectiveness. Also, there are no evaluations comparing the VISTA program with other federal anti-poverty and volunteer programs. Further, an OIG audit or evaluation assessing the program's effectiveness has never been conducted.

Evidence: The American Customer Satisfaction Index, the national measure for customer satisfaction, rated the AmeriCorps*VISTA program overall with a score of 76, five points higher than the average score for federal agencies. A study conducted by the University of Michigan Business School, the American Society for Quality and the CFI Group, assesses the program's performance in supporting VISTA sponsoring organizations and providing them with the services and information they need to achieve their program goals.