The following questions were asked during the 2005 National Conference on Volunteering and Service in August 2005, the State Commission Executive Director’s Meeting in September 2005, and the six Regional Rulemaking Training Sessions that occurred during the months of October and November 2005.

Background

During rulemaking, defining new requirements for evaluation was one of the most difficult elements to work through. Our intent in defining the evaluation requirement in the rule is to offer opportunities for standardization while affording you the most flexibility in deciding what you want to evaluate, and how you want to evaluate it. We don’t want to be prescriptive, but want to allow each of you to determine your own evaluation design.

We recognize that understanding how to measure and particularly how to attribute impact in a complex and rapidly changing environment is extremely challenging, and will most certainly be an evolving process. We look forward to continuing to learn in this area in partnership with our grantees and other partners in national and volunteer service across the nation.

Requirements

1. What are the evaluation requirements for AmeriCorps grantees?

As articulated in the AmeriCorps regulations Sections 2522.700-2522.740, all AmeriCorps State and National grantees that receive an average annual Corporation grant of $500,000 or more must conduct an independent evaluation. An independent evaluation uses an external evaluator who has no formal or personal relationship with, or stake in the administration, management, or finances of the grantee or of the program to be evaluated.

All other AmeriCorps State and National grantees must conduct an internal evaluation. An internal evaluation is designed and conducted by qualified program staff or other stakeholders, such as board members, partners, or volunteer affiliates.

All evaluations must cover at least one year of Corporation-funded service activity. The $500,000 threshold is calculated by averaging your Corporation grant over the last three years you have received Corporation funding, at the time you recompete.

Please note that the $500,000 threshold represents Corporation funding, not total budget with match. Only 21% of State grantees receive an average annual grant of $500,000 or more and are required to provide an independent evaluation. For these grantees, participation in national or state-wide evaluation studies also satisfies the requirement. See Questions 22 and 23, below for more detail on Commission- or Corporation-sponsored statewide and national evaluations. The regulations can be found at www.gpoaccess.gov/ecfr and http://www.americorps.gov/about/ac/rulemaking.asp

In summary:

2. When do the requirements regarding evaluation go into effect?

The requirements for evaluation will take effect in the 2007 grant competition.

3. What are the requirements for the internal and independent evaluation plans for an applicant recompeting in 2007, and thereafter?

If you are recompeting in 2007 or thereafter, you are required to submit “a summary of your evaluation efforts or plan to date, and a copy of any evaluation that has been completed, as part of your application for funding” (45 CFR § 2522.730). If you recompete again in 2010 or beyond, you are required to submit a completed evaluation with your application. The Corporation will consider the results of your evaluation “in assessing the quality and outcomes of your program” (45 CFR § 2522.470). If you are submitting a continuation application in 2007, you do not have to submit an evaluation summary or plan until you recompete.

4. What does the Corporation expect of a grantee that conducts an independent evaluation?

If you receive an average of $500,000 or more per year from the Corporation, averaged over the last three years of funding you have received before you recompete, we expect you to conduct an independent evaluation by contracting with an external evaluator. The AmeriCorps regulations describe how this evaluation should provide evidence of a causal relationship between program activities and outcomes (45 CFR § 2522.700). You may consider using an experimental or quasi-experimental design, or compare your results with national/state/local data. Your external evaluation method should match the size, scale, and purpose of your program.

5. What does the Corporation expect of a grantee that conducts an internal evaluation?

In our ongoing effort to reduce burden on grantees, especially those with smaller grants, grantees an average annual grant under $500,000 may submit an internal evaluation.  The primary difference between the independent evaluations that grantees that receive $500,000 or over are required to submit and the internal evaluation is who conducts the evaluation study.  Your own staff and other stakeholders can serve as internal evaluators. 

We encourage you to design your internal evaluation so that it will yield data most useful to you.  You may opt for an impact evaluation, or you may conduct a process or management evaluation.  You are not required to conduct an experimental or quasi-experimental evaluation that proves causality, which is required of grantees that receive $500,000 or over, although you are allowed to conduct this type of study.  We expect the same high quality that we expect of a larger grantee, regardless of the type of evaluation you decide to conduct.

Resources

6. What training and technical assistance (TTA) resources are available through the Corporation to help grantees develop their evaluation plans?

The Corporation will provide TTA to help program managers work with their evaluators to plan and manage their program evaluations. Project STAR, the Corporation’s performance measurement and evaluation TTA provider has prepared a sample evaluation plan that can be found on the Project STAR web site, listed here. Upon request, Project STAR will hold workshops at state/regional/national levels and develop training modules to help program managers understand:

• the Corporation’s evaluation requirements;
• the multiple uses of evaluation for improving program management, impact and sustainability;
• elements of an evaluation plan;
• how to select and work with external/internal evaluator(s);
• how to involve stakeholders in the evaluation;
• basic evaluation designs;
• how to develop evaluation questions; and
• how to summarize, analyze and use evaluation findings.

The Corporation’s web site includes a number of useful evaluation tools and resources http://www.nationalserviceresources.org/. The American Evaluation Association, Harvard Family Research Project, the United Way, the U.S. Department of Education, and the W.K. Kellogg Foundation also include useful information on evaluation on their web sites (see list at the end of these FAQs).

7. Are AmeriCorps grantees required to comply with federal-wide requirements that focus on information collection? For example, do grantees need to submit evaluation surveys to the Office of Management and Budget (OMB)?

No. These requirements apply only to Federal Executive Departments and agencies, branches of the military and other establishments of the Executive Branch of the federal government.

8. Are AmeriCorps grantees required to comply with federal-wide requirements that focus on human subjects research, as articulated by the U.S. Department of Human Services (www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm)?

No. The Corporation is not included as an agency covered by this regulation. However, all AmeriCorps grantees should ensure that they are in compliance with rules and regulations for Institutional Review Board approval of research as implemented by your own agency or educational institution.

9. How much should I budget for evaluation?

A range for the cost of evaluation you may want to consider is 5% to 10% of the total budget of the program (W.K. Kellogg Evaluation Handbook, pg. 54). The complexity of the evaluation plan and expertise and experience of the evaluator will determine the cost.

10. Where can I locate an independent evaluator and what should be the selection criteria for choosing one?

Universities can be good sources for evaluators, as can referrals from your peers. The state commission in your state may be able to provide a list of college and university contacts that have evaluation expertise. National conferences can also be excellent sources for evaluators. Please see additional resources listed at the end of this FAQ, including the American Evaluation Association’s “Find an Evaluator” pages which list research firms/evaluators available by state.

Review and Use of Evaluations by the Corporation

11. When will my internal and independent evaluation plan, evaluation summary, or evaluation report be reviewed?

Internal and independent evaluation plans, summaries, and reports will be reviewed as part of the grant application review process.

12. What will the Corporation do with my evaluation?

Your evaluation is primarily a tool for you to use to strengthen your program and your impact. It provides you data for continual improvement, adjustment, and action. The Corporation will assess your evaluation as part of our assessment of the quality and outcomes of your program. Evaluation results will also be mined nationally to identify and share promising practices with the field.

Other Issues

13. What is the relationship between my performance measures and my independent or internal evaluation?

Performance measurement is the process of systematically and regularly collecting and monitoring data related to the direction of observed changes in communities, participants (members), or end beneficiaries receiving your program’s services. Evaluation is a more in-depth, rigorous effort to measure the impact of programs. While performance measurement and evaluation both include systematic data collection and measurement of progress, evaluation uses scientifically-based research methods to assess the effectiveness of programs by comparing the observed program outcomes with what would have happened in the absence of the program. You are strongly encouraged to include the data you collect for performance measurement in your evaluation study. You will find more information on how evaluation differs from performance measurement in the AmeriCorps regulations Section 2522.700.

14. Should my evaluation focus on the entire spectrum of expected outcomes or a particular slice of the expected outcomes?

This is up to you. The focus of your evaluation depends on what you decide to be the most relevant indicators to measure. We encourage you to link your evaluation design to your performance measures and primary service activities, since you are already collecting and analyzing these data.

15. Is an evaluation required to focus on primary service category (impact in the community) or can it focus on member development?

You may evaluate any aspect of your program that you choose.

16. As a multi-site grantee, am I expected to perform a multi-site evaluation and compare findings to national data?

This depends on your evaluation design. If you are a multi-site grantee, it would be logical for you to evaluate at least a representative sample of your operating sites, if not all of the sites. The most important factor is that the sites you choose to evaluate are appropriate within the context of your evaluation design and methodology, and representative of all of your sites, as opposed to selecting any particular number or type of sites.

17. As a national grantee that serves as an umbrella organization for many different kinds of service activities (i.e. we support mentoring, health, public safety, and environmental programs) what should I evaluate?

You and your internal or independent evaluator need to determine what to evaluate. You may want to consider focusing your evaluation on what you are already measuring with your performance measures. If you have multiple and very different performance measures, you may want to evaluate elements common to all your programs, such as volunteer leveraging or capacity-building results.

18. How should capacity building be evaluated?

You and your internal or independent evaluator would need to determine the set of evaluation procedures and indicators that will best capture the organizational capacities that your program is developing.

19. If my organization receives AmeriCorps funding from more than one source (i.e. state formula, competitive, and national), for more than one project, am I required to participate in two, or even three evaluations?

Yes, unless the organization has included the project in a national or a statewide evaluation. If your program receives more than one kind of type of AmeriCorps funds, you will work with your state commission and your Program Officer to determine the most effective way to frame your evaluation efforts.

For State Commissions

20. As a commission, must I consult with my Corporation Program Officer in determining our evaluation policy?

As a state commission, we expect you to determine your own evaluation policy for your formula portfolio. You are encouraged to confer with your Program Officer when developing your evaluation policy.

21. When a commission submits a former formula program to the Corporation for competitive review, will its application be considered a new competitive application or will it be considered a recompeting application that requires an evaluation plan, summary, or evaluation report?

A formula program that re-applies and is submitted as a competitive application will be considered a recompeting application, if it satisfies the Corporation’s definition of “same project,” below. If your project satisfies the definition, you will be required to submit an evaluation plan, summary, or evaluation report when you recompete. If your project does not satisfy the definition, it will be considered new and will not be required to submit an evaluation plan, summary, or completed evaluation.

Two projects will be considered the same if they:

• address the same issue areas;
• address the same priorities;
• address the same objectives;
• serve the same target communities and population;
• utilize the same sites; AND
• use the same program staff and members.

22. Can a commission conduct a statewide evaluation rather than requiring individual programs to conduct evaluations?

A statewide independent evaluation is permitted, with prior approval by the Corporation, providing it covers all AmeriCorps programs in a state. Results must be disaggregated by program to ensure consistency across the entire national portfolio.

The Corporation will start approving commission plans for statewide evaluations as part of your Administrative application in 2008. If you are a state commission and plan to initiate a statewide evaluation prior to 2008, please contact your Program Officer, who will outline what you need to submit for one-time approval.

23. When will AmeriCorps provide more information regarding “one or more strategies …including, potentially, national Corporation-administered evaluations” cited in the preamble to the Rule?

From time to time, the Corporation’s Office of Research and Policy Development (RPD) undertakes national evaluations. For example, in 2005 RPD launched a national study of Youth Corps programs that includes random assignment. In the past, RPD also conducted a longitudinal survey, and has produced state profiles. The Corporation will keep you apprised of such national evaluation efforts as they emerge. We encourage you to participate if invited to do so. Due to resource constraints most Corporation evaluation studies will focus on certain types of programs. For example, the Corporation may beinterested in conducting a study of mentoring programs, and only a nationally representative or random sample ofprograms conducting these activities may be asked to participate in the study.

If a program participates in a national evaluation conducted by the Corporation, that program will be exempt from the requirement to provide an internal or independent evaluation during the next two three year grant cycles. Results must be disaggregated by program to ensure consistency across the entire national portfolio.

Additional Resources

Inclusion in this list does not connote endorsement of the resource by the Corporation.
The following are presented as a sample of available resources.

Corporation for National and Community Service (CNCS) Resource Center
http://www.nationalserviceresources.org/
Search for keyword “evaluation” and pull up a variety of evaluation resources, including, information on evaluation elements and uses, forms with criteria for evaluating AmeriCorps members and Senior Corps volunteers, and the Project STAR pages. The Resource Center lends CNCS grantees hard copies and ship items free of charge.

Project STAR Support and Training for Assessing Results
http://www.nationalservice.gov/resources
Project STAR is the performance measurement and evaluation training and technical assistance provider for CNCS grantees. Find Project STAR web pages by doing a key word search, or via the “Training Providers” portal on the left panel. Project STAR’s evaluation resources for CNCS grantees include tips for preparing an evaluation plan, a sample evaluation plan, and guidance on selecting an independent evaluator.

American Evaluation Association
http://www.eval.org
This professional association of evaluators is devoted to improving evaluation practices at all levels. The “Resources” section of their website identifies consultants (potential evaluators), foundations funding applied research, data analysis software, scientific research methods, and links to government and NGO evaluation divisions/units. Refer the Topical Interest Groups (TIGs) for activity specific evaluation contacts.

The Bruner Foundation
http://www.brunerfoundation.org/ei/
This group provides resources on logic models, effectiveness, and evaluative thinking.

CDC Evaluation Working Group
http://www.cdc.gov/eval/
This web site has a public health focus, but the evaluation guidance can be applied more generally to other service program evaluations. The web site highlights a framework for program evaluation describing evaluation standards, steps, and a comprehensive list of online resources.

Resources for Methods in Evaluation and Social Research.
http://gsociology.icaap.org/methods
These resources focus on how-to do evaluation research and the methods used: surveys, focus groups, sampling, interviews, qualitative research,h and other methods.

Statistics Help
http://nilesonline.com/stats/
Whether you are a beginner or a pro, this site will help you to understand and use statistics, such as mean, median, percent, per capita and rates, standard deviation and normal distribution, margin of error and confidence interval, data analysis, and sample sizes.

The Evaluation Exchange
http://www.gse.harvard.edu/hfrp/eval.html
Published by the Harvard Family Research Project, this periodical highlights innovative methods and approaches to evaluation, emerging trends in evaluation practice, and practical evaluation applications relating to childcare, family, school and community. The Evaluation Exchange goes out to its subscribers free of charge four times per year.

The Evaluation Cookbook
http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.html
The Evaluation Cookbook gives guidance on using a wide range of evaluation methods, including, preparing questionnaires, conducting focus groups and interviews, doing pre and post tests, and experiments.

United Way of America’s Outcome Measurement Resource Network
http://national.unitedway.org/outcomes
The Resource Center Network offers information, down-loadable documents, and links to resources that help to measure program and community level outcomes.

W.K. Kellogg Foundation’s Evaluation Toolkit
http://www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&LanguageID=0
This toolkit provides a wealth of information on evaluation approaches, preparing evaluation plans and evaluation methods. The web site also hosts a number of excellent articles and presentations. Upon request, the Kellogg Foundation will mail additional resources free of charge.

Performance Measure Amendments and Corrective Action Plans
for Formula, State Competitive Grantees, and National Grantees

1. When does the provision in the rule regarding Corrective Action Plans take effect?

The provision in the rule regarding Corrective Action Plans is already in effect.

2. When do we decide that we need to submit a Corrective Action Plan?

Assess the need to submit a Corrective Action Plan at the same time you collect data for performance measurement. If at any point you collect data that indicates you are not going to meet your performance measurement targets, initiate a Corrective Action Plan by contacting your Program Officer, or if you are a formula grantee, your state commission.

Corrective Action Plans should be submitted within 30 days of noticing that you are not on track for meeting any of your performance measures. If you are a formula program, the state commission that approves the plan will forward a copy to the Corporation within 15 days of approving the plan. If you have questions about determining what constitutes being not on track for meeting your performance measures, please contact your Program Officer or if you are a formula grantee, your state commission.

3. What topics need to be addressed in a Corrective Action Plan?

According to the AmeriCorps Rule, your Corrective Action Plan needs to be in writing and should include all of the following:

• The factors impacting your performance goals.
• The strategy you are using and the corrective action you are taking to get back on track toward your established performance measures.
• The timeframe within which you plan to achieve getting back on track with your performance measures.

4. What is the role of the Program Officer in drafting a Corrective Action Plan?

You should draft your Corrective Action Plan and timeline in consultation with your Program Officer. The final plan should represent your agreed-upon strategy to get back on track in terms of meeting your performance measures.

5. After a Corrective Action Plan is put into place, how long does a program have to meet its requirements?

Your Corrective Action Plan will include a timeline that determines when you are expected to be back on track with your performance measures. Typically, programs are allowed 60-90 days to correct their course and start making progress again towards meeting performance measures.

6. What topics need to be addressed in a Request to Amend Performance Measures? When should these be submitted?

You may change your performance measures only if the Corporation, or, for formula programs, the state commission, approves your request to do so. Requests should address your plans to:

• Adjust your performance measure or target based on experience so that your program’s goals are more realistic and manageable.
• Replace a measure related to one issue area with one related to a different issue area that is more aligned with your program service activity.
• Redefine the service that individuals perform under the grant.
• Eliminate an activity because you have been unable to secure necessary matching funding.
• Replace one measure with another.

A request to amend your performance measures should include all of the following:

• Why you are not on track to meet your performance measures, if that is the case.
• How you have been tracking your performance measures.
• Evidence of the corrective action you have taken.
• Any new proposed performance measures or targets.
• Your plan to ensure that you meet any new measures.

Submit your request to amend your performance measures within 30 days of determining that you are not meeting your performance measures. If you are a formula program, the state commission that approves the request will forward a copy to the Corporation within 15 days of approving the request.

7. Is there any difference between the requirements and process for submitting Corrective Action Plans for formula and competitively funded programs?

There is no difference in the requirements and the process except formula programs submit their Corrective Action Plans and Requests for Amendment to their state commission rather than directly to the Corporation.

8. Is each method used to address unmet performance measures independent of each other or are they consecutive improvement strategies (e.g., is a Corrective Action Plan always required prior to submitting a Request to Amend Performance Measures?)

The methods can be either independent, or consecutive. The AmeriCorps Rule states that if you are not meeting your performance measures, you should develop and submit to the Corporation or your state commission a Corrective Action Plan, OR a Request to Amend Performance Measures. A Request to Amend Performance Measures may be submitted without a Corrective Action Plan, and vice versa.

For example, in some cases, a program might first develop and engage in corrective action, and then decide to make a request to change their performance measures. In other situations, a program might change its performance measures with no need for corrective action, but later may need to develop a Corrective Action Plan. And in a third situation, a program might develop and engage in a Corrective Action Plan that includes a request for new performance measures.