![[logo] US EPA](https://webarchive.library.unt.edu/eot2008/20090511134113im_/http://www.epa.gov/epafiles/images/logo_epaseal.gif){kind=logo}
Common Questions about Program Evaluation
- Why should you perform Program Evaluations?
- What types of Program Evaluations are there?
- What are some common concerns about performing Program Evaluation?
- What questions should you ask before beginning any Evaluation?
- How do you determine if the Program Evaluation was a success?
Why should you perform Program Evaluations?
- Help design a project or program.
- Ensure program goals & objectives are being met.
- Determine allocated resources are yielding the greatest environmental benefit.
- Identify what works well, what does not, and why.
- Identify program areas that need improvement.
- Inform management decisions regarding the program.
- Inform environmental guidance & develop future policy related to a specific project, program or issue.
- Examine and describe effective programs for duplication.
- Report on progress and communicate a program's value.
- Comply with Government Performance and Results Act.
Program Evaluation in GPRA (PDF Format 10KB)
[ return to Top ]
What types of Program Evaluations are there?
- Outcome
Outcome evaluations assess the reasons for differences between outcomes and objectives (e.g., why the number and quality of permits issued exceeded or fell short of objectives). Outcome evaluations may include an examination of program processes and activities to understand how outcomes are achieved and how quality and productivity could be improved.
- Process
A process evaluation assesses the extent to which a program or process is operating as intended and identifies opportunities for streamlining or otherwise improving it. Process evaluations often begin with an analysis of how a program currently operates. Process evaluations may also assess the extent to which program activities conform to statutory and regulatory requirements, Agency policies, program design or customer expectations.
- Formative
A formative evaluation is conducted early in the planning stages or early in implementation. It helps to define the scope of a program or project and to identify appropriate goals and objectives. Formative evaluations can also be used to pre-test ideas and strategies.
- Impact
Impact evaluations assess program effectiveness in terms of end results (e.g., Is the water swimmable? Why or why not?), including intended and unintended results. They may also assess the net effect of a program by comparing impacts with an estimate of what would have happened in the absence of the program.
- Cost-effectiveness
Cost-effectiveness analyses identify program benefits, outputs or outcomes and compare them with the internal and external costs of producing them.
[ return to Top ]
What questions should you ask before beginning any Evaluation?
- What is the goal/purpose of the evaluation?
- Is it required?
- Is it the right time to conduct an evaluation?
- What data and information are needed?
- Who should design the evaluation?
- Who should carry out the evaluation?
- Who is the audience for the evaluation?
- What are their expectations?
- What type of evaluation should be conducted?
- How will the evaluation be used?
When creating a program, program evaluation can help answer...
- What problem is the program attempting to address?
- What are the anticipated impacts of the program?
- What are a program's goals and objectives?
- What would be considered a success?
- Can success be measured?
- What arrangements have been made for data collection?
- Who are the people involved in a program/project?
- Are there people interested in a project that may not be involved?
- Have their concerns been addressed?
- How long will the program last?
- What is the budget for the program?
Once a program is underway, program evaluation can help answer...
- Does a program have clearly stated goals and objectives?
- Is a program making progress toward these goals and objectives?
- Are any activities or program approaches impeding progress toward achievement of goals?
- Are activities being conducted according to a proposed time line?
- Do improvements or adjustments need to be made in the program?
Closing out a program, program evaluation can help answer...
- Did the project meet overall goal(s)?
- What components were the most effective?
- What components were less effective?
- What lessons learned have the potential to be replicated or transferred to other programs?
- Were the results worth the cost?
[ return to Top ]
What are some common concerns about performing Program Evaluation?
~ A program evaluation
diverts resources
A program evaluation should pay for itself because it helps to determine
what does and does not work in a program.
~ A program evaluation
may produce negative results
An evaluation may reveal problems in accomplishing the work of a program
as well as successes, but both types of information are significant. The
potential discovery of problems should be viewed as an opportunity to
learn and improve the program.
~ A program evaluation
will increase the burden for program staff
Often, program staff are responsible for collecting evaluation information
because they are most familiar with, and have the most contact with the
program. Despite a potential for increased burden, staff can benefit greatly
from evaluation because it provides information that can help them learn
more about their program.
~ A program evaluation
is too complicated
The evaluation process systematizes what is often already done on an informal
basis -- figuring out whether a program's objectives are being met, which
aspects of the program are working and which program aspects are not as
effective.
~ Program evaluations are
too broad in scope
A program evaluation does not need to cover all of what EPA considers
a "program." It is possible to look at parts of a program without evaluating
the entire thing. Often, it is more instructive to limit the scope of
an evaluation and focus on specific pieces rather than an entire program.
~ Program evaluation is
simply another form of monitoring
There are indeed times when the information collected to monitor a program
overlaps with information needed for an evaluation, but the two processes
ultimately ask different questions.
[ return to Top ]
How do you determine if the Program Evaluation was a Success?
- Are the results timely and relevant?
- Are the results technically accurate?
- Are the results easy to follow?
- Are the results presented with clear findings and recommendations?
[ return to Top ]