MCC is committed to conducting independent impact evaluations of its programs as an integral part of its focus on results. These rigorous assessments of project impact often enhance the design of programs, provide critical information regarding the performance of specific activities, and contribute to a broader understanding of development effectiveness.
The Essence of the Impact Evaluation
An impact evaluation measures the changes in individual, household or community income and well-being that result from a particular project or program. The distinctive feature of an impact evaluation is the use of a counterfactual, which identifies what would have happened to the beneficiaries absent the program. This counterfactual is critical to understanding the improvements in people’s lives that are directly caused by the program.
Impact evaluation at MCC is part of a broader monitoring and evaluation (M&E) approach that covers all MCC-funded programs. Performance monitoring activities measure implementation progress and intermediate results, while impact evaluations are designed to measure the impact of projects on the wellbeing of beneficiaries.
Independent Assessments
MCC uses teams of independent professional researchers to carry out impact evaluations. These teams are selected in a competitive process that includes some of the world’s most experienced and respected impact evaluation specialists. MCC’s use of independent professionals is intended to ensure that the impact evaluations represent an unbiased assessment of the activities being studied.
How MCC Impact Evaluations Contribute to Development
MCC expects that the results of its impact evaluations will help guide future investment decisions. These impact evaluations further the work of the MCC and the development field, in general, in a number of ways:
- Impact evaluations help MCC decide whether to expand a particular type of program or avoid it in the future. Projects that have a large positive impact on beneficiaries can be expanded or become models for future MCC projects in similar settings.
- Impact evaluations help MCC economists enhance the precision of their future calculations of economic rates of return (ERRs).
- Impact evaluations contribute to the global academic discourse on what works in the development field. MCC will make the results of its impact evaluations publicly available to be used by other donors, researchers and non-governmental organizations.
Conducting an Impact Evaluation
There are several methods for conducting impact evaluations, with the use of random assignment to create treatment and control groups producing the most rigorous results. When random assignment is not feasible, MCC may also use other methods to construct a credible comparison group, such as double difference, regression discontinuity, propensity score matching, or other types of regression analysis. MCC also considers the feasibility and cost-effectiveness of having a separate control or comparison group to ensure that impact evaluation resources are used where they will provide the most useful information.
Technical feasibility is a prerequisite for all evaluations, but there are other important considerations to take into account when selecting projects to evaluate. In order of importance, the criteria are:
- Learning potential: The ability of MCC or others to apply the results of the impact evaluations to future funding decisions or project designs.
- What lessons will the impact evaluation provide for ongoing and future investments in similar interventions? Typical MCC impact evaluations are designed to test whether the program had an impact on beneficiary income or other measures of well-being. The learning potential of the impact evaluation might also involve testing different implementation approaches or combinations of interventions.
- What portion of MCC funds are committed to this sector or activity? Has the project been replicated in several MCA eligible or Threshold countries? Is it likely to be part of future programs?
- What is the timeframe for getting the results of the impact evaluation? Will there be actionable evidence of effectiveness within the Compact term?
- Need for Evidence: When the quantity and quality of evidence justifying a project is low, MCC or other donors may nonetheless decide to fund the activity based on plausible and positive anecdotal evidence, but will include as part of the project design a rigorous impact evaluation to test assumptions about its effectiveness. Impact evaluation methods are especially useful when testing a pilot program.
- Is there already evidence that a particular program or activity is effective? Is the evidence based on rigorous impact evaluations that address selection bias issues?
- If so, do results have external validity across different contexts and populations?
- Is there an expectation that a successful program would be expanded within the country with either MCC or other funding?
- Feasibility: The practical considerations of implementing a particular impact evaluation design. The impact evaluation method used should ensure reliable results and have broad country and institutional support, and there should be a strong probability that the data will be applied to future design and funding decisions.
- Can the impact evaluation be conducted at a reasonable cost? Although many obstacles to implementing successful impact evaluations can be overcome when there is a strong demand for evidence, in some cases, valid concerns about cost and feasibility can limit the ability to proceed. For example, the rehabilitation of a national highway may have such a broad group of beneficiaries that it is not feasible to establish a conventional control group.
- Will the impact evaluation be high quality in terms of method and power? In some cases, even the best-designed studies may provide evidence that is so weak that it will not be useful for guiding future decisions. In such cases, the cost of conducting an impact evaluation may not be justified, even if the question is very important.
Sectors
MCC Impact Evaluations in Progress
Sort the table by clicking on the column headers. Arrows indicate ascending and descending order.
| Country | Project/Activity | Focus of Evaluation | Methodology |
|---|---|---|---|
 Armenia |
Water-to-Market Activity | What is the impact of on-farm water management training on farming practices, agricultural productivity, and the income of rural farming households? | Randomized control trial |
 Benin |
Access to Land Project | What is the impact of land tenure on investment and income? | Randomized control trial |
 Burkina Faso |
Threshold Program | What is the impact of school construction and other complementary interventions on primary school enrollment and completion rates? The impact evaluation will also assess food consumption and resource allocation within households. | Regression discontinuity |
 El Salvador |
Connectivity Project | Impact of road improvements on travel cost and time, land prices, and household income | Regression Discontinuity |
El Salvador |
Human Development Project | What is the impact of improved technical middle schools on completion rates, employment, and income? | Matching |
El Salvador |
Water and Sanitation Sub-Activity | Impact of water and sanitation on the cost of water, water consumption, illness, time use, and household income | TBD |
 Georgia |
Agribusiness Development Activity (ADA) | How does the provision of ADA grants to farmers and farm-related businesses impact household income, poverty levels, and job creation? | Randomized control trial |
Georgia |
Samtskhe-Javakheti Road Rehabilitation Activity | How does the road rehabilitation effect/cause economic development, new businesses, and economic and social integration in the region? | Propensity Score Match and GIS analysis |
 Ghana |
Development of Agricultural Productivity and Value-Added Project | Does the Farmer-Based Organization (FBO) training program encourage farmers to adopt new technologies or techniques? What is the magnitude of spillover from MiDA-trained farmers on other farmers nearby and on those in the MiDA-trained farmers’ social networks? Does the FBO training program cause farmers to get higher yields and sales? | Randomized control trial |
 Honduras |
Rural Development Project | What is the impact of increased productivity and business skills on incomes? | Randomized control trial |
 Nicaragua |
Training & Property Rights | Does property regularization alone unlock growth, or does the impact of property regularization depend on access to business services? | Randomized control trial |
 Niger |
Threshold Program | Do the program interventions, independently or in combination, result in increases in girls’ primary education enrollment, attendance and completion rates? | TBD |
 Tanzania |
Energy Sector | What impact does access to electricity have on the wellbeing of the beneficiaries, including education outcomes, household income, business income, and contribution to income and resource allocation within households? | TBD |