MCC is committed to conducting independent impact evaluations of its programs as an integral part of its focus on results. These rigorous assessments of project impact often enhance the design of programs, provide critical information regarding the performance of specific activities, and contribute to a broader understanding of development effectiveness.
The Essence of the Impact Evaluation
An impact evaluation measures the changes in individual, household or community income and well-being that result from a particular project or program. The distinctive feature of an impact evaluation is the use of a counterfactual, which identifies what would have happened to the beneficiaries absent the program. This counterfactual is critical to understanding the improvements in people’s lives that are directly caused by the program.
Impact evaluation at MCC is part of a broader monitoring and evaluation (M&E) approach that covers all MCC-funded programs. Performance monitoring activities measure implementation progress and intermediate results, while impact evaluations are designed to measure the impact of projects on the wellbeing of beneficiaries.
Independent Assessments
MCC uses teams of independent professional researchers to carry out impact evaluations. These teams are selected in a competitive process that includes some of the world’s most experienced and respected impact evaluation specialists. MCC’s use of independent professionals is intended to ensure that the impact evaluations represent an unbiased assessment of the activities being studied.
How MCC Impact Evaluations Contribute to Development
MCC expects that the results of its impact evaluations will help guide future investment decisions. These impact evaluations further the work of the MCC and the development field, in general, in a number of ways:
- Impact evaluations help MCC decide whether to expand a particular type of program or avoid it in the future. Projects that have a large positive impact on beneficiaries can be expanded or become models for future MCC projects in similar settings.
- Impact evaluations help MCC economists enhance the precision of their future calculations of economic rates of return (ERRs).
- Impact evaluations contribute to the global academic discourse on what works in the development field. MCC will make the results of its impact evaluations publicly available to be used by other donors, researchers and non-governmental organizations.
Conducting an Impact Evaluation
There are several methods for conducting impact evaluations, with the use of random assignment to create treatment and control groups producing the most rigorous results. When random assignment is not feasible, MCC may also use other methods to construct a credible comparison group, such as double difference, regression discontinuity, propensity score matching, or other types of regression analysis. MCC also considers the feasibility and cost-effectiveness of having a separate control or comparison group to ensure that impact evaluation resources are used where they will provide the most useful information.
Technical feasibility is a prerequisite for all evaluations, but there are other important considerations to take into account when selecting projects to evaluate. In order of importance, the criteria are:
- Learning potential: The ability of MCC or others to apply the results of the impact evaluations to future funding decisions or project designs.
- What lessons will the impact evaluation provide for ongoing and future investments in similar interventions? Typical MCC impact evaluations are designed to test whether the program had an impact on beneficiary income or other measures of well-being. The learning potential of the impact evaluation might also involve testing different implementation approaches or combinations of interventions.
- What portion of MCC funds are committed to this sector or activity? Has the project been replicated in several MCA eligible or Threshold countries? Is it likely to be part of future programs?
- What is the timeframe for getting the results of the impact evaluation? Will there be actionable evidence of effectiveness within the Compact term?
Need for Evidence: When the quantity and quality of evidence justifying a project is low, MCC or other donors may nonetheless decide to fund the activity based on plausible and positive anecdotal evidence, but will include as part of the project design a rigorous impact evaluation to test assumptions about its effectiveness. Impact evaluation methods are especially useful when testing a pilot program.
- Is there already evidence that a particular program or activity is effective? Is the evidence based on rigorous impact evaluations that address selection bias issues?
- If so, do results have external validity across different contexts and populations?
- Is there an expectation that a successful program would be expanded within the country with either MCC or other funding?
Feasibility: The practical considerations of implementing a particular impact evaluation design. The impact evaluation method used should ensure reliable results and have broad country and institutional support, and there should be a strong probability that the data will be applied to future design and funding decisions.
- Can the impact evaluation be conducted at a reasonable cost? Although many obstacles to implementing successful impact evaluations can be overcome when there is a strong demand for evidence, in some cases, valid concerns about cost and feasibility can limit the ability to proceed. For example, the rehabilitation of a national highway may have such a broad group of beneficiaries that it is not feasible to establish a conventional control group.
- Will the impact evaluation be high quality in terms of method and power? In some cases, even the best-designed studies may provide evidence that is so weak that it will not be useful for guiding future decisions. In such cases, the cost of conducting an impact evaluation may not be justified, even if the question is very important.
Country Program | Evaluation | Focus of Evaluation | Methodology |
---|---|---|---|
Armenia Compact | Impact Evaluation for Armenia’s Rural Road Rehabilitation | What is the effect of rural road rehabilitation on the quality and accessibility of roads, agricultural productivity and profits, and household well-being? |
Regression discontinuity |
Benin Compact | Impact Evaluation for Benin’s Access to Land Project | What is the impact of land tenure on investment and income? |
Randomized control trial |
Benin Compact | Impact Evaluation for Benin’s Financial Innovation and Expansion Challenge Facility | What is the impact of the intervention on transaction costs and loan portfolio quality, and deposit and lending activity of microfinance institutions? What is the impact on the operating costs and profits of grantees? Does the intervention improve the financial performance of clients? (cash flow, gain better access to credit) |
Matching |
El Salvador Compact | Impact Evaluation for El Salvador’s Water and Sanitation Sub-Activity | What is the impact of water and sanitation on the cost of water, water consumption, illness, time use, and household income? |
Matching |
Armenia Compact | Impact Evaluation for Armenia’s Water to Market Activity | What is the impact of on-farm water management training on farming practices, agricultural productivity, and the income of rural farming households? |
Randomized control trial |
Burkina Faso Threshold Program | Impact Evaluation for Burkina Faso’s Threshold Program | What was the impact of the program on school enrollment? What was the impact of the program on test scores? Were the impacts different for girls than for boys? |
Regression discontinuity |
El Salvador Compact | Impact Evaluation for El Salvador’s Connectivity Project | What is the impact of road improvements on travel cost and time, land prices, and household income? |
Regression discontinuity |
El Salvador Compact | Impact Evaluation for El Salvador’s Education and Training Activity | What is the impact of improved technical middle schools on completion rates, employment, and income? |
Matching |
El Salvador Compact | Impact Evaluation for El Salvador’s Rural Electrification Sub-Activity | What is the impact of electrification on the cost of energy, energy consumption, time allocation, and household income? |
Matching |
Georgia Compact | Impact Evaluation for Georgia’s Agribusiness Development Activity (ADA) | How does the provision of ADA grants to farmers and farm-related businesses impact household income, poverty levels, and job creation? |
Randomized control trial |
Georgia Compact | Impact Evaluation for Georgia’s Samtskhe-Javakheti Road Rehabilitation Activity | How does the road rehabilitation effect/cause economic development, new businesses, and economic and social integration in the region? |
Matching & GIS analysis |
Ghana Compact | Impact Evaluation for Agricultural Training in Ghana | Does the FBO training program cause farmers to adopt new technologies or techniques, such as using land more intensively and efficiently, choosing crops that are more competitive, or optimizing the use of inputs, including labor? What is the magnitude of any spillover from the trained farmers on proximate farmers and those in the trained farmers’ social networks? Does the FBO training program cause farmers to increase their yields, sales, incomes, and enhance their access to social services? |
Randomized control trial |
Ghana Compact | Impact Evaluation for Ghana’s Feeder Roads Activity | Do improved roads lead to higher farm income through reduced input cost and higher producer price at the farm gate that are associated with reduced travel time and vehicle operating cost (transport cost)? |
Difference-in-difference |
Honduras Compact | Impact Evaluation for Honduras’ Rural Development Project | What is the impact of increased productivity and business skills on incomes? |
Randomized control trial |
Honduras Compact | Impact Evaluation for Honduras’ Transportation Project | Do decreased transport costs lead to increased incomes? |
Matching & GIS analysis |
Mozambique Compact | Impact Evaluation for Mozambique’s Rural Water Supply Project | Does increased access to improved rural water points improve health outcomes with respect to diarrhea in particular and reduce time to collect water? Do improved and appropriate technology of rural water points increase household income and raise productivity? |
Randomized control trial |
Niger Threshold Program | Impact Evaluation for Niger’s Threshold Program | Do the program interventions, independently or in combination, result in increases in girls’ primary education enrollment, attendance and completion rates? |
Randomized control trial |
Tanzania Compact | Impact Evaluation for Tanzania Electricity Distribution Systems Rehabilitation & Extension Activity | Does access to electricity lead to a) increased household income and better health and education outcomes and b) increased business activity, including new firms, capital investments and greater levels of investment? If impact is detected, what is the magnitude of this impact and what is the magnitude in comparison to the costs? |
Randomized control trial, regression discontinuity |
Tanzania Compact | Impact Evaluation for Tanzania’s Mainland Trunk Roads & Zanzibar Rural Roads | Does a reduction in transport costs and travel times lead to increased access to markets and economic activity in towns/villages near a road? |
Difference-in-difference with matching |
Nicaragua Compact | Impact Evaluation for Nicaragua’s Rural Business Development Project | What is the impact of agriculture technical assistance on the well-being (consumption) of beneficiaries? |
Randomized control trial |
Armenia Compact | Impact Evaluation for Irrigation Infrastructure Improvements in Armenia | What is the impact of canal improvements on the quality and reliability of irrigation water, agricultural productivity and household income? |
Matching, difference-in-difference |
El Salvador Compact | Impact Evaluation for Business Development in El Salvador | What impact did the offer of productive development services have on beneficiaries’ |
Randomized control trial |
El Salvador Compact | Impact Evaluation for Scholarships in El Salvador | What is the impact of secondary school scholarships on recipients’ education and labor market outcomes? |
Randomized control trial |
Nicaragua Compact | Impact Evaluation for Road Improvements in Nicaragua | Does a reduction in transport costs and travel times lead to increased movement of goods and people on roads and increased access to goods and decreased costs of consumption, services and inputs? Was the project cost effective, as analyzed through re-estimated economic rates of return, comparisons to original estimates, and assessment of impact? |
Difference-in-difference with matching |
Ghana Compact | Impact Evaluation for Community Services Water Activity in Ghana | How has the investment in water systems improved health conditions for the beneficiary communities? Are the improvements in health conditions attributable to the improved water systems, and why? Have the improvements in health resulted in increased availability of labor for economic activities? |
Difference-in-difference with matching |
Impact Evaluation Data Sets
Impact Evaluation for Armenia’s Water to Market Activity
In Armenia, the MCC program’s goal is to reduce rural poverty through better economic performance in the agricultural sector by improving irrigation infrastructure and training farmers. To evaluate the impact of the farmer training in on-farm water management and high value agriculture, the Farming Practices Survey was developed. The baseline was completed by Jen Consult/AREG NGO in January 2008 with a sample of approximately 5,000 farming households in 350 communities across Armenia. The first follow-up survey was completed in 2009 and the final survey will be completed in 2011.
- Armenia Water to Market Activity Evaluation Data (CSV format)
file (3 MB) - Armenia Water to Market Activity Evaluation Data (STATA format)
file (3 MB) - Armenia Water to Market Activity Evaluation Data (XML format)
file (2 MB)
Impact Evaluation for Burkina Faso’s Threshold Program
In 2007-08, Mathematica Policy Research and the University of Ouagadougou conducted household and school surveys as part of MCC’s evaluation of the Burkina Faso Threshold Program, which sought to increase educational attainment through the construction of schools and complementary interventions. The surveys included 291 villages, of which 132 were participant villages and 159 comparison villages. The household questionnaire asked about household demographics, children’s educational outcomes (enrollment and attendance), and parents’ perceptions of education. The school survey asked about schools’ characteristics and children’s attendance and enrollment.
- Burkina Faso Threshold Program Evaluation Data (CSV format)
file (4 MB) - Burkina Faso Threshold Program Evaluation Data (STATA format)
file (5 MB) - Burkina Faso Threshold Program Evaluation Data (XML format)
file (3 MB)
Impact Evaluation for Tanzania’s Mainland Trunk Roads & Zanzibar Rural Roads
MCC aims to reduce travel times and provide access to basic social services in Tanzania by rehabilitating a portfolio of trunk roads on the mainland and selected rural roads on Pemba Island. The evaluation will examine the project’s household- and community-level effects on the well‐being of the people and villages along the roads. Economic Development Initiatives conducted a survey of 3,000 households in 200 communities in 2009. Baseline data is available now, and future data will be posted as it becomes available.
- Tanzania Roads Evaluation Data (CSV format)
file (8 MB) - Tanzania Roads Evaluation Data (STATA format)
file (9 MB) - Tanzania Roads Evaluation Data (XML format)
file (9 MB)