Independent Evaluations

Overview

“

The release of the Millennium Challenge Corporation’s (MCC) first impact evaluations is a real step forward in demonstrating transparency in U.S. development efforts.  Efforts like this are transforming the way development is done with a clear emphasis on the principles of results, accountability, and impact.

”

Liz Schrayer

USGLC

October 25, 2012

“

The MCC’s willingness to share information on both the successes and challenges of its programs and to apply the lessons learned is a model for the aid community.

”

Sam Worthington

InterAction

“

From its inception, MCC was designed with a commitment to transparency and evidence-based decision making.  The release of these results puts that promise into action.

”

Jim Kolbe

MFAN Chair

“

I tip my hat to the MCC for not only putting the evaluations out there, but giving us a bunch of useful lessons in the process.

”

Markus Goldstein

World Bank

October 23, 2012

“

We hope these evaluations encourage Congress to continue to invest in improving the lives of people in developing countries.

”

Rev. David Beckmann

MFAN and Bread for the World

October 25, 2012

“

This path-breaking effort is putting to the test long-held assumptions about how to promote development and is just the type of learning we need if we are to use our assistance resources effectively.

”

George Ingram

MFAN

October 25, 2012

“

The evidence base created will not only inform other development organizations, but also the investments of our member companies in agriculture value chains.

”

Jennifer Potter

Initiative for Global Development

October 25, 2012

“

The five impact evaluations are for just 2% of the overall MCC budget, so they’re just a start but they offer an initial glimpse into both the effectiveness of these programs and real efforts by the agency to take measuring results seriously.

”

John Glenn

USGLC

October 25, 2012

MCC is releasing its first set of independent impact evaluations for farmer training activities in five countries. Impact evaluations are an important part of MCC’s evidence-based approach that uses data and evidence to guide decision making for selection of country partners, investment decisions, and measuring results. They help MCC measure increases in income, the ultimate impact on MCC’s continuum of results.  MCC uses impact evaluation to learn and adapt our practice based on that learning.

  • Rigorous: An impact evaluation is defined by the ability to estimate the counterfactual (what would have happened to the same group of individuals if they had not received MCC’s assistance).
  • Independent: MCC uses teams of independent professional researchers to carry out its evaluations.
  • Rare: The use of impact evaluations in global development remains fairly rare.  In farmer training, for example, an expert study identified only three impact evaluations using experimental designs in farmer training anywhere in the world over the past decade.

While there are many tools to measure results and foster learning, there is no more rigorous tool than an impact evaluation to:

  • Test attribution: Impact evaluations, through use of a counterfactual, makes it possible to know whether observed impacts were caused by an MCC investment or by external factors that affected everyone.
  • Test assumptions about what works: Impact evaluations can be used to test traditional assumptions about how planned interventions are expected to lead to poverty reduction.
  • Build evidence: Impact evaluation findings inform future program design, so program planners can rely less on assumptions and more on evidence about what works.

The First Five Evaluations

Return to top

At a Glance

The First Five Impact Evaluations
Country Activity Evaluated
Armenia Compact

Improved Profitability of WUA Member
Ghana Compact

Commercial Training
Honduras Compact

Farmer Training and Development
Nicaragua Compact

Rural Business Development Services and Technical and Financial Assistance
El Salvador Compact

Production and Business Services
  • The independent impact evaluations are of farmer training activities in Armenia, El Salvador, Ghana, Honduras, and Nicaragua.
  • The farmer training activities represent 13 percent of these five compact, and 2 percent of MCC’s global portfolio.
  • According to MCA monitoring data, we were very successful in meeting or exceeding our targets for these activities. The average completion rate of output and outcome targets specific to the activities covered by these evaluations is: Ghana (103 percent), Armenia (103 percent), Nicaragua (112 percent), El Salvador (131 percent), and Honduras (158 percent). 
  • Impact evaluations allow MCC to take a step further—to test how outcome achievements translate into farm income and ultimately household income.
  • Three evaluations detect increases in farm income: El Salvador dairy (farm incomes doubled), Ghana northern region (crop incomes increased), and Nicaragua (farm incomes increased 15-30 percent).
  • Increases in household income are not yet detected. This is raising interesting questions about how to achieve and measure changes in household income.
  • MCC is using impact evaluation findings to test traditional assumptions about what works, learn lessons, adapt business practice, and improve effectiveness.

Download the M & E Policy

What People are Saying

Fact Sheet

What People are Saying Fact Sheet

Read what people are saying about MCC's first five impact evaluations.

Impact and Performance

Return to top

Impact Evaluations

Impact evaluations are the most rigorous form of evaluations because they make it possible to know whether the observed impacts were caused specifically by an MCC investment or, alternatively, are the result of external factors that affected program participants and non-participants, like increased market prices for agricultural goods, national policy changes or favorable weather conditions. Impact evaluations compare what happened with the MCC investment to what would have happened without it, through the use of a counterfactual.

Performance Evaluations

Performance evaluations are a valuable tool for estimating the contribution of MCC investments to changes in trends on outcomes, including farm and household income. Performance evaluations are less rigorous and cannot attribute causal impact to MCC investments because they do not use a counterfactual. However, they are useful to compare changes in the situation before and after MCC’s investment and provide details on how an investment might have contributed to changes in beneficiary income, how it might have contributed to changes in outcomes and why.

How We Choose

There are several critical factors that MCC considers when deciding to invest in an impact or a performance evaluation:

  • Learning potential: For programs where the assumptions underlying the project logic are based on limited evidence, there is a strong case for an impact evaluation. A rigorous impact evaluation tests assumptions about a project’s effectiveness and contributes substantially to MCC’s future decision-making, as well as the global evidence base.
  • Feasibility: The feasibility of designing and implementing a strong impact evaluation is based on how well the evaluators are able to estimate a counterfactual and how feasible it is to maintain that counterfactual through the duration of the evaluation period.
  • Strong stakeholder commitment: Identifying a control group and ensuring adherence to an impact evaluation design require significant commitment and collaboration by sector staff, program implementers and evaluators, both within MCC and among partner countries.
  • Appropriate timing: The evaluation timeline must be informed by the project logic, particularly with regard to assumptions about how long it will take for expected impacts to occur. By collecting data too early, evaluations may underestimate the impacts on outcomes of interest or miss important lessons.
  • Proper coordination: Evaluations require close coordination with program implementation. Program designers, implementers and evaluators must work together to understand and define the program logic, estimate how long expected impacts are likely to take to accrue and identify what is most important to learn about how the program works. This is particularly true for impact evaluations, which require coordination and commitment among various stakeholders to estimate a counterfactual.

Incorporating evaluations—particularly impact evaluations—into program operations is not easy, but this is a challenge that MCC embraces to ensure accountability for results and to improve learning on what works. This commitment to evaluation helps distinguish MCC in the international development community.

Download the fact sheet

The Counterfactual

An impact evaluation is defined by the ability to estimate the counterfactual, what would have happened to the same group of program participants if they had not received MCC’s assistance. The most rigorous method for estimating the counterfactual to measure attributable program impacts is through randomized control trials. In many programs, there are financial and/or logistical constraints to providing all eligible individuals or groups with an intervention. Random selection (such as through a lottery) is a fair and transparent way to select which eligible individuals or groups should receive the intervention first.

Because randomized control trials randomly select individuals that will and will not be exposed to program benefits, evaluators can compare the groups to measure their impacts. This use of a statistically identical control group creates the greatest opportunity for learning what works and for measuring program impacts.

When a randomized control trial is not feasible, MCC may use other methods to construct a credible comparison group, such as a propensity score matching, difference in differences or regression discontinuity.

List of Evaluations

Return to top

This table lists planned and completed independent evaluations.
Independent Evaluations
Country Program Activity Evaluated Focus of Evaluation Type Status
Ghana Compact Rural Development Project

How has the investment in water systems improved health conditions for the beneficiary communities? Are the improvements in health conditions attributable to the improved water systems, and why? Have the improvements in health resulted in increased availability of labor for economic activities?

 Impact Planned
Nicaragua Compact Transportation Project

Does a reduction in transport costs and travel times lead to increased movement of goods and people on roads and increased access to goods and decreased costs of consumption, services and inputs? Was the project cost effective, as analyzed through re-estimated economic rates of return, comparisons to original estimates, and assessment of impact?

 Impact Completed
El Salvador Compact Human Development Project

What is the impact of secondary school scholarships on recipients’ education and labor market outcomes?

 Impact Planned
El Salvador Compact Production and Business Services

What impact did the offer of productive development services have on beneficiaries’
incomes and employment?

 Impact Completed
Armenia Compact Irrigated Agriculture Project

What is the impact of canal improvements on the quality and reliability of irrigation water, agricultural productivity and household income?

 Impact Planned
Nicaragua Compact Rural Business Development Services and Technical and Financial Assistance

What is the impact of agriculture technical assistance on the well-being (consumption) of beneficiaries?

 Impact Completed
Tanzania Compact Transport Sector Project

Does a reduction in transport costs and travel times lead to increased access to markets and economic activity in towns/villages near a road?

 Impact Planned
Tanzania Compact Energy Sector Project

Does access to electricity lead to a) increased household income and better health and education outcomes and b) increased business activity, including new firms, capital investments and greater levels of investment?  If impact is detected, what is the magnitude of this impact and what is the magnitude in comparison to the costs?

 Impact Planned
Niger Threshold Program Girl’s Primary Education

Do the program interventions, independently or in combination, result in increases in girls’ primary education enrollment, attendance and completion rates?

 Impact Planned
Mozambique Compact Water and Sanitation Project

Does increased access to improved rural water points improve health outcomes with respect to diarrhea in particular and reduce time to collect water?  Do improved and appropriate technology of rural water points increase household income and raise productivity?

 Impact Planned
Honduras Compact Transportation Project

Do decreased transport costs lead to increased incomes?

 Impact Completed
Honduras Compact Farmer Training and Development

What is the impact of increased productivity and business skills on incomes?

 Impact Completed
Ghana Compact Agriculture Project

Do improved roads lead to higher farm income through reduced input cost and higher producer price at the farm gate that are associated with reduced travel time and vehicle operating cost (transport cost)?

 Impact Planned
Ghana Compact Commercial Training

Does the FBO training program cause farmers to adopt new technologies or techniques, such as using land more intensively and efficiently, choosing crops that are more competitive, or optimizing the use of inputs, including labor? What is the magnitude of any spillover from the trained farmers on proximate farmers and those in the trained farmers’ social networks? Does the FBO training program cause farmers to increase their yields, sales, incomes, and enhance their access to social services?

 Impact Completed
Georgia Compact Regional Infrastructure Rehabilitation Project

How does the road rehabilitation effect/cause economic development, new businesses, and economic and social integration in the region?

 Impact Planned
Georgia Compact Enterprise Development Project

How does the provision of ADA grants to farmers and farm-related businesses impact household income, poverty levels, and job creation?

 Performance Planned
El Salvador Compact Human Development Project

What is the impact of electrification on the cost of energy, energy consumption, time allocation, and household income?

 Impact Completed
El Salvador Compact Human Development Project

What is the impact of improved technical middle schools on completion rates, employment, and income?

 Impact Planned
El Salvador Compact Connectivity Project

What is the impact of road improvements on travel cost and time, land prices, and household income?

 Impact Planned
Burkina Faso Threshold Program Girls’ Education Promotion Project

What was the impact of the program on school enrollment? What was the impact of the program on test scores? Were the impacts different for girls than for boys?

 Impact Completed
Armenia Compact Improved Profitability of WUA Member

What is the impact of on-farm water management training on farming practices, agricultural productivity, and the income of rural farming households?

 Impact Completed
El Salvador Compact Human Development Project

What is the impact of water and sanitation on the cost of water, water consumption, illness, time use, and household income?

 Impact Planned
Benin Compact Access to Financial Services Project

What is the impact of the intervention on transaction costs and loan portfolio quality, and deposit and lending activity of microfinance institutions?  What is the impact on the operating costs and profits of grantees?  Does the intervention improve the financial performance of clients?  (cash flow, gain better access to credit)

 Impact Planned
Benin Compact Access to Land Project

What is the impact of land tenure on investment and income?

 Impact Planned
Armenia Compact Rural Road Rehabilitation Project

What is the effect of rural road rehabilitation on the quality and accessibility of roads, agricultural productivity and profits, and household well-being?

 Impact Planned