MCC’s impact evaluations challenge us all to do better

Posted on December 6, 2012 by Howard W. Buffett, Executive Director, The Howard G. Buffett Foundation

The role of philanthropy is changing for the better, and with the October 24th release of its first five impact evaluations, the Millennium Challenge Corporation (MCC) is helping lead that change. For too long, funding for development – both private and public – has not been held accountable on measuring real impact. There are many reasons for this. It can be challenging, time-consuming and expensive to determine precisely which elements of complex aid packages are reducing poverty, and it can be even more difficult to justify investing in building evidence for what works when resources are scarce and the development need is so great. Any philanthropist constantly faces this tension, and we have found that it becomes tempting to look only at easily measurable activities like farmer training or direct measures of success like improving farmer yields – areas where we can directly observe how we are addressing poverty. The truth is, this approach both inhibits creative solutions and confines long-term decision-making. In the end, until we know what really works and why, we are destined to repeat the mistakes of the past and run the risk of missing enormous opportunities to build support for the real “game changers.”

I learned firsthand about MCC’s work when I served in government, and I often find myself gaining new insights from their experiences. Their recent release of the analysis of their impact evaluation is one of those moments. In fact, of the five impact evaluations just released, The Howard G. Buffett Foundation’s interests overlap with four of them: we also do smallholder farming training in El Salvador, Nicaragua, Honduras, and Ghana. Given the scale at which MCC works, we contacted them to learn as much as we could from their experience.

MCC has some important lessons for the development community – other U.S. agencies, donors, multinational corporations working in emerging markets, and philanthropies alike stand to learn a great deal. In the recently released “Impact Evaluations of Agriculture Projects,” part of MCC’s compelling Principles into Practice series, they have identified five key maxims that should be taken into account by all of us:

1. Define early the program logic and objectives of the evaluation, and how to integrate the two.
2. Engage early and communicate often.
3. Foster joint ownership by aligning incentives.
4. Match evaluation methodology and program design.
5. Focus on long-term impacts but be prepared to show early results.

Gaining these important insights has taken time, expertise and resiliency. As MCC has been bold to admit, in a refreshingly forthright and transparent fashion, it also takes occasional failures. As our foundation has learned, mistakes are unavoidable, but they only truly become mistakes when you fail to learn from them. Our foundation has spent the last 15 years and over $300 million across more than 70 countries, and we have made our fair share of errors along the way. Without rigorous evaluations to understand why goals are not always fully achieved, we can miss out on learning from those mistakes and miss the opportunity to share that learning with others.

MCC’s first set of independent impact evaluations raises some interesting questions about training farmers and how to measure the most important impact we all want to achieve: fostering prosperity and improving livelihoods. For example, in the three projects where MCC’s investments in training and improved inputs led to increased farmer incomes, these increases did not also result in increased income for the household as a whole. How can this be true?  MCC has several theories: small farmer households typically derive income from a number of sources and it may be that increased farm income reduces the pressure to produce income elsewhere; it may be a data collection problem where current measures of household income are not accurately capturing reality; or it may be some other reason. MCC does not yet have a definitive answer, but by identifying the question, they can better guide their analysis of existing data and improve the data collection for the pipeline of project evaluations that are in process to find the answer. This will in turn inform program design, which will help all of us investing in farmer training. MCC’s approach and knowledge base has already encouraged our foundation to make a larger investment in agriculture in El Salvador, and it has informed the design of a new initiative that we are beginning in Ghana.

Learning lessons in the absence of a strong evaluation mechanism is a painful, time-consuming and expensive way to learn, but it is not uncommon in development. By focusing on learning as much as we have focused on doing, we can all become smarter, more efficient and more focused on ideas that actually have more lasting impact in the long run. That’s why MCC’s approach is effective: they invest in accountability and transparency upfront so they can learn faster and improve their work more quickly and cost-effectively over the long-run. By sharing that knowledge with all of us, we all have the opportunity to learn. 

This transparency and honesty sets an example for our sector, and we all stand to benefit by being more efficient in our investments. MCC’s first five impact evaluations are a landmark step toward smarter, better and more accountable development around the world –and at a time when it could not be needed more.