“When Will We Ever Learn?” Mexico and Britain Take the Question Seriously

This is a joint post with Christina DroggitisThis May will mark the five-year anniversary of CGD’s Evaluation Gap Working Group’s final report, "When Will We Ever Learn: Improving Lives Through Impact Evaluation". The report noted a large gap in evidence about whether development programs actually work and recommended creating an independent international collaboration to promote more and better impact evaluations to close this gap. The International Initiative for Impact Evaluation (3ie) was formed as a result of this recommendation. The report also stressed the need for countries, both donors and recipients, to make larger commitments towards high-quality evaluation work. These commitments, it argued, should include supporting 3ie financially, as well as generating and applying knowledge from impact evaluations of their own development programs.So how are we doing? In general, the last five years have seen a dramatic expansion in impact evaluation work including more funding, more studies initiated and published, more efforts to make studies relevant to policy, and more efforts to get evidence from impact evaluations incorporated in public policy. Yet, two countries – Mexico and the United Kingdom – stand out as leaders in supporting the generation and use of impact evaluations, having worked hard in ways that fulfill both of these main recommendations of the Evaluation Gap report.Mexico was an active member of the Evaluation Gap Working Group with participation by representatives from the Ministries of Education and Health as well as hosting a regional consultation and signing on as a founding member of 3ie. Even before the Working Group was convened, Mexico had conducted impact evaluations with critical impact on public policy, notably in the case of PROGRESA-Oportunidades, and its congress passed legislation mandating evaluations of social programs in 2001. In response to this legislation, Mexico later created CONEVAL, an independent agency tasked with evaluating the country’s social development programs. CONEVAL will be co-hosting an international conference with 3ie and other agencies this June in Cuernavaca, with a plenary session marking the 5th anniversary of the Evaluation Gap report.Likewise, as a frequent and early supporter of 3ie, the UK’s DfID has committed significant funding to 3ie programs. It is the largest bilateral supporter, having committed more than US$21 million through 2013, and just barely displacing the Gates Foundation as the leading contributor to 3ie. Regarding the report’s recommendation for donors to improve their own impact evaluation work, the UK has strengthened its evaluation efforts and created an Independent Commission for AID Impact (ICAI), which plans to use evidence from independent evaluations to maximize impact and effectiveness of aid programs.This is not to say that there aren’t others out there making strides in impact evaluation. The U.S. Millennium Challenge Corporation (MCC) deserves credit for making rigorous impact evaluation a core feature of its programming (discussed recently by Sarah Jane Staats). MCC also played a role in creating 3ie and has made modest contributions as a supporting member. Other countries that support 3ie include Uganda, Pakistan, Australia, Denmark, Ireland, the Netherlands, Norway, and Sweden. The Gates Foundation and Hewlett Foundation are major financial contributors to 3ie and are joined in membership by NGOs like Saved the Children and multilateral institutions like the African Development Bank. USAID recently announced a new evaluation policy and commitment to join 3ie as well.Spain is not a member of 3ie, but it committed €11 million to support impact evaluation between July 2007-July 2010 through the Spanish Trust Fund for Impact Evaluation (SIEF), managed by the World Bank. Beyond supporting impact evaluation in developing countries, Spain has committed resources towards evaluating its own policymaking through the Spanish Agency for the Evaluation of Public Policies and Quality of Services (Agencia Estatal de Evaluación de Políticas Públicas y Calidad de los Servicios Públicos, or AEVAL). India has recently created an independent evaluation agency as well. The DAC Network on Development Evaluation is continuing to push for more and better impact evaluation among official development agencies.While independent actions by countries are key to closing the evaluation gap, progress is likely to be much faster if countries and agencies collectively commit to increase the number of impact evaluations and adhere to high standards of quality. First, countries should continue efforts to generate knowledge from their own evaluation of development programs. Secondly, they should consider joining and supporting 3ie, an organization committed to independence, credibility and high standards for evidence, that can also facilitate widespread demand for quality impact evaluations as well as assist in the supply side of administering these evaluations. Mexico and the UK deserve recognition for taking a lead in both these ways.