Closing the Evaluation Gap: 3ie One Year On (Are Conditional Cash Transfer Programs Improving Human Capital?)

This is a joint post with William Savedoff.Policymakers, researchers, and development experts gathered at CGD on May 4 to discuss the implications of existing research on conditional cash transfers (CCTs) and recommend ways for the development community to improve impact evaluations of interventions, like CCTs, in the future.  The workshop, entitled Closing the Evaluation Gap: 3ie One Year On (Are Conditional Cash Transfer Programs Improving Human Capital?), was jointly hosted by CGD and the International Initiative for Impact Evaluation (3ie), and included a series of presentations and high-level panel discussions focused on the effect of CCT programs on outcomes such as educational attainment, grade progression, birth weight, and access to family planning methods.In a recent blog post, Evaluation Gap Working Group co-chair and CGD senior fellow William Savedoff explained the motivation for the workshop, highlighting the heightened need to address the ways in which evidence and policymaking interact, as well as the importance of continued improvements in evaluation systems in determining the efficacy of social interventions.  We were thrilled with the positive responses from the workshop’s participants and audience members, and invigorated by the discussion and recommendations that arose as a result of the event.The importance of well-designed and delivered research on individual social programs was a central theme of the day-long seminar.  3ie deputy director Marie Gaarder called for researchers to evaluate the overall causal chain for CCTs and health outcomes, and introduced a special issue of the newly-launched Journal of Development Effectiveness that focuses on CCT programs.  Exploring ways in which researchers seek to “open the black box” by going beyond measurement of average effects, Gaarder noted the importance of analyzing heterogeneity (for example, how contraceptive use changed more among extremely poor women than among those who were better off) and distinguishing mechanisms (for example, assessing whether birth weight was higher for program participants because of higher income, better nutrition, or empowerment).The seminar outlined the different channels through which financial incentives can influence changes in behavior, and the capacity for these incentives (especially when conditioned) to have a profound effect on indicators such as the utilization of health and education services by the poor.  Participants also discussed the limitations of CCT programs, noting that emerging evidence suggests that these interventions may have little effect on overall health and education outcomes such as coverage of basic health interventions and school achievement among urban children.  Speakers and panelists emphasized the need for a deeper understanding of the relative cost-effectiveness of investing in the supply versus the demand-side within the health and education systems, as well as the potential negative implications of encouraging utilization of services without a corresponding effort to increase the quality of these services.Over lunch, CGD president Nancy Birdsall recognized the successes of 3ie’s first year of operation, and congratulated the organizations that have supported it. She highlighted the importance of 3ie’s role as a provider of public goods that should be financed by all major actors, going on to challenge participants from institutions like the World Bank and Inter-American Development Bank to ask why their organizations have not yet joined. She reminded those present that the initial idea of financing 3ie with a small levy, perhaps 0.05% of all disbursements by bilateral and multilateral agencies, has still not come to fruition.3ie executive director Howard White also discussed 3ie’s successes, but concentrated his remarks on what it takes to conduct quality impact evaluations. He also mentioned the capacity for mistakes to occur when researchers fail to consider how projects are being implemented or investigate anomalies that are heard in the field.In the event’s final session, a group of panelists—moderated by former president of the Global Development Network Lyn Squire —discussed how researchers can more effectively bring evidence-based results into the policy process.  Ruth Levine, director of evaluation, policy analysis and learning at USAID, emphasized the importance of identifying and creating long-term “durable” incentives for health and education-promoting behaviors if programs are to have lasting impact. Levine also recommended that researchers ascribe to an strategy dubbed “The 3Ps”—Predicting likely problems to inform research, Prescribing clear core messages, and drawing Pictures (such as GIS maps)—to provide compelling visual demonstrations of information.  Miguel Szekely, former undersecretary of education in Mexico, and Squire stressed that “bridging the gap” between research and policy will require radical thinking, underscoring the importance of devising incentives for policymakers themselves to demand good evidence for developing more effective programs.This final topic—bridging the gap between researchers and policymakers—is one of our central concerns as we move forward. To follow our progress and be part of the debate, please sign up to receive CGD’s Evaluation Gap newsletter.