Good Quality Evaluations for Good Policy: Findings and Recommendations from Aid Agency Evaluations in Global Health

Evaluations are key to learning and accountability yet their usefulness depends on the quality of their evidence and analysis. This brief summarizes the key findings of “Evaluating Evaluations,” a CGD Working Paper that assessed the quality of aid agency evaluations in global health. By looking at a representative sample of evaluations—both impact and performance evaluations—from major health funders, the study authors developed 10 recommendations to improve the quality of such evaluations and, consequently, increase their usefulness.

Good quality evaluations are needed for good policy

Overseas development assistance has always been subject to evaluation but, in recent years, an international consensus has emerged over the need for more and better evaluation. Agencies have increased funding for evaluations and implemented new evaluation policies. While the number of evaluations has increased, we know relatively little about their quality and use. A few public reports—focused primarily on US government agencies—find that less than half of evaluations meet high standards of quality. Reports at the World Bank and the UK’s Independent Commission on Aid Impact raise concerns that evaluation results are not widely utilized. These two issues—quality and usage—are likely to be related. A poor-quality evaluation is less convincing to policymakers, project managers, and constituents. To fill in some of these gaps, the CGD study is—to our knowledge—the first assessment of the methodological quality of global health program evaluations across different funders.

Around the world, tight and over-stretched budgets continue to put pressure on development assistance programs. Earlier this year, the US Administration proposed major cuts to foreign aid spending. Now more than ever, better evidence about what works in global development is needed to drive decisions on where to allocate resources to achieve results and value for money. While there has been tremendous progress over the past decade, it is critical to redouble efforts to ensure evaluations are not only of good quality but also relevant to policy.

What is a good evaluation? Relevance, validity, and reliability

To assess the quality of aid program evaluations, “Evaluating Evaluations” focused on three criteria, grounded in the methodological literature, that are essential to the quality of the evidence: relevance, validity, and reliability. The authors defined evaluations as relevant if the evaluation addressed questions related to the means or ends of an intervention, and used appropriate data to answer those questions. Evaluations were considered valid if analyses were methodologically sound and conclusions were derived logically and consistently from the findings. Evaluations were considered reliable if the method and analysis would be likely to yield similar conclusions if the evaluation were repeated in the same or similar context.

While acknowledging that funders conduct or commission evaluations with different aims, the study argues that these standards apply to all kinds of evaluations regardless of their questions and methods. For example, the sample included both “impact evaluations”—whose primary purpose is to measure and attribute impact to a particular intervention—and “performance evaluations”—that focus on questions related to such things as managerial efficiency, outputs, or beneficiary satisfaction.

The relevance, validity, and reliability of an evaluation affects whether its findings are helpful and should be heeded—whether the reader is an agency administrator trying to improve managerial efficiency; a parliamentarian trying to judge whether foreign aid is used appropriately; or a government official trying to design a new policy.

How do evaluations standup?

“Evaluating Evaluations” assessed a sample of 37 global health program evaluations. The sample was randomly drawn from a list of 299 global health program evaluations that were published between 2009 and 2014, and conducted or commissioned by five major funders of development assistance for health—United States Agency for International Development (USAID), the Global Fund to Fight AIDS, Tuberculosis, and Malaria (the Global Fund), the President’s Emergency Fund for AIDS Relief (PEPFAR), the United Kingdom Department for International Development (DFID), and the International Development Association (IDA) at the World Bank. Two reviewers independently assessed each evaluation in the sample and finalized responses after discussion and reconciliation.

The sample contained many high-quality evaluations, demonstrating that good evaluations are possible using a variety of methods and in different contexts. However, the share of high-quality evaluations was also relatively small, showing considerable opportunity for improvement (Figure 1).

Relevance of data and objectives

Most evaluations had relevant objectives. Three-quarters of the evaluations addressed questions that strongly reflected the main objectives of the program being evaluated. However, evaluations were much less likely to have relevant data. Only 16 percent of the evaluations collected information that was fully relevant to the questions they posed. About 40 percent were judged to at least “partially” use data that addressed their main research questions. The remainder of the evaluations lacked such data.

Validity and reliability of data collection

One-fifth of the evaluations met the highest standard for sampling validity and reliability, 11 percent scored in the middle range, and 68 percent had a low score. To reach the highest score, evaluations had to meet several criteria including clearly justifying their data collection methods and using unbiased data sources. Most of the evaluations with the lowest score used convenience sampling without justification or lacked information on data collection methods.

Validity and reliability of analysis

Finally, we assessed analytical validity and reliability primarily in terms of whether evaluations addressed potential confounding factors and adequately explained their methods of analysis. Sixteen percent of the evaluations had high analytical validity and reliability. Another 35 percent scored in the middle range and 49 percent received a low score.

Baseline information, data collection, and overall analysis

Better scoring evaluations tended to have access to baseline information, used data collection methods that paid attention to bias, and conducted analyses that explicitly drew on the information collected. For example, the lack of baseline information affected an evaluation of a large-scale fund to address infectious diseases in Myanmar, without which basic trends in disease prevalence and intermediate behavioral indicators could not be tracked or analyzed. Similarly, without baseline information, evaluations that focused on project implementation or management practices had to ask key informants to recall how processes had changed rather than compare their practices to information gathered during baseline interviews.

Data collection methods also affected the strength of findings. For example, a performance evaluation of a program to improve treatment of maternal and childhood illnesses in Madagascar used a multi-stage random sample of community health workers, which provided convincing information about implementation and effectiveness of training. By contrast, a performance evaluation of an orthopaedic training program in Tanzania relied on information from respondents who volunteered to participate—a possible source of bias which in turn limited the scope of what the evaluation could conclude about implementation and effectiveness.

Finally, most evaluations relied on a mix of information sources but many analyses failed to show how they derived their conclusions from that information. For example, evaluations of a reproductive health program in India and of a water and sanitation project in Ecuador reached conclusions that were not explicitly linked to the information sources they listed. As a result, the credibility of the findings essentially rested on the judgment of the researchers and could not be validated by retracing their analyses.

Impact and performance evaluations

A great deal of the current debate over aid evaluations treats randomized controlled trials, or RCTs, as if they were displacing other forms of evaluation. To the contrary, less than 2 percent of the 299 evaluations identified by the study were RCTs. When considering all impact evaluations using experimental or quasi-experimental methods, the study found that these accounted for only about six percent of all evaluations; performance evaluations comprised the vast majority.

The impact evaluations generally scored better than performance evaluations on the validity and reliability criteria (Figure 2). About 38 percent of the impact evaluations and 10 percent of the performance evaluations scored high on sampling validity and reliability. Impact evaluations were also more likely to have high analytical validity and reliability—31 percent compared to 5 percent. 

This difference is partially explained by the fact that most impact evaluations rely on methods that explicitly address potential bias through statistical sampling techniques. But performance evaluations, including those using qualitative methods, also have ways to collect information that enhance validity and reliability. For example, qualitative studies can address recall bias by comparing end of project perspectives with those expressed in baseline interviews.