PEPFAR Reauthorization V: Science = Life

While AIDS activists in the 1980's were surely right that Silence = Death, it was not voices and attention alone that brought the life-sustaining advances of AZT treatment and its successors. For all the flaws in process along the way, it was breakthroughs in biomedical science that yielded the therapies that literally raise people from their deathbeds to be productive and active members of their families and communities. Without the investments in the science behind the drugs, the conversation today at the XVII International AIDS Conference in Mexico City would be far different – or would not be occurring at all.In sharp contrast to the biomedical and pharmacological advances, though, the science of implementation of large-scale programs is far less developed. We have relatively little solid evidence to go on to ensure that the billions of donor dollars on offer are going to bring about the greatest individual and social benefit, with the fewest costs to nations' ability to deal with other health priorities. Instead of findings from systematic study, for the most part we are left with common sense, instinct and belief, shaped by the occasional compelling anecdote or observational study. The early "emergency"-oriented nature of PEPFAR, which eschewed "the R word" (research), did little to fill the knowledge gaps. As the Institute of Medicine noted in its 2007 assessment of PEPFAR's performance, "There are still more questions than answers about how to best provide ART" in low-income, high-burden countries. And the paucity of knowledge about the effects of PEPFAR's HIV prevention efforts has been even more profound – and more lethal.The good news is that Congress noticed, and took the opportunity of the recent Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (also known as PEPFAR reauthorization) to bring some much needed energy and resources to careful evaluation of program approaches. Specifically, the legislation calls for the program to include three types of implementation monitoring and evaluation:

IMPACT EVALUATION RESEARCH- The term 'impact evaluation research' means the application of research methods and statistical analysis to measure the extent to which change in a population-based outcome can beattributed to program intervention instead of other environmental factors.OPERATIONS RESEARCH- The term 'operations research' means the application of social science research methods, statistical analysis, and other appropriate scientific methods to judge, compare, and improve policies and program outcomes, from the earliest stages of defining and designing programs through their development and implementation, with the objective of the rapid dissemination of conclusions and concrete impact on programming.PROGRAM MONITORING- The term 'program monitoring' means the collection, analysis, and use of routine program data to determine how well a program is carried out; and how much the program costs.

Over the next 18 months, the Institute of Medicine is charged with preparing a plan for collecting baseline data (missing the first time around) and follow-up data for a study (or set of studies) that in four years can provide:

(i) an assessment of the performance of United States-assisted global HIV/AIDS programs; and(ii) an evaluation of the impact on health of prevention, treatment, and care efforts that are supported by United States funding, including multilateral and bilateral programs involving joint operations.…The study … shall include--(i) an assessment of progress toward prevention, treatment, and care targets;(ii) an assessment of the effects on health systems, including on the financing and management of health systems and the quality of service delivery and staffing;(iii) an assessment of efforts to address gender-specific aspects of HIV/AIDS, including gender related constraints to accessing services and addressing underlying social and economic vulnerabilities of women and men;(iv) an evaluation of the impact of treatment and care programs on 5-year survival rates, drug adherence, and the emergence of drug resistance;(v) an evaluation of the impact of prevention programs on HIV incidence in relevant population groups;(vi) an evaluation of the impact on child health and welfare of interventions authorized under this Act on behalf of orphans and vulnerable children;(vii) an evaluation of the impact of programs and activities authorized in this Act on child mortality; and(viii) recommendations for improving the programs.

And as for evaluation methods, the framers of the legislation signal that they are not talking about helicoptering in a consulting team for a week to interview 25 people who might have a view about the success of the program. They are instead talking about something that looks a lot more like, yes, science:

Assessments and impact evaluations conducted under the study shall utilize sound statistical methods and techniques for the behavioral sciences, including random assignment methodologies as feasible. Qualitative data on process variables should be used for assessments and impact evaluations, wherever possible.

To those of us who have been dismayed at the erosion of serious evaluation in the U.S. development assistance program (with the notable exception of the Millennium Challenge Corporation), and who care a lot about whether all the good works in aid programs are generating the most sustainable benefit – knowledge – this initiative is welcome indeed. But it will come to naught, like the impressive evaluation plans described in hundreds of other project documents, without the technical skill and patience to design the right studies; the protected funding so resources are in place when it comes time to collect the subsequent rounds of data; and the willingness on the part of program leaders to share both successes and failures. The legislative language is a great start. We'll stay tuned to see what happens next.