Do Trees Grow on Money? The Problem of Attribution

This is a joint post with Nancy Birdsall. 

How can donors know if their aid is making a difference? This question is tougher than it seems. Attributing results to donor inputs seems straightforward if the donor pays for progress on a measurable outcome, as CGD has proposed for Cash on Delivery Aid (COD Aid). If the desired results are achieved—say an increase above an agreed baseline in the number of kids completing primary school and taking a competency test—then the program has demonstrated value for money, no?  In fact even an outcome-driven, pay-for-performance approach doesn’t make attribution easy—or even necessarily appropriate.

Why? Because “performance” is a function not only, or even mostly, of a donor’s money on the table.  It is a function of hundreds of decisions about use of resources, politics and personnel, changes of bureaucratic rules and customs, institutional fixes or lack thereof, all at multiple levels from national to local. Some of those decisions may be hard to identify and impossible to attribute to the prospective donor payment.  Many decisions may be made in an ongoing process of innovating and adapting to newly discovered constraints: so-called “crawling the design space”.

Should all those decisions (or non-decisions) be attributed to the prospect of a donor-funded reward?  Did the payment inspire and catalyze a process or not? And what about the time horizon? The problem is not just what can be attributed, but when.  Just because the payment for performance is made based on annual progress doesn’t mean the overall result of a program shouldn’t be assessed over a longer time period: five or even ten years.

Consider the example of paying for reduced deforestation. In 2007, Norway made a courageous commitment to allocate more than $500 million per year to reward forest-rich countries in the tropics for reducing the climate emissions associated with deforestation.  Between 2008 and 2010, bilateral payment-for-performance agreements were concluded with Brazil (up to $1 billion), Indonesia (up to $1 billion) and Guyana (up to $250 million). 

These deals represent the first round of experimentation with a simple, outcomes-based approach to promote conservation of tropical forests where decades of traditional development projects have mostly failed. Paralleling the growing focus on results in Aid World, the three agreements were developed in Reducing Emissions from Deforestation and forest Degradation (REDD+) World, the mechanism to reduce forest-based emissions negotiated under the United Nations Framework Convention on Climate Change (UNFCCC).

The REDD+ approach has much in common with COD Aid being pioneered by the UK and other donors in education, health, energy access and other sectors. Accordingly, policy-makers and practitioners from both Climate World and Aid World – not to mention Norwegian taxpayers – are interested in the lessons learned from the pioneering efforts in these three countries.

Are the agreements making a difference in reducing deforestation?  We think it is too soon to know – and that we may never know in a way that can be quantified, because reducing deforestation from what it might have been otherwise is likely to be the outcome of many factors, and the kind of ongoing, adaptive process referred to above.

The dramatic drop in Brazil’s deforestation rate over the last decade was already underway at the time of the agreement with Norway in 2008 – and has continued despite slow disbursement of funds earned from subsequent reductions in deforestation. Does that mean the agreement made no difference? In the case of Indonesia, the deforestation rate has actually increased since negotiation of the Letter of Intent with Norway in 2010; the question is whether it might have increased more than it did without the agreement, which prompted a moratorium on new concessions on peatlands and primary forest areas.   

Understanding the impacts of the Norwegian money in each country – much less relationships that hold across countries – is not at all straightforward.  We don’t know what would have happened in the absence of the agreements—we lack counterfactuals. Moreover, with only three cases, we don’t have sufficient data to conduct statistical analysis to identify the relative importance of outcomes-based finance among the many factors that might influence the deforestation rate.

While a billion dollars is an eye-catching figure, in fact it is trivial compared to the sizes of Brazil’s and Indonesia’s economies, and the money to be made from business-as-usual conversion or degradation of forests in favor of globally-traded commodities, such as palm oil in Indonesia or gold in Guyana. A more appropriate test of the potential of REDD+ would require a bigger prize more in line with the effort required to claim it. 

Our hunch is that the political symbolism of a partnership among equals in addressing a global problem has punched above its weight, i.e., it has already been important in influencing decision-making in forest-rich countries.

For example, we think it is certainly plausible that the agreement in Brazil helped to consolidate and maintain the political will behind law enforcement and other efforts already underway. 

In Guyana, the agreement reinforced a movement started by then-President Jagdeo to promote low-carbon development, and to prevent already-low deforestation rates from rising.  It funded and likely accelerated development of an MRV system that is far ahead of similar programs in other small, lower-income countries. It also prompted initiation of improvements in forest governance, including through the regulatory functions of the Geology and Mines Commission.

In Indonesia, it makes little sense to expect that reforms directly prompted by the agreement with Norway and designed to provide a platform for REDD+ – such as increased transparency of forest-related data, and establishment of a national REDD+ agency – would already have resulted in reducing the deforestation rate; some fundamental reforms simply take time to translate into impact on the ground. 

Nonetheless, in Indonesia and elsewhere, it would be surprising indeed if the head-of-state level attention and national-level debate on forests prompted by Norway’s billion-dollar pledges were not having some influence on broader domestic and international trends relevant to deforestation – including the uptick in attention to getting deforestation out of commodity supply chains, increasing visibility and attention to forest-related crime and corruption, and increasing legitimacy for indigenous peoples’ rights to forest resources.  And at the same time, all of those trends make investment in REDD+ more attractive.

These kinds of influences cannot be quantified, especially not in the short run.

While the jury is still out, several serious efforts are underway to assess the impacts of the Norwegian rainforest billions, and engage the thorny question of attribution.  A “real-time” evaluation commissioned by the Government of Norway will be published in the near future, and no doubt will provide significant illumination.  Researchers affiliated with the Center for International Forestry Research (CIFOR) are updating a Qualitative Comparative Analysis (covering 12 countries) to illuminate the factors – of which the existence of a payment-for-performance agreement will now be one – that are associated with significant forest policy reform in the context of REDD+ initiatives. 

And as part of CGD’s Norwegian-funded Tropical Forests for Climate and Development Initiative we are commissioning journalists to conduct interviews with insiders and informed observers to gain a better understanding of how the Norwegian agreements have affected the political economy of forest management in each country.  

We look forward to a debate on the attribution of impact based on more evidence and analysis. In the meantime, both more money and more cases are needed to support conclusions about “what works” in outcomes-based finance for forests between now and the hoped-for global climate agreement to be implemented in 2020.