The World Bank’s Independent Evaluation Group has begun an evaluation of one of the Bank’s flagship products—Doing Business. It is asking whether Doing Business indicators capture reform priorities and whether using Doing Business indicators to drive Bank operations makes sense. I am one of three peer reviewers for this exercise which is in two phases. Here, I present four questions that I believe must underpin all aspects of the Independent Evaluation Group (IEG) evaluation.
Is the scientific and empirical basis of the index sound?
If one believes the theory that governments cannot address multiple constraints simultaneously and that a focus on binding constraint(s) is needed, then the question is whether regulatory issues are the binding constraint. The Doing Business-driven private sector agenda assumes that regulatory reforms are always the highest payoff reforms; the review must not fall into the same ideological trap. The World Bank’s own Enterprise Surveys might help determine which constraints are most important; they are not mentioned in the concept note but could be used as a check to determine what actually matters to firms. Beyond helping to establish whether the regulatory agenda should be the focus of private sector reforms, the Enterprise Surveys could also be useful in understanding whether the de jure issues reported to be affecting the “representative” firm (in Doing Business) are also reported to de facto be the issues affecting most firms of all sizes (in the Enterprise Surveys).
The Doing Business index ought to be a measure of regulatory/policy best practice in each of the dimensions covered. But it makes no attempt to be that. It does not try to establish what aspects of, say, electricity provision or contract enforcement matter most to firms in any given country. It only measures the costs of taxes and regulations, not the benefits. That’s just a silly basis for an index that purports to influence policy and guide World Bank technical assistance. The Bank can do better–indeed, it *does* do better in a lot of its technical assistance on these very same topics.
Should Doing Business drive private sector operations?
The Doing Business indicators have become the central focus of the Bank’s regulatory and private sector operations, promoting the central concept that fewer regulations in all areas of economic activity will pay off for the Bank’s clients. As January Makamba, a member of the Tanzanian parliament, said to the Financial Times last week, “A lot of policy recommendations and prescriptions, and judgments on FDI direction….in developing countries have been based on this report.”
In evaluating the success of this approach, the IEG team must ask the following questions—what if the reform priorities suggested by the Doing Business project are not the priorities of the government? What if they are not consistent with the views of the country team, particularly the operations staff responsible for designing and implementing private sector projects?
My view is that Bank staff are often in a situation where they must pursue reforms related to the Doing Business indicators rather than relying on other sources of data or their own, more detailed, knowledge of the country. The question is: what would the private sector operations officer have done if they had not required to do a Doing Business project? Would the alternative project have been better for private sector development? For revenue mobilization? For macroeconomic stabilization? It is difficult to answer this question through a literature review but asking it is central to measuring the “success” of reforms. It may be worthwhile for the IEG team to look at country-level documents other than those directly focused on Doing Business projects. Related to this, it would be useful to be upfront about the definition of success and whether it relates to simply to improving the Doing Business ranking or to real outcomes in the economy.
What are the pros and cons of a deterministic method?
Doing Business is deterministic. The advantage is that it is an actionable index; the drawback that the particular process—time and cost to start a business in City X in Country Y—may not be a precise measure of what actually prevails in Country Y (or even in City X for that matter). In most of the indices we have, there are broad cross-country patterns that are very likely correct, in the sense that New Zealand is almost certainly better than the DRC on almost any index you can find. But the likely size of the standard errors means that it is difficult to definitively proclaim an improvement without a huge change that probably requires many years. This is one of the pluses of the Worldwide Governance Indicators; it does try to estimate standard errors and it takes a big change, say like the trend for Venezuela, to separate out changes from the noise. Doing Business on the other hand has regularly recorded large swings in country rankings from year to year and attributed those to success or failure of reforms.
Frequent changes to methods may also lead to spurious jumps in rankings. Let us take the example of India. Justin Sandefur and Divyanshi Wadhwa argue convincingly that a change in Doing Business methods—rather than any real reform—is the explanation for India’s recent improvement of 30 places in the Doing Business rankings; applying a consistent methodology suggests a change of only five places.
Can data management practices be improved?
Recent events have raised concerns about ad hoc changes to the Doing Business data; this concern was also raised (as I recall) in the very first IEG review of Doing Business which picked up over two thousand ad hoc undocumented changes to the data. The Doing Business project lags relative to the overall data practices of the World Bank; its sloppiness in not properly documenting changes reflects badly on other valuable data and indices produced by the Bank. It is impossible to measure “successful” outcomes if values are being modified without any documentation. One thing that would be useful is to see if the values of the Doing Business data values for, say, 2010 are the same now as they were when they were reported in 2010. If not, then how does one declare the project to be a success (or not)? In my view, the Doing Business project *must* record changes to the data in a transparent manner if it is to retain any credibility.
The methodological issues raised here are not unique to Doing Business. At CGD, we produce an index called the Commitment to Development Index. We have also grappled with questions about data aggregation, most recently in an external review. An honest and open dialogue about the pros and cons of using indices is critical to using them in the best possible way.
I am grateful to Alan Gelb, Charles Kenny, Christian Meyer, Scott Morris, Justin Sandefur, and Gaiv Tata for their contributions to this post.
Disclaimer
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.
