*REVISED Version May 2008
In development economics, statistical analysis often begins with data from many observational units--households, companies, or countries--over just a few time periods. Two analysis techniques, the "difference" and "system" generalized method of moments (GMM) estimators, are becoming popular for studying causal relationships among variables in this "short panel" setting. This working paper by CGD research fellow David Roodman dramatizes a common but underappreciated problem in how the estimators often are applied. It explains the risks, offers some simple techniques for reducing them, and illustrates, with reference to two early and widely noted applications to studying what causes economic growth: Forbes (2000) on income inequality and Levine, Loayza, and Beck (2000) on financial sector development. Endogenous causation proves hard to rule out in both papers, meaning that we cannot be as confident, after all, that inequality or financial development causes growth.
Judging by current practice, many researchers do not fully appreciate that popular implementations of these estimators can by default generate results that simultaneously are invalid yet appear valid. The potential for "Type I errors"--false positives--is therefore substantial, especially after amplification by publication bias.
In particular, the estimators tend to generate many instruments compared to the number of observational units or data observations, which can cause a number of problems already documented in the literature. The instruments can overfit endogenous variables, failing to expunge their endogenous components and biasing coefficient estimates. They can vitiate the Hansen J test for joint validity of those instruments, as well as the difference-in-Sargan/Hansen test for subsets of instruments. The weakness of these specification tests is a particular concern for system GMM, whose distinctive instruments are only valid under a non-trivial assumption.
The reproduction Forbes data set and the original Levine, Loayza, and Beck data set, both used in the paper, are available for download, along with a Stata command file that generates all the empirical results referred to in the paper and another that runs the simulations in section 4.
This paper is forthcoming in the Oxford Bulletin of Economics and Statistics. David Roodman is also the author of the most popular program implementing these estimators (see his CGD working paper 103 How to Do xtabond2).
Rights & Permissions
You may use and disseminate CGD’s publications under these conditions.
