Skip to main content
SpringerLink
Log in

Publication bias as a threat to the validity of meta-analytic results

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

This paper reviews the evidence in support of the contention that publication bias is a potential threat to the validity of meta-analytic results in criminology and similar fields. It then provides a critique of the traditional file drawer or failsafe N method for examining publication bias, and an overview of four newer methods that can be used to detect publication bias. These include two (trim and fill and cumulative meta-analysis) that enable the researcher to estimate the magnitude of the influence of publication bias on the overall mean effect size. Advantages and limitations of both traditional and newer methods are examined. The methods reviewed are illustrated through their application to a meta-analysis of the effects of drug courts on recidivism by Wilson et al. (Journal of Experimental Criminology, 2, 459–487, 2006).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Publication bias was originally defined as the publication or non-publication of studies depending on the direction and statistical significance of the results, and the first systematic investigations of publication bias focused on this aspect of the problem. However, as readers will appreciate, there are numerous potential mechanisms for the suppression of information that go well beyond the simple definition given above, including language bias (selective inclusion of studies published in English); availability bias (selective inclusion of studies that are easily accessible to the researcher); cost bias (selective inclusion of studies that are available free or at low cost); familiarity bias (selective inclusion of studies only from one’s own discipline); outcome bias (selective reporting by the author of a primary study of some outcomes but not others depending on the direction and statistical significance of the results) and duplication bias (some findings are likely to be published more than once and may be included more than once in a meta-analysis) . In addition, data may “go missing” for reasons other than those generally considered as causing publication bias, including financial, political, ideological, and professional competing interests of investigators, research sponsors, journal editors and other parties. As all of these sources of bias lead to the same consequence, namely that the literature located by a systematic reviewer will be unrepresentative of the population of completed studies, all raise the same threat to validity. Readers should bear in mind that when they read “publication bias” any or all of these biases may be implied.

References

  • Begg, C. B. (1994). Publication Bias. In H. Cooper & L. Hedges (Eds.), The Handbook of Research Synthesis, NY: Russell Sage.

  • Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A, 151, 419–463.

    Article  Google Scholar 

  • Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.

    Article  Google Scholar 

  • Borenstein, M. (2005). Software for publication bias. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.

  • Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive Meta Analysis, Version 2. Biostat, Englewood, NJ.

  • Braga, A. (2005). Hot Spots Policing and Crime Prevention: A Systematic Review of Randomized Controlled Trials. Journal of Experimental Criminology, 1, 3.

  • Chan, A-W., & Altman, D. G. (2005). Outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ, 330, 753.

    Article  Google Scholar 

  • Chan, A-W., Hrobjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. Journal of the American Medical Association, 291, 2457–2465.

    Article  Google Scholar 

  • Cooper, H. M. (1979). Statistically combining independent studies: A meta-analysis of sex differences in conformity research. Journal of Personality and Social Psychology, 37, 131–146.

    Article  Google Scholar 

  • Copas, J. B. (1999). What works? Selectivity models and meta-analysis. Journal of the Royal Statistical Association, 162, 95–109.

    Article  Google Scholar 

  • Copas, J. B., & Shi, J. Q. (2001). A sensitivity analysis for publication bias in systematic reviews. Statistical Methods in Medical Research, 10, 251–265.

    Article  Google Scholar 

  • Deffenbacher, K., Bornstein, B., Penrod, S., & McGorty, K. (2004). Meta-Analytic Review of the Effects of High Stress on Eyewitness Memory. Law and Human Behavior, 28, 687–706.

    Article  Google Scholar 

  • Del Vecchio, T., & O’Leary, K. D. (2004). The effectiveness of anger treatments for specific anger problems: A meta-analytic review. Clinical Psychology Review, 24, 15–34.

    Article  Google Scholar 

  • Dickersin, K. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 11–34). Chichester, UK: Wiley.

  • Dreznick, M. (2003). Heterosocial competence of rapists and child molesters: a meta-analysis. Journal of Sex Research, 40, 170–178.

    Article  Google Scholar 

  • Duval, S. (2005). The “trim and fill” method. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 127–144). Chichester, UK: Wiley.

  • Duval, S. J., & Tweedie, R. L. (2000a). A non-parametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98.

    Article  Google Scholar 

  • Duval, S. J., & Tweedie, R. L. (2000b). Trim and Fill: A simple funnel plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 276–284.

    Article  Google Scholar 

  • Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315, 629–634.

    Google Scholar 

  • Egger, M., Davey Smith, G., & Altman, D. G. (2000). Systematic reviews in health care: Meta-analysis in context. London: BMJ Books.

  • Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246–255.

    Article  Google Scholar 

  • Hedges, L. V., & Vevea, J. L. (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21, 299–332.

    Article  Google Scholar 

  • Hedges, L., & Vevea, J. (2005). The selection model approach to publication bias. In H. Rothstein, A.J. Sutton, & M. Borenstein (Eds.). Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.

  • Illescas, S., Sanchez-Meca, J., & Garrido, V. (2001). Treatment of offenders and recidivism: Assessment of the effectiveness of programmes applied in Europe. Psychology in Spain, 5, 47–62.

    Google Scholar 

  • Ioannidis, J. (2005). Differentiating biases from genuine heterogeneity. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.). Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.

  • Krleža-Jeric, K., Chan, A-W., Dickersin, K., Sim, I., Grimshaw, J., Gluud, C., for the Ottawa Group. (2005) Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa Statement (part 1). BMJ, 330, 956–958.

  • Landenberger, N., & Lipsey, M. W. (2005). The positive effects of cognitive-behavioral programs for offenders: A meta-analysis of factors associated with associated with effective treatment. Journal of Experimental Criminology, 1, 451–476.

    Article  Google Scholar 

  • Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine, 327, 248–254.

    Article  Google Scholar 

  • Light, R., & Pillemer, D. (1984). Summing Up: the Science of Reviewing Research. Cambridge, MA: Harvard University Press.

  • Light, R. J., Singer, J. D., & Willett, J. B. (1994). The visual presentation and interpretation of meta-analyses. In H. M. Cooper, & L. V. Hedges (Eds.), The Handbook of Research Synthesis. New York: Russell Sage Foundation.

  • Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.

    Article  Google Scholar 

  • Losel, F., & Schmucker, M. (2005). The effectiveness of treatment for sexual offenders: A comprehensive meta-analysis.

  • McCord, J. (2003). Cures that harm: Unanticipated outcomes of crime prevention programs. Annals of the American Academy of Political and Social Science, 587, 16–30.

    Article  Google Scholar 

  • Mullen, B., Muellerleile, P., Bryant, B. (2001). Cumulative meta-analysis: A consideration of indicators of sufficiency and stability. Personality and Social Psychology Bulletin, 27, 1450–1462.

    Article  Google Scholar 

  • NY vs GlaxoSmithKline. Filing to the Supreme Court of the State of New York. June 2, 2004.

  • Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159.

    Article  Google Scholar 

  • Pratt, T., Cullen, C., Blevins, K., Daigle, L., & Unnver, J. (2002). The relationship of attention deficit hyperactivity disorder to crime and delinquency: A meta-analysis. International Journal of Police Science and Management, 4, 344–360.

    Article  Google Scholar 

  • Rhoades, L., & Eisenberger, R. (2002). Perceived organizational support: A review of the literature. Journal of Applied Psychology, 87, 698–714.

    Article  Google Scholar 

  • Richy, F., & Reginster, J. (2006). A Simple Method for Detecting and Adjusting Meta-Analyses for Publication Bias. The Internet Journal of Epidemiology, 3, 2.

    Google Scholar 

  • Rosenberg, M. S., Adams, D. C., & Gurevitch, J. (2000). MetaWin: Statistical Software for Meta-Analysis, Version 2.0. Sunderland, MA: Sinauer Associates.

  • Rosenthal, R. (1979).The ‘File Drawer Problem’ and Tolerance for Null Results. Psychological Bulletin, 86, 638–641.

    Google Scholar 

  • Sterne, J. A. C., Bradburn, M. J., & Egger, M. (2001a). Meta-Analysis in Stata. In M. Egger, G. Davey Smith, & D. G. Altman (Eds.), Systematic Reviews in Health Care: Meta Analysis in Context (pp. 347–369). London: BMJ.

  • Sterne, J. A. C., Egger, M., & Davey Smith, G. (2001b). Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. BMJ, 323, 101–105.

    Article  Google Scholar 

  • Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. . In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.

  • Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046–1055.

    Article  Google Scholar 

  • Sterne, J. A. C., & Egger, M. (2005). Regression methods to detect publication and other bias in meta-analysis. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.

  • Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119–1129.

    Article  Google Scholar 

  • Sterne, J. A. C., Harris, R., Harbord, R., & Steichen, T. (2007). What meta-analysis features are available in Stata? Retrieved from http://www.stata.com/support/faqs/stat/meta.html on 21 June 2007.

  • Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113–2126.

    Article  Google Scholar 

  • Wall Street Journal, (2004, November 1). E-Mails Suggest Merck Knew Vioxx’s Dangers at Early Stage.

  • Weisburd, D., Petrosino, A., & Mason, G. (1993). Design sensitivity in criminal justice experiments. Crime and Justice, 17, 337–379.

    Article  Google Scholar 

  • Wilson, D. B., Mitchell, O., & MacKenzie, D. (2006). A systematic review of drug court effects on recidivism. Journal of Experimental Criminology, 2, 459–487.

    Article  Google Scholar 

  • Zhang, J., Ding, E. L., & Song, Y. (2006). Adverse effects of cyclooxygenase 2 inhibitors on renal and arrhythmia events: meta-analysis of randomized trials. JAMA, 296, 1619–1632.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Management Zicklin School of Business, Baruch College—CUNY, New York, NY, USA

    Hannah R. Rothstein

Authors
  1. Hannah R. Rothstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hannah R. Rothstein.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rothstein, H.R. Publication bias as a threat to the validity of meta-analytic results. J Exp Criminol 4, 61–81 (2008). https://doi.org/10.1007/s11292-007-9046-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-007-9046-9

Keywords

Search

Navigation