Skip to main content

Advertisement

SpringerLink
Log in

Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review

  • Meta-Analysis
  • Published:
Educational Psychology Review Aims and scope Submit manuscript

Abstract

Meta-analyses are used to make educational decisions in policy and practice. Publication bias refers to the extent to which published literature is more likely to have statistically significant results and larger sample sizes than studies that do not make it through the publication process. The purpose of the present study is to estimate the extent to which publication bias is present in a broad set of education and special education journals. We reviewed 222 meta-analyses to describe the prevalence of publication bias tests, and further identified 29 that met inclusion criteria for effect size extraction. Descriptive data reveal that 58% of meta-analyses (n = 128) documented no effort to test for possible publication bias, and analyses of 72 difference statistics revealed that published studies were associated with significantly larger effect sizes than unpublished studies (d = 0.64). Exploratory moderator analyses revealed that effect size metric was a significant predictor of the difference between published and unpublished studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

References preceded by asterisks were included in the meta-review.

  • *Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: a meta-analysis. Journal of Educational Psychology, 104(1), 250. https://doi.org/10.1037/a0026147.

  • *Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017.

  • *Alfieri, L., Nokes-Malach, T. J., & Schunn, C. D. (2013). Learning through case comparisons: a meta-analytic review. Educational Psychologist, 48(2), 87–113. https://doi.org/10.1080/00461520.2013.775712.

  • *Aloe, A. M., Amo, L. C., & Shanahan, M. E. (2014). Classroom management self-efficacy and burnout: a multivariate meta-analysis. Educational Psychology Review, 26(1), 101–126. https://doi.org/10.1007/s10648-013-9244-0.

  • Banda, D. R., & Therrien, W. J. (2008). A teacher’s guide to meta-analysis. Teaching Exceptional Children, 41(2), 66–71. https://doi.org/10.1177/004005990804100208.

    Article  Google Scholar 

  • Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Medicine, 7(9), e1000326.

    Article  Google Scholar 

  • Becker, B. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 35–48). Chichester: Wiley.

    Google Scholar 

  • Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088–1101. https://doi.org/10.2307/2533446.

  • Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.

    Book  Google Scholar 

  • *Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380–400. https://doi.org/10.1037/a0031084.

  • Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292.

    Article  Google Scholar 

  • Cook, B. G., & Therrien, W. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders. https://doi.org/10.1177/0198742917709473.

  • Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., et al. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269(21), 2749–2753.

    Article  Google Scholar 

  • Cooper, H., & Koenka, A. C. (2012). The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), 446.

    Article  Google Scholar 

  • *Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: a meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8.

  • Dickersin, K. (2005). Publication bias: recognizing the problem, understanding its origin and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Boren- stein (Eds.), Publication bias in metaanalysis: Prevention, assessment and adjustments (pp.11–33). Chichester: England, Wiley.

  • *Dunst, C. J. (2014). Meta-analysis of the effects of puppet shows on attitudes toward and knowledge of individuals with disabilities. Exceptional Children, 80(2), 136–148. https://doi.org/10.1177/001440291408000201.

  • Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x.

    Article  Google Scholar 

  • Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634. https://doi.org/10.1136/bmj.315.7109.629.

    Article  Google Scholar 

  • Every Student Succeeds Act of 2015 (2015). Pub. L. No. 114-95 § 114 Stat. 1177.

  • Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. https://doi.org/10.1037/a0024445.

    Article  Google Scholar 

  • Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177/1745691612459059.

    Article  Google Scholar 

  • *Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: a meta-analysis. Exceptional Children, 53(3), 199–208. https://doi.org/10.1177/004005998401600318.

  • *Gajda, A., Karwowski, M., & Beghetto, R. A. (2016). Creativity and academic achievement: a meta-analysis. Journal of Educational Psychology. Advance online publication: https://doi.org/10.1037/edu0000133.

  • *Ginns, P., Martin, A. J., & Marsh, H. W. (2013). Designing instructional text in a conversational style: a meta-analysis. Educational Psychology Review, 25(4), 445–472. https://doi.org/10.1007/s10648-013-9228-0.

  • Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203.

    Article  Google Scholar 

  • Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.

    Article  Google Scholar 

  • Higgins, JPT, Green S (editors). (2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.

  • Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Library.

  • Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. Publication bias in meta-analysis: prevention, assessment and adjustments, 48–72.

  • Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Library.

  • *Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: a meta-analysis. Journal of Educational Psychology, 104(1), 48–73. https://doi.org/10.1037/a0026223.

  • Individuals with Disabilities Education Act (IDEA) of 2004 (2004) 20 U.S.C. § 1401 et seq.

  • Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.

    Article  Google Scholar 

  • Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235–241.

    Article  Google Scholar 

  • *Klassen, R. M., Tze, V. M., & Hannok, W. (2011). Internalizing problems of adults with learning disabilities: a meta-analysis. Journal of Learning Disabilities, 46(4), 317–327. https://doi.org/10.1177/0022219411422260.

  • Kromrey, J. D., & Rendina-Gobioff, G. (2006). On knowing what we do not know: an empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66(3), 357–373.

  • Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration. https://doi.org/10.4073/cmg.2016.1.

  • *Kulik, J. A., Bangert, R. L., & Williams, G. W. (1983). Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 75(1), 19–26. https://doi.org/10.1037/0022-0663.75.1.19.

  • *Lee, J., & Yoon, S. Y. (2015). The effects of repeated reading on reading fluency for students with reading disabilities: a meta-analysis. Journal of Learning Disabilities. https://doi.org/10.1177/0022219415605194.

  • *Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243. https://doi.org/10.1007/s10648-010-9125-8.

  • Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage.

    Google Scholar 

  • *Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918. https://doi.org/10.1037/a0037222.

  • *Maeda, Y., & Yoon, S. Y. (2013). A meta-analysis on gender differences in mental rotation ability measured by the Purdue spatial visualization tests: visualization of rotations (PSVT: R). Educational Psychology Review, 25(1), 69–94. https://doi.org/10.1007/s10648-012-9215-x.

  • McLeod, B. D., & Weisz, J. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235–251. https://doi.org/10.1037/0022-006x.72.2.235.

    Article  Google Scholar 

  • *Murawski, W. W., & Swanson, H. L. (2001). A meta-analysis of co-teaching research where are the data? Remedial and Special Education, 22(5), 258–267. https://doi.org/10.1177/074193250102200501.

  • *Nelson, J. M., & Harwood, H. (2011). Learning disabilities and anxiety: a meta-analysis. Journal of Learning Disabilities, 44(1), 3–17. https://doi.org/10.1177/0022219409359939.

  • *Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: a meta-analysis. School Psychology Review, 44(2), 224–240. https://doi.org/10.17105/spr-14-0008.1.

  • Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159. https://doi.org/10.2307/1164923.

    Google Scholar 

  • *Owen, K. B., Parker, P. D., Van Zanden, B., MacMillan, F., Astell-Burt, T., & Lonsdale, C. (2016). Physical activity and school engagement in youth: a systematic review and meta-analysis. Educational Psychologist, 51(2), 129–145. https://doi.org/10.1080/00461520.2016.1151793.

  • *Pinquart, M. (2016). Associations of parenting styles and dimensions with academic achievement in children and adolescents: a meta-analysis. Educational Psychology Review, 28(3), 475–493. https://doi.org/10.1007/s10648-015-9338-y.

  • Polanin, J. R., Maynard, B. R., & Dell, N. A. (2017). Overviews in education research: a systematic review and analysis. Review of Educational Research, 87(1), 172–203.

    Article  Google Scholar 

  • Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: a meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067.

    Article  Google Scholar 

  • Rosenthal, R. (1979). The file drawer problem and tolerance for null result. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638.

    Article  Google Scholar 

  • Rothstein, H. R., Bornstein, M., & Sutton, A. J. (2006). Publication bias in meta-analysis. Prevention, assessment, and adjustments. Chichester: Wiley.

    Google Scholar 

  • *Schwinger, M., Wirthwein, L., Lemmer, G., & Steinmayr, R. (2014). Academic self-handicapping and achievement: a meta-analysis. Journal of Educational Psychology, 106(3), 744–761. https://doi.org/10.1037/a0035832.

  • Spaulding, L. S. (2009). Best practices and interventions in special education: how do we know what works? Teaching Exceptional Children Plus, 5(3), 1–13.

    Google Scholar 

  • *Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. https://doi.org/10.1037/a0032447.

  • Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 145–174). Chichester: Wiley.

    Google Scholar 

  • Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055.

  • *Swanson, H. L. (1999). Reading research for students with LD: a meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32(6), 504–532. https://doi.org/10.1177/002221949903200605.

  • Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 4, 13–30. https://doi.org/10.1002/jrsm.1091.

    Article  Google Scholar 

  • *Tenenbaum, H. R., & Ruck, M. D. (2007). Are teachers’ expectations different for racial minority than for European American students? A meta-analysis. Journal of Educational Psychology, 99(2), 253–273. https://doi.org/10.1037/0022-0663.99.2.253.

  • *Theule, J., Wiener, J., Tannock, R., & Jenkins, J. M. (2012). Parenting stress in families of children with ADHD a meta-analysis. Journal of Emotional and Behavioral Disorders, 21(1), 3–17. https://doi.org/10.1177/1063426610387433.

  • Torgerson, C. J. (2006). Publication bias: the Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54(1), 89–102.

    Article  Google Scholar 

  • Torgerson, C., Porthouse, J., & Brooks, G. (2003). A systematic review and meta-analysis of randomised controlled trials evaluating interventions in adult literacy and numeracy. Journal of Research in Reading, 26(3), 234–255.

    Article  Google Scholar 

  • *Vasquez, A. C., Patall, E. A., Fong, C. J., Corrigan, A. S., & Pine, L. (2016). Parent autonomy support, academic achievement, and psychosocial functioning: a meta-analysis of research. Educational Psychology Review, 28(3), 605–644. https://doi.org/10.1007/s10648-015-9329-z.

  • Wilson, D. (2009). Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. Journal of Experimental Criminology, 5, 429–440. https://doi.org/10.1007/s11292-009-9085-5.

    Article  Google Scholar 

  • *Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. https://doi.org/10.1037/a0031311.

Download references

Author information

Authors and Affiliations

  1. Virginia Commonwealth University, 1015 W Main Street, Richmond, VA, 23284, USA

    Jason C. Chow & Eric Ekholm

Authors
  1. Jason C. Chow
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Ekholm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason C. Chow.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chow, J.C., Ekholm, E. Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review. Educ Psychol Rev 30, 727–744 (2018). https://doi.org/10.1007/s10648-018-9437-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10648-018-9437-7

Keywords

Search

Navigation