Skip to main content
SpringerLink
Log in

Replication Research, Publication Bias, and Applied Behavior Analysis

  • Commentary
  • Published:
Perspectives on Behavior Science Aims and scope Submit manuscript

Abstract

The “replication crisis” describes recent difficulties in replicating studies in various scientific fields, most notably psychology. The available evidence primarily documents replication failures for group research designs. However, we argue that contingencies of publication bias that led to the “replication crisis” also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication of SCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect. The resulting research literature may unjustifiably inflate confidence about intervention effects, limit researchers’ ability to delineate intervention boundary conditions, and diminish the credibility of our science. To counter problems of publication bias in ABA, we recommend that journals that publish SCRD research establish journal standards for publication of noneffect studies; that our research community adopt open sharing of SCRD protocols and data; and that members of our community routinely publish systematic literature reviews that include gray (i.e., unpublished) research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barton, E. E., Reichow, B., Schnitz, A., Smith, I. C., & Sherlock, D. (2015). A systematic review of sensory-based treatments for children with disabilities. Research in Developmental Disabilities, 37, 64–80.

    Article  PubMed  Google Scholar 

  • Bloom, S. E., Iwata, B. A., Fritz, J. N., Roscoe, E. M., & Carreau, A. B. (2011). Classroom application of a trial-based functional analysis. Journal of Applied Behavior Analysis, 44, 19–31.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bullock, C. E., Fisher, W. W., & Hagopian, L. P. (2017). Description and validation of a computerized behavioral data program: “BDataPro.”. The Behavior Analyst, 40, 275–285. https://doi.org/10.1007/s40614-016-0079-0.

    Article  Google Scholar 

  • Carr, E. G., & McDowell, J. J. (1980). Social control of self-injurious behavior of organic etiology. Behavior Therapy, 11, 402–409.

    Article  Google Scholar 

  • Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18. https://doi.org/10.1037/0022-006X.66.1.7.

    Article  PubMed  Google Scholar 

  • Cook, B. G., Collins, L. W., Cook, S. C., & Cook, L. (2016). A replication by any other name: A systematic review of replicative intervention studies. Remedial & Special Education, 37, 223–234.

    Article  Google Scholar 

  • Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Cooper, L. J., Wacker, D. P., Sasso, G. M., Reimers, T. M., & Donn, L. K. (1990). Using parents as therapists to evaluate appropriate behavior of their children: Application to a tertiary diagnostic clinic. Journal of Applied Behavior Analysis, 23, 285–296.

    Article  PubMed  PubMed Central  Google Scholar 

  • Council for Exceptional Children. (2014). Council for Exceptional Children standards for evidence-based practices in special education. Arlington, VA: Author.

    Google Scholar 

  • Cox, A. L., Gast, D. L., Luscre, D., & Ayres, K. M. (2009). The effects of weighted vests on appropriate in-seat behaviors of elementary-age students with autism and severe to profound intellectual disabilities. Focus on Autism & Other Developmental Disabilities, 24, 17–26.

    Article  Google Scholar 

  • Davis, T. N., Dacus, S., Strickland, E., Copeland, D., Chan, J. M., Blenden, K., et al. (2013). The effects of a weighted vest on aggressive and self-injurious behavior in a child with autism. Developmental Neurorehabilitation, 16, 210–215.

    Article  PubMed  Google Scholar 

  • Denton, T. F., & Meindl, J. N. (2016). The effect of colored overlays on reading fluency in individuals with dyslexia. Behavior Analysis in Practice, 9, 191–198.

    Article  PubMed  Google Scholar 

  • Deochand, N., Costello, M. S., & Deochand, M. E. (2018). Behavioral research with planaria. Perspectives on Behavior Science, 41, 447–464. https://doi.org/10.1007/s40614-018-00176-w.

    Article  Google Scholar 

  • Dixon, M. R., Reed, D. D., Smith, T., Belisle, J., & Jackson, R. E. (2015). Research rankings of behavior analytic graduate training programs and their faculty. Behavior Analysis in Practice, 8, 7–15.

    Article  PubMed  PubMed Central  Google Scholar 

  • Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: it's all in the mind, but whose mind? PloS ONE, 7, e29081.

    Article  PubMed  PubMed Central  Google Scholar 

  • Driessen, E., Hollon, S. D., Bockting, C. L., Cuijpers, P., & Turner, E. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS ONE, 10, e0137864.

    Article  PubMed  PubMed Central  Google Scholar 

  • Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 621.

    Article  PubMed  PubMed Central  Google Scholar 

  • Eldevik, S., Hastings, R. P., Hughes, J. C., Jahr, E., Eikeseth, S., & Cross, S. (2009). Meta-analysis of early intensive behavioral intervention for children with autism. Journal of Clinical Child & Adolescent Psychology, 38, 439–450.

    Article  Google Scholar 

  • Evanschitzky, H., & Armstrong, J. S. (2010). Replications of forecasting research. International Journal of Forecasting, 26, 4–8.

    Article  Google Scholar 

  • Ferster, C. B., & Skinner, B. F. (1957). Schedules of reinforcement. Cambridge, MA: B. F. Skinner Foundation.

    Book  Google Scholar 

  • Foxx, R. M., & Mulick, J. A. (Eds.). (2016). Controversial therapies for autism and intellectual disabilities: Fad, fashion, and science in professional practice (2nd ed.). New York, NY: Routledge.

    Google Scholar 

  • Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502–1505.

    Article  PubMed  Google Scholar 

  • Gage, N. A., Cook, B. G., & Reichow, B. (2017). Publication bias in special education meta-analyses. Exceptional Children, 83, 428–445.

    Google Scholar 

  • Ganz, J. B., Earles-Vollrath, T. L., Mason, R. A., Rispoli, M. J., Heath, A. K., & Parker, R. I. (2011). An aggregate study of single-case research involving aided AAC: Participant characteristics of individuals with autism spectrum disorders. Research in Autism Spectrum Disorders, 5, 1500–1509.

    Article  Google Scholar 

  • Gelman, A., & Fung, K. (2016). The power of the “Power Pose.” Slate Magazine. Retrieved January 20, 2019, from http://www.slate.com/articles/health_and_science/science/2016/01/amy_cuddy_s_power_pose_research_is_the_latest_example_of_scientific_overreach.html.

  • Gilroy, S. P., Franck, C. T., & Hantula, D. A. (2017). The discounting model selector: Statistical software for delay discounting applications. Journal of the Experimental Analysis of Behavior, 107(3), 388–401. https://doi.org/10.1002/jeab.257.

    Article  PubMed  Google Scholar 

  • Gilroy, S. P., Kaplan, B. A., Reed, D. D., Koffarnus, M. N., & Hantula, D. A. (2018). The demand curve analyzer: Behavioral economic software for applied research. Journal of the Experimental Analysis of Behavior. https://doi.org/10.1002/jeab.479.

  • Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018). Improving psychological science through transparency and openness: An overview. Perspectives on Behavior Science. https://doi.org/10.1007/s40614-018-00186-8.

  • Hanley, G. P. (2017). Editor’s note. Journal of Applied Behavior Analysis, 50, 3–7.

    Article  PubMed  Google Scholar 

  • Hantula, D. A. (2016a). Editorial: A very special issue. The Behavior Analyst, 39, 1–5. https://doi.org/10.1007/s40614-016-0066-5.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hantula, D. A. (2016b). Expanding the scope: Beyond the familiar and beyond the page. The Behavior Analyst, 39, 189–196. https://doi.org/10.1007/s40614-016-0078-1.

    Article  Google Scholar 

  • Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.

    Google Scholar 

  • Hubbard, R., & Armstrong, J. S. (1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing, 11, 233–248.

    Article  Google Scholar 

  • Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ioannidis, J. P. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645–654.

    Article  PubMed  Google Scholar 

  • Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S. (1982). Toward a functional analysis of self-injury. Analysis & Intervention in Developmental Disabilities, 2, 3–20.

    Article  Google Scholar 

  • Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S. (1994). Toward a functional analysis of self-injury. Journal of Applied Behavior Analysis, 27, 197–209.

    Article  PubMed  PubMed Central  Google Scholar 

  • Jamshidi, L., Heyvaert, M., Declercq, L., Fernández-Castilla, B., Ferron, J. M., Moeyaert, M., et al. (2017). Methodological quality of meta-analyses of single-case experimental studies. Research in Developmental Disabilities. Advance online publication. https://doi.org/10.1016/j.ridd.2017.12.016.

  • Jessel, J., Hanley, G. P., & Ghaemmaghami, M. (2016). Interview-informed synthesized contingency analyses: Thirty replications and reanalysis. Journal of Applied Behavior Analysis, 49, 576–595.

    Article  PubMed  Google Scholar 

  • Kaplan, B. A., Gilroy, S. P., Reed, D. D., Koffarnus, M. N., & Hursh, S. R. (2018). The R package beezdemand: Behavioral economic easy demand. Perspectives on Behavior Science. https://doi.org/10.1007/s40614-018-00187-7.

  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality & Social Psychology Review, 2, 196–217.

    Article  Google Scholar 

  • Kittelman, A., Gion, C., Horner, R. H., Levin, J. R., & Kratochwill, T. R. (2018). Establishing journalistic standards for the publication of negative results. Remedial & Special Education., 39, 171–176.

    Article  Google Scholar 

  • Koegel, R. L., & Koegel, L. K. (2006). Pivotal response treatments for autism: Communication, social, and academic development. Baltimore, MD: Paul H. Brookes.

    Google Scholar 

  • Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2013). Single-case intervention research design standards. Remedial & Special Education, 34, 26–38.

    Article  Google Scholar 

  • Kratochwill, T. R., Levin, J. R., & Horner, R. H. (2018). Negative results: Conceptual and methodological dimensions in single-case intervention research. Remedial & Special Education, 39, 67–76.

    Article  Google Scholar 

  • Lane, J. (2010). Let’s make science metrics more useful. Nature, 464, 488–489.

    Article  PubMed  Google Scholar 

  • Ledford, J. R., Barton, E. E., Hardy, J. K., Elam, K., Seabolt, J., Shanks, M., et al. (2016). What equivocal data from single case comparison studies reveal about evidence-based practices in early childhood special education. Journal of Early Intervention, 38, 79–91.

    Article  Google Scholar 

  • Lilienfeld, S. O. (2017). Psychology’s replication crisis and the grant culture: Righting the ship. Perspectives on Psychological Science, 12(4), 660–664. https://doi.org/10.1177/1745691616687745.

    Article  PubMed  Google Scholar 

  • Lonigan, C. J., Elbert, J. C., & Johnson, S. B. (1998). Empirically supported psychosocial interventions for children: An overview. Journal of Clinical Child Psychology, 27, 138–145. https://doi.org/10.1207/s15374424jccp2702_1.

    Article  PubMed  Google Scholar 

  • Losinski, M., Cook, K., Hirsch, S., & Sanders, S. (2017). The effects of deep pressure therapies and antecedent exercise on stereotypical behaviors of students with autism spectrum disorders. Behavioral Disorders, 42, 196–208.

    Article  Google Scholar 

  • Maggin, D. M., Talbott, E., Van Acker, E. Y., & Kumm, S. (2017). Quality indicators for systematic reviews in behavioral disorders. Behavioral Disorders, 42, 52–64.

    Article  Google Scholar 

  • Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy & Research, 1, 161–175.

    Article  Google Scholar 

  • Mahoney, M. J. (1994). Scientist as subject: The psychological imperative. Clinton Corners, NY: Percheron Press.

    Google Scholar 

  • Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43, 304–316.

    Article  Google Scholar 

  • Mitchell, M., Leachman, M., & Masterson, K. (2016). Funding down, tuition up: State Cuts to higher education threaten quality and affordability at public colleges. Washington, DC: Center on Budget & Policy Priorities Retrieved January 20, 2019, from http://www.cbpp.org/research/state-budget-and-tax/funding-down-tuition-up#_ftn33.

    Google Scholar 

  • National Autism Center. (2015). Findings and conclusions: National standards project, phase 2. Randolph, MA: Author.

    Google Scholar 

  • Normand, M. P., & Knoll, M. L. (2006). The effects of a stimulus-stimulus pairing procedure on the unprompted vocalizations of a young child diagnosed with autism. Analysis of Verbal Behavior, 22, 81–85.

    Article  PubMed  Google Scholar 

  • Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.

    Article  PubMed  Google Scholar 

  • Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.

    Article  Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.

    Article  Google Scholar 

  • Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531–536.

    Article  PubMed  Google Scholar 

  • Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.

    Article  PubMed  Google Scholar 

  • Perone, M. (2018). How I learned to stop worrying and love replication failures. Perspectives on Behavior Science. Advance online publication. https://doi.org/10.1007/s40614-018-0153-x.

  • Pierce, W. D., & Cheyney, C. D. (2017). Behavior analysis and learning: A biobehavioral approach. New York, NY: Routledge.

    Book  Google Scholar 

  • Quigley, S. P., Peterson, L., Frieder, J. E., & Peterson, S. (2011). Effects of a weighted vest on problem behaviors during functional analyses in children with pervasive developmental disorders. Research in Autism Spectrum Disorders, 5, 529–538.

    Article  Google Scholar 

  • Reichow, B. (2012). Overview of meta-analyses on early intensive behavioral intervention for young children with autism spectrum disorders. Journal of Autism & Developmental Disorders, 42, 512–520.

    Article  Google Scholar 

  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638.

    Article  Google Scholar 

  • Shillingsburg, M. A., Hollander, D. L., Yosick, R. N., Bowen, C., & Muskat, L. R. (2015). Stimulus-stimulus pairing to increase vocalizations in children with language delays: A review. Analysis of Verbal Behavior, 3, 215–235.

    Article  Google Scholar 

  • Shadish, W. R., Zelinsky, N. A., Vevea, J. L., & Kratochwill, T. R. (2016). A survey of publication practices of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis, 49, 656–673.

    Article  PubMed  Google Scholar 

  • Sham, E., & Smith, T. (2014). Publication bias in studies of an applied behavior-analytic intervention: An initial analysis. Journal of Applied Behavior Analysis, 47, 663–678.

    Article  PubMed  Google Scholar 

  • Sidman, M. (1960). Tactics of scientific research. Cambridge, MA: Cambridge Center for Behavioral Studies.

    Google Scholar 

  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  PubMed  Google Scholar 

  • Skinner, B. F. (1938). The behavior of organisms: an experimental analysis. Cambridge, MA: B. F. Skinner Foundation.

    Google Scholar 

  • Skinner, B. F. (1953). Science and human behavior. Cambridge, MA: B. F. Skinner Foundation.

    Google Scholar 

  • Slocum, T. A., Detrich, R., Wilczynski, S. M., Spencer, T. D., Lewis, T., & Wolfe, K. (2014). The evidence-based practice of applied behavior analysis. The Behavior Analyst, 37, 41–56.

    Article  PubMed  PubMed Central  Google Scholar 

  • Tincani, M., & De Mers, M. (2016). Meta-analysis of single-case research design studies on instructional pacing. Behavior Modification, 40, 799–824.

    Article  PubMed  Google Scholar 

  • Tincani, M., & Travers, J. C. (2018). Publishing single-case experimental research studies that do not demonstrate experimental control. Remedial & Special Education, 39, 118–128.

    Article  Google Scholar 

  • Travers, J. C., Tincani, M. J., & Lang, R. (2014). Facilitated communication denies people with disabilities their voice. Research & Practice in Severe Disabilities, 39, 195–202.

    Article  Google Scholar 

  • UNESCO. (2018). Global open access portal. Retrieved January 20, 2019, from http://www.unesco.org/new/en/communication-and-information/portals-and-platforms/goap/open-science-movement

  • Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A., Kucharczyk, S., et al. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism & Developmental Disorders, 45, 1951–1966.

    Article  Google Scholar 

  • Yoon, S. Y., & Bennett, G. M. (2000). Effects of a stimulus-stimulus pairing procedure on conditioning vocal sounds as reinforcers. Analysis of Verbal Behavior, 1, 75–88.

    Article  Google Scholar 

  • Zane, T., Davis, C., & Rosswurm, M. (2014). The cost of fad treatments in autism. Journal of Early & Intensive Behavior Intervention, 5, 44–51.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Teaching and Learning, Temple University, 1301 Cecil B. Moore Ave., Ritter Hall 351, Philadelphia, PA, 19122, USA

    Matt Tincani

  2. Department of Special Education, University of Kansas, Joseph R. Pearson Hall, Rm. 547, 1122 W. Campus Rd, Lawrence, KS, 66045-3101, USA

    Jason Travers

Authors
  1. Matt Tincani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jason Travers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matt Tincani.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tincani, M., Travers, J. Replication Research, Publication Bias, and Applied Behavior Analysis. Perspect Behav Sci 42, 59–75 (2019). https://doi.org/10.1007/s40614-019-00191-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40614-019-00191-5

Keywords

Search

Navigation