Abstract
Despite promising reports of broad cognitive benefit in studies of cognitive training, it has been argued that the reliance of many studies on no-intervention control groups (passive controls) make these reports difficult to interpret because placebo effects cannot be ruled out. Although researchers have recently been trying to incorporate more active controls, in which participants engage in an alternate intervention, previous work has been contentious as to whether this actually yields meaningfully different results. To better understand the influence of passive and active control groups on cognitive interventions, we conducted two meta-analyses to estimate their relative effect sizes. While the first one broadly surveyed the literature by compiling data from 34 meta-analyses, the second one synthesized data from 42 empirical studies that simultaneously employed both types of controls. Both analyses showed no meaningful performance difference between passive and active controls, suggesting that current active control placebo paradigms might not be appropriately designed to reliably capture these non-specific effects or that these effects are minimal in this literature.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that none of the clinical studies contained any healthy controls, so all population groups are indeed mutually exclusive.
References
Ackerman, P. L., Beier, M. E., & Boyle, M. O. (2005). Working memory and intelligence: the same or different constructs? Psychological Bulletin, 131(1), 30–60. https://doi.org/10.1037/0033-2909.131.1.30.
Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin & Review, 22(2), 366–377. https://doi.org/10.3758/s13423-014-0699-x.
Au, J., Buschkuehl, M., Duncan, G. J., & Jaeggi, S. M. (2016). There is no convincing evidence that working memory training is NOT effective: a reply to Melby-Lervåg and Hulme (2015). Psychonomic Bulletin & Review, 23(1), 331–337. https://doi.org/10.3758/s13423-015-0967-4.
Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128(4), 612–637.
Beecher, H. K. (1955). The powerful placebo. Journal of the American Medical Association, 159(17), 1602–1606.
Benedetti, F., Mayberg, H. S., Wager, T. D., Stohler, C. S., & Zubieta, J.-K. (2005). Neurobiological mechanisms of the placebo effect. Journal of Neuroscience, 25(45), 10390–10402. https://doi.org/10.1523/JNEUROSCI.3458-05.2005.
Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M., & Gratton, G. (2008). The effects of video game playing on attention, memory, and executive control. Acta Psychologica, 129(3), 387–398. https://doi.org/10.1016/j.actpsy.2008.09.005.
Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445–454. https://doi.org/10.1177/1745691613491271.
Borenstein, M. (2009). Introduction to meta-analysis. Chichester: Wiley.
Borenstein, M., Higgins, J., & Rothstein, H. (2005). Comprehensive meta-analysis version 2 (version 2). Englewood: Biostat.
Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230.
Buschkuehl, M., Hernandez-Garcia, L., Jaeggi, S. M., Bernard, J. A., & Jonides, J. (2014). Neural effects of short-term training on working memory. Cognitive, Affective, & Behavioral Neuroscience, 14(1), 147–160. https://doi.org/10.3758/s13415-013-0244-9.
Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014). Intrinsic motivation and extrinsic incentives jointly predict performance: a 40-year meta-analysis. Psychological Bulletin, 140(4), 980–1008. https://doi.org/10.1037/a0035661.
Chein, J. M., & Morrison, A. B. (2010). Expanding the mind’s workspace: training and transfer effects with a complex working memory span task. Psychonomic Bulletin & Review, 17(2), 193–199. https://doi.org/10.3758/PBR.17.2.193.
Cleophas, T. J., & Zwinderman, A. H. (2017). Meta-meta-analysis. In T. J. Cleophas & A. H. Zwinderman (Eds.), Modern meta-analysis: review and update of methodologies (pp. 135–143). https://doi.org/10.1007/978-3-319-55895-0_11.
Dahlin, E., Neely, A. S., Larsson, A., Bäckman, L., & Nyberg, L. (2008). Transfer of learning after updating training mediated by the striatum. Science, 320(5882), 1510–1512. https://doi.org/10.1126/science.1155466.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629.
Foroughi, C. K., Monfort, S. S., Paczynski, M., McKnight, P. E., & Greenwood, P. M. (2016). Placebo effects in cognitive training. Proceedings of the National Academy of Sciences, 113(27), 7470–7474. https://doi.org/10.1073/pnas.1601243113.
Green, M. W., Taylor, M. A., Elliman, N. A., & Rhodes, O. (2001). Placebo expectancy effects in the relationship between glucose and cognition. British Journal of Nutrition, 86(02), 173. https://doi.org/10.1079/BJN2001398.
Green, C. S., Bavelier, D., Kramer, A. F., Vinogradov, S., Ansorge, U., Ball, K. K., et al. (2019). Improving methodological standards in behavioral interventions for cognitive enhancement. Journal of Cognitive Enhancement. https://doi.org/10.1007/s41465-018-0115-y.
Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the placebo powerless? The New England Journal of Medicine, 2001(344), 1594–1602.
Hróbjartsson, A., & Gøtzsche, P. C. (2004). Is the placebo powerless? Update of a systematic review with 52 new randomized trials comparing placebo with no treatment. Journal of Internal Medicine, 256(2), 91–100.
Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. The Cochrane Library. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003974.pub3/full. Accessed 1 June 2017.
Hsu, N. S., Buschkuehl, M., Jonides, J., & Jaeggi, S. M. (2013). Potential mechanisms underlying working memory training and transfer. Presented at the Psychonomic Society Annual Meeting. Toronto: Ontario.
Hsu, N. S., Jaeggi, S. M., & Novick, J. M. (2017). A common neural hub resolves syntactic and non-syntactic conflict through cooperation with task-specific networks. Brain and Language, 166, 63–77. https://doi.org/10.1016/j.bandl.2016.12.006.
Hussey, E. K., Harbison, J. I., Teubner-Rhodes, S. E., Mishler, A., Velnoskey, K., & Novick, J. M. (2017). Memory and language improvements following cognitive control training. Journal of Experimental Psychology. Learning, Memory, and Cognition, 43(1), 23–58. https://doi.org/10.1037/xlm0000283.
Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences, 105(19), 6829–6833. https://doi.org/10.1073/pnas.0801268105.
Jaeggi, S. M., Buschkuehl, M., Shah, P., & Jonides, J. (2014). The role of individual differences in cognitive training and transfer. Memory & Cognition, 42(3), 464–480. https://doi.org/10.3758/s13421-013-0364-z.
Jarosz, A. F., & Wiley, J. (2014). What are the odds? A practical guide to computing and reporting Bayes factors. The Journal of Problem Solving, 7(1). https://doi.org/10.7771/1932-6246.1167.
Katz, B., Jaeggi, S. M., Buschkuehl, M., Shah, P., & Jonides, J. (2018). The effect of monetary compensation on cognitive training outcomes. Learning and Motivation, 63, 77–90. https://doi.org/10.1016/j.lmot.2017.12.002.
Kirsch, I. (2005). Placebo psychotherapy: synonym or oxymoron? Journal of Clinical Psychology, 61(7), 791–803. https://doi.org/10.1002/jclp.20126.
Klauer, K. J., & Phye, G. D. (2008). Inductive reasoning: a training approach. Review of Educational Research, 78(1), 85–123. https://doi.org/10.3102/0034654307313402.
Lee, C. H., Cook, S., Lee, J. S., & Han, B. (2016). Comparison of two meta-analysis methods: inverse-variance-weighted average and weighted sum of Z-scores. Genomics & Informatics, 14(4), 173–180. https://doi.org/10.5808/GI.2016.14.4.173.
Li, D., Christ, S. E., & Cowan, N. (2014). Domain-general and domain-specific functional networks in working memory. NeuroImage, 102(02), 646–656. https://doi.org/10.1016/j.neuroimage.2014.08.028.
Looby, A., & Earleywine, M. (2011). Expectation to receive methylphenidate enhances subjective arousal but not cognitive performance. Experimental and Clinical Psychopharmacology, 19(6), 433–444. https://doi.org/10.1037/a0025252.
Melby-Lervåg, M., & Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Developmental Psychology, 49(2), 270–291. https://doi.org/10.1037/a0028228.
Melby-Lervåg, M., & Hulme, C. (2016). There is no convincing evidence that working memory training is effective: a reply to Au et al. (2014) and Karbach and Verhaeghen (2014). Psychonomic Bulletin & Review, 23(1), 324–330. https://doi.org/10.3758/s13423-015-0862-z.
Melby-Lervåg, M., Redick, T. S., & Hulme, C. (2016). Working memory training does not improve performance on measures of intelligence or other measures of “far transfer” evidence from a meta-analytic review. Perspectives on Psychological Science, 11(4), 512–534.
Moher, D. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Annals of Internal Medicine, 151(4), 264. https://doi.org/10.7326/0003-4819-151-4-200908180-00135.
Morey, D., Rouder, J. N., & Jamil, T. (2014). Bayes factor: computation of Bayes factors for common designs (R package version 0.9.8). Retrieved from http://CRAN.R-project.org/package=BayesFactor. Accessed 1 Aug 2017.
Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364–386. https://doi.org/10.1177/1094428106291059.
Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: investigating participant demand characteristics. The Journal of general psychology, 135(2), 151-166.
Oelhafen, S., Nikolaidis, A., Padovani, T., Blaser, D., Koenig, T., & Perrig, W. J. (2013). Increased parietal activity after training of interference control. Neuropsychologia, 51(13), 2781–2790. https://doi.org/10.1016/j.neuropsychologia.2013.08.012.
Opitz, B., Schneiders, J. A., Krick, C. M., & Mecklinger, A. (2014). Selective transfer of visual working memory training on Chinese character learning. Neuropsychologia, 53, 1–11. https://doi.org/10.1016/j.neuropsychologia.2013.10.017.
Pahor, A., Jaeggi, S. M., & Seitz, A. R. (2018). Brain training. In ELS (pp. 1–9). https://doi.org/10.1002/9780470015902.a0028037.
Price, D. D., Finniss, D. G., & Benedetti, F. (2008). A comprehensive review of the placebo effect: recent advances and current thought. Annual Review of Psychology, 59(1), 565–590. https://doi.org/10.1146/annurev.psych.59.113006.095941.
R Core Team. (2013). R: a language and environment for statistical computing. Retrieved from http://www.R-project.org/
Rass, O., Forsyth, J. K., Bolbecker, A. R., Hetrick, W. P., Breier, A., Lysaker, P. H., & O’Donnell, B. F. (2012). Computer-assisted cognitive remediation for schizophrenia: a randomized single-blind pilot study. Schizophrenia Research, 139(1–3), 92–98. https://doi.org/10.1016/j.schres.2012.05.016.
Rebok, G. (2015). Selecting control groups: to what should we compare behavioral interventions? In L. N. Gitlin & S. J. Czaja (Eds.), Behavioral intervention research: designing, evaluating, and implementing (pp. 139–160). New York, NY: Springer Publishing Company.
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011). Interpretation of random effects meta-analyses. BMJ, 342. https://doi.org/10.1136/bmj.d549.
Rohatgi, A. (2017). WebPlotDigitizer Version 3.12 (Version 3.12). Retrieved from http://arohatgi.info/WebPlotDigitizer. Accessed 1 Dec 2016.
Rosenthal, R. (1991). Meta-analytic procedures for social research (1st ed.). Newbury Park: SAGE Publications, Inc..
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. https://doi.org/10.3758/PBR.16.2.225.
Salmi, J., Nyberg, L., & Laine, M. (2018). Working memory training mostly engages general-purpose large-scale networks for learning. Neuroscience and Biobehavioral Reviews, 93, 108–122. https://doi.org/10.1016/j.neubiorev.2018.03.019.
Schmiedek, F., Lövdén, M., & Lindenberger, U. (2010). Hundred days of cognitive training enhance broad cognitive abilities in adulthood: findings from the COGITO Study. Frontiers in Aging Neuroscience, 2. https://doi.org/10.3389/fnagi.2010.00027.
Schneiders, J. A., Opitz, B., Krick, C. M., & Mecklinger, A. (2011). Separating intra-modal and across-modal training effects in visual working memory: an fMRI investigation. Cerebral Cortex, 21(11), 2555–2564. https://doi.org/10.1093/cercor/bhr037.
Schwarz, K., & Büchel, C. (2015). Cognition and the placebo effect—dissociating subjective perception and actual performance. PLoS One, 10(7), 1–12. https://doi.org/10.1371/journal.pone.0130492.
Schweizer, K. (2007). Investigating the relationship of working memory tasks and fluid intelligence tests by means of the fixed-links model in considering the impurity problem. Intelligence, 35(6), 591–604. https://doi.org/10.1016/j.intell.2006.11.004.
Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138(4), 628–654. https://doi.org/10.1037/a0027473.
Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. L. (2016). Do “brain-training” programs work? Psychological Science in the Public Interest, 17(3), 103–186. https://doi.org/10.1177/1529100616661983.
Smith, S. P., Stibric, M., & Smithson, D. (2013). Exploring the effectiveness of commercial and custom-built games for cognitive training. Computers in Human Behavior, 29(6), 2388–2393. https://doi.org/10.1016/j.chb.2013.05.014.
Soveri, A., Antfolk, J., Karlsson, L., Salo, B., & Laine, M. (2017). Working memory training revisited: a multi-level meta-analysis of n-back training studies. Psychonomic Bulletin & Review, 1–20. https://doi.org/10.3758/s13423-016-1217-0.
Stephenson, C. L., & Halpern, D. F. (2013). Improved matrix reasoning is limited to training on tasks with a visuospatial component. Intelligence, 41(5), 341–357. https://doi.org/10.1016/j.intell.2013.05.006.
Thompson, T. W., Waskom, M. L., Garel, K.-L. A., Cardenas-Iniguez, C., Reynolds, G. O., Winter, R., et al. (2013). Failure of working memory training to enhance cognition or intelligence. PLoS One, 8(5), e63614. https://doi.org/10.1371/journal.pone.0063614.
Thorell, L. B., Lindqvist, S., Bergman Nutley, S., Bohlin, G., & Klingberg, T. (2009). Training and transfer effects of executive functions in preschool children. Developmental Science, 12(1), 106–113. https://doi.org/10.1111/j.1467-7687.2008.00745.x.
Tsai, N., Buschkuehl, M., Kamarsu, S., Shah, P., Jonides, J., & Jaeggi, S. M. (2018). (Un)great expectations: the role of placebo effects in cognitive training. Journal of Applied Research in Memory and Cognition. https://doi.org/10.1016/j.jarmac.2018.06.001.
Vartanian, O., Coady, L., & Blackler, K. (2016). 3D multiple object tracking boosts working memory span: implications for cognitive training in military populations. Military Psychology, 28(5), 353–360. https://doi.org/10.1037/mil0000125.
Wager, T. D., & Atlas, L. Y. (2015). The neuroscience of placebo effects: connecting context, learning and health. Nature Reviews. Neuroscience, 16(7), 403–418. https://doi.org/10.1038/nrn3976.
Weicker, J., Villringer, A., & Thöne-Otto, A. (2016). Can impaired working memory functioning be improved by training? A meta-analysis with a special focus on brain injured patients. Neuropsychology, 30(2), 190–212. https://doi.org/10.1037/neu0000227.
Willis, S. L. (2001). Methodological issues in behavioral intervention research with the elderly. In Handbook of the psychology of aging (5th ed., pp. 78–108). San Diego: Academic Press.
Funding
This work was supported in part by the National Science Foundation Graduate Research Fellowship, Grant No. DGE-1321846 (JA) and the National Institute of Health/National Institute on Aging, Grant Nos. 1R01MH111742 and 1K02AG054665 (SMJ).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
MB and KB are employed at the MIND Research Institute, whose interests are related to this work. SMJ has an indirect financial interest in the MIND Research Institute. No other authors declare any competing financial interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Au, J., Gibson, B.C., Bunarjo, K. et al. Quantifying the Difference Between Active and Passive Control Groups in Cognitive Interventions Using Two Meta-analytical Approaches. J Cogn Enhanc 4, 192–210 (2020). https://doi.org/10.1007/s41465-020-00164-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41465-020-00164-6