Abstract
Cognitive training research faces a number of methodological challenges. Some of these are general to evaluation studies of behavioral interventions, like selection effects that confound the comparison of treatment and control groups with preexisting differences in participants’ characteristics. Some challenges are also specific to cognitive training research, like the difficulty to tell improvements in general cognitive abilities from improvements in rather task-specific skills. Here, an overview of most important challenges is provided along an established typology of different kinds of validity (statistical conclusion, internal, external, and construct validity) that serve as central criteria for evaluating intervention studies. Besides standard approaches to ensure validity, like using randomized assignment to experimental conditions, emphasis is put on design elements that can help to raise the construct validity of the treatment (like adding active control groups) and of the outcome measures (like using latent factors based on measurement models). These considerations regarding study design are complemented with an overview of data-analytical approaches based on structural equation modeling, which have a number of advantages in comparison to the still predominant approaches based on analysis of variance.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Boot, W. R., & Simons, D. J. (2012). Advances in video game methods and reporting practices (but still room for improvement): A commentary on Strobach, Frensch, and Schubert (2012). Acta Psychologica, 141, 276–277. doi:10.1016/j.actpsy.2012.06.011.
Fiedler, K. (2011). Voodoo correlations are everywhere—Not only in neuroscience. Perspectives on Psychological Science, 6, 163–171. doi:10.1177/1745691611400237.
Green, C. S., Strobach, T., & Schubert, T. (2014). On methodological standards in training and transfer experiments. Psychological Research, 78, 756–772. doi:10.1007/s00426-013-0535-3.
Guo, S., & Fraser, W. M. (2014). Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: Sage.
Hayes, T. R., Petrov, A. A., & Sederberg, P. B. (2015). Do we really become smarter when our fluid-intelligence test scores improve? Intelligence, 48, 1–14. Retrieved from http://dx.doi.org/10.1016/j.intell.2014.10.005.
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.
Lövdén, M., Bäckman, L., Lindenberger, U., Schaefer, S., & Schmiedek, F. (2010). A theoretical framework for the study of adult cognitive plasticity. Psychological Bulletin, 136, 659–676. doi:10.1037/a0020080.
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 6, 487–498. Retrieved from http://dx.doi.org/10.1037/a0039400.
McArdle, J. J. (2009). Latent variable modelling of differences and changes with longitudinal data. Annual Review of Psychology, 60, 577–605. doi:10.1146/annurev.psych.60.110707.163612.
McArdle, J. J., & Prindle, J. J. (2008). A latent change score analysis of a randomized clinical trial in reasoning training. Psychology and Aging, 23, 702–719. doi:10.1037/a0014349.
Moody, D. E. (2009). Can intelligence be increased by training on a task of working memory? Intelligence, 37, 327–328. doi:10.1016/j.intell.2009.04.005.
Murnane, R. J., & Willett, J. B. (2010). Methods matter: Improving causal inference in educational and social science research. Oxford: Oxford University Press.
Noack, H., Lövdén, M., Schmiedek, F., & Lindenberger, U. (2009). Cognitive plasticity in adulthood and old age: Gauging the generality of cognitive intervention effects. Restorative Neurology and Neuroscience, 27, 435–453. doi:10.3233/RNN-2009-0496.
Noack, H., Lövdén, M., & Schmiedek, F. (2014). On the validity and generality of transfer effects in cognitive training research. Psychological Research, 78, 773–789. doi:10.1007/s00426-014-0564-6.
Redick, T. S. (2015). Working memory training and interpreting interactions in intelligence interventions. Intelligence, 50, 14–20. Retrieved from http://dx.doi.org/10.1016/j.intell.2015.01.014.
Rode, C., Robson, R., Purviance, A., Geary, D. C., & Mayr, U. (2014). Is working memory training effective? A study in a school setting. PLoS One, 9(8), e104796. doi:10.1371/journal.pone.0104796.
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177. doi:10.1037//1082-989X.7.2.147.
Schubert, T., & Strobach, T. (2012). Video game experience and optimized cognitive control skills—On false positives and false negatives: Reply to Boot and Simons (2012). Acta Psychologica, 141, 278–280. doi:10.1016/j.actpsy.2012.06.010.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for causal inference. Boston, MA: Houghton Mifflin.
Shipstead, Z., Hicks, K. L., & Engle, R. W. (2012a). Cogmed working memory training: Does the evidence support the claims? Journal of Applied Research in Memory and Cognition, 1, 185–193. doi:10.1016/j.jarmac.2012.06.003.x.
Shipstead, Z., Redick, T. S., & Engle, R. W. (2012b). Is working memory training effective? Psychological Bulletin, 138(4), 628–654. doi:10.1037/a0027473.
Tidwell, J. W., Dougherty, M. R., Chrabaszcz, J. R., Thomas, R. P., & Mendoza, J. L. (2014). What counts as evidence for working memory training? Problems with correlated gains and dichotomization. Psychonomic Bulletin & Review, 21, 620–628. doi:10.3758/s13423-013-0560-7.
Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70. doi:10.1177/109442810031002.
von Bastian, C. C., & Oberauer, K. (2014). Effects and mechanisms of working memory training: A review. Psychological Research, 78, 803–820. doi:10.1007/s00426-013-0524-6.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Schmiedek, F. (2016). Methods and Designs. In: Strobach, T., Karbach, J. (eds) Cognitive Training. Springer, Cham. https://doi.org/10.1007/978-3-319-42662-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-42662-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-42660-0
Online ISBN: 978-3-319-42662-4
eBook Packages: Behavioral Science and PsychologyBehavioral Science and Psychology (R0)