A general factor of intelligence fails to account for changes in tests’ scores after cognitive practice: A longitudinal multi-group latent-variable study

Highlights

• 477 participants completed twice a set of standardized intelligence tests.

• Participants were randomly assigned to three groups.

• The three groups showed remarkable improvements in tests’ scores.

• Longitudinal multi-group latent variable analyses were computed.

• The tapped common latent factor fails to account for the improvements.

Abstract

As a general rule, the repeated administration of tests measuring a given cognitive ability in the same participants reveals increased scores. This brings to life the well-known practice effect and it must be taken into account in research aimed at the proper assessment of changes after the completion of cognitive training programs. Here we focus in one specific research question: Are changes in test scores accounted for by the tapped underlying cognitive construct/factor? The evaluation of the factor of interest by several measures is required for that purpose. 477 university students completed twice a battery of four heterogeneous standardized intelligence tests within a time lapse of four weeks. Between the pre-test and the post-test sessions, some participants completed eighteen practice sessions based on memory span tasks, other participants completed eighteen practice sessions based on processing speed tasks, and a third group of participants did nothing between testing sessions. The three groups showed remarkable changes in test scores from the pre-test to the post-test intelligence session. However, results from multi-group longitudinal latent variable analyses revealed that the identified latent factor tapped by the specific intelligence measures fails to account for the observed changes.

Introduction

Practice effects are broadly acknowledged in the cognitive abilities literature (Anastasi, 1934, Colom et al., 2010, Hunt, 2011, Jensen, 1980, Reeve and Lam, 2005). When the same individuals complete the same (or parallel) standardized tests, their scores show remarkable improvements. However, as discussed by Jensen (1998) among others (Colom et al., 2002, Colom et al., 2006, te Nijenhuis et al., 2007), specific measures tap cognitive abilities at three levels: general ability (such as the general factor of intelligence, or g), group abilities (such as verbal or spatial ability), and concrete skills required by the measure (such as vocabulary or mental rotation of 2D objects).

Within this general framework, recent research aimed at testing changes after the completion of cognitive training programs has produced heated discussions regarding the nature of the changes observed in the measures administered before and after the training regime (Buschkuehl and Jaeggi, 2010, Conway and Getz, 2010, Haier, 2014, Moody, 2009, Shipstead et al., 2010, Shipstead et al., 2012, Tidwell et al., 2013). The changes may or may not be accounted for by the underlying construct of interest. Thus, for instance, the pioneering work by Jaeggi, Buschkuehl, Jonides, and Perrig (2008) observed changes in fluid intelligence measures after completion of a challenging cognitive training program based on the dual n-back task. This report stimulated a number of investigations aimed at replicating the finding (Buschkuehl et al., 2014, Colom et al., 2013, Chooi and Thompson, 2012, Harrison et al., 2013, Jaušovec and Jaušovec, 2012, Redick et al., 2012, Rudebeck et al., 2012, Shipstead et al., 2012, Stephenson and Halpern, 2013, von Bastian and Oberauer, 2013).

The meta-analysis published by Melby-Lervåg and Hulme (2012) concluded that short-term cognitive training fails to improve performance on far-transfer measures. Nevertheless, their results support the conclusion that the completed programs might improve performance on near-transfer measures, meaning that specific cognitive skills seem sensitive to training. The meta-analysis by Au et al. (2014), focused on reports analyzing the effect of cognitive training programs based on the n-back task, showed a positive, albeit small, impact on fluid intelligence measures. The weighted average effect size was .24, which is equivalent to 3.6 IQ points. The authors suggested that even these small increments might impact performance on real-life settings (see also Herrnstein & Murray, 1994 for a similar argument).

Importantly, it has been proposed that the latter statement may be relevant if and only if observed increments in test scores are accounted for by the tapped latent factor representing the construct of interest. In this regard, te Nijenhuis et al. (2007) reported a meta-analysis of sixty-four studies using a test-retest design, finding a perfect negative correlation between the vectors defined by tests’ scores changes and the g loadings of these tests. Their main conclusion was that observed improvements on scores were test-specific and unrelated to the general factor of intelligence (g). However, this study was based on the method of correlated vectors, which has been questioned on several grounds (Ashton & Lee, 2005). Latent-variable analyses are more robust and, therefore, may provide better answers to the question of interest (Dolan and Hamaker, 2001, Haier, 2014).

These latent-variable analyses require appropriate sample sizes. Published research reports analyzing changes after completion of cognitive training programs consider small samples on a regular basis. Thus, for instance, the meta-analysis by Au et al. (2014) is based on reports with sample sizes ranging from 3 to 30 participants (see their Table 1) and this precludes the application of the recommended latent-variable analyses.

To help fill this gap, here we report a study considering a large number of participants (477). All these participants completed screen versions of four heterogeneous standardized intelligence tests on two occasions, separated by four weeks. Participants were randomly assigned to three groups comprising more than one hundred participants each. Between the pre-test and post-test sessions, the first group completed 18 practice sessions based on memory span tasks, the second group completed 18 practice sessions based on processing speed tasks, whereas the third group did nothing. These three groups were systematically compared using multi-group longitudinal latent-variable analyses in order to examine the main research question, namely, are changes in tests’ scores from a pre-test to a post-test intelligence session accounted for by the tapped latent trait?