On the reliability of behavioral measures of cognitive control: retest reliability of task-inhibition effect, task-preparation effect, Stroop-like interference, and conflict adaptation effect

In the structural-equation modeling approach presented by Miyake et al. (2000), three subcomponents of cognitive control were identified as latent variables: “shifting of task-sets”, “monitoring and updating of working memory”, and “inhibition of prepotent response tendencies”. Later empirical work using structural equation modeling confirmed the task-shifting and working-memory factors, but not the “inhibition of prepotent responses” factor (see Friedman & Miyake, 2017; Karr et al., 2018; Miyake & Friedman, 2012, for reviews). Instead, Friedman et al. (2008) proposed a “common executive function” factor that partially overlaps with the task-shifting and working-memory factors. They describe this common factor as “the ability to maintain and manage goals, and use those goals to bias ongoing processing” (Friedman & Miyake, 2017, citation from section “5.1.1. Hypothesized functions for the Common EF factor”). The working-memory factor is characterized by the ability to update some of the current working-memory content, while at the same time maintaining other working-memory content for later retrieval. This factor is measured by memory tasks that require participants to attend to sequentially presented items from different categories, and later recall the last item from each category.

The task-shifting factor is described as the ability to rapidly replace task-sets in Friedman and Miyake’s (2017) framework, and the authors suggest that participants might differ in the speed of task-set replacement (see also Miyake & Friedman, 2012). The task-shifting factor is measured by cued task-switching paradigms, where the currently relevant task-set changes from trial to trial and is indicated by a task cue that is presented prior to the target stimulus (Meiran, 1996).

One popular measure that can be extracted from task-switching paradigms is the “task-switch cost”, defined as the performance difference between task-switch trials and task-repetition trials in a cued task-switching paradigm. For instance, Friedman and Miyake (2004) tested more than 200 participants with three cued task-switching paradigms, and found good reliability of task-switch costs (with Spearman-Brown corrected split-half reliabilities ranging from r = 0.43 to r = 0.82). The reliability of task-switch costs has been confirmed in several other studies (e.g., Friedman et al., 2008; Miyake et al., 2000; Paap et al., 2017; Pettigrew & Martin, 2016).

However, it is widely acknowledged that task-switch costs represent a mixture of different effects (see Kiesel et al., 2010; Koch et al., 2018, for reviews). One subcomponent of task-switch costs is task-level inhibition (e.g., Allport & Wylie, 1999; Goschke, 2000). Task-level inhibition can be measured with “N − 2 task repetition costs” (e.g., Mayr & Keele, 2000; see Koch et al., 2010, for a review), which is a sequential measure where different kinds of task-switching sequences are compared. For example, Gade and Koch (2005) used three tasks, and in each trial, the task was indicated by an explicit instruction cue. As stimuli, they used colored (red vs. blue) symbols (a digit or a letter) that varied in size (small vs. large), so that there were three varying perceptual dimensions, and the task cue indicated the relevant stimulus dimension for selecting the target attribute (e.g., small vs. large for the size dimension). When the authors analyzed the sequential transitions, they found that sequences of the ABA type (N − 2 task repetitions, e.g., color–size–color) resulted in worse performance (e.g., higher reaction time [RT]) than sequences of the CBA type (N − 2 switches, e.g. symbol-size-color). The finding of higher RT in the last trial of an ABA versus CBA task sequence speaks in favor of a process that inhibits aspects of the preceding task set when shifting to a new task set; this is because accounts in terms of persisting activation of previously established task representations (task sets) would predict better performance for ABA relative to CBA (Mayr & Keele, 2000; see Koch et al., 2018, for a recent discussion). Even though some other effects in task switching have been related to inhibitory processing, N − 2 repetition costs arguably represent the most unambiguous case for inhibition in task switching to date (Koch et al., 2010; and see also Grange et al., 2017, for a recent discussion). Yet, even though the experimental evidence for the existence (and replicability on the group level) of N − 2 repetition costs in task switching is very robust (i.e., they have been replicated many times with different paradigms and in different participant samples, see Koch et al., 2018, for a recent review), only few studies examined its split-half and retest reliability.

To our knowledge, three studies so far have assessed split-half reliability of N − 2 repetition costs. Kowalczyk and Grange (2017) used three different versions of task switching and found split-half reliabilities of N − 2 repetition costs between r = 0.37 and r = 0.60 (these are corrected reliability scores; note that split-half reliability is usually corrected for attenuation by applying the Spearman–Brown correction). Pettigrew and Martin (2016) reported a split-half reliability of N − 2 repetition costs of r = 0.44, and Rey-Mermet et al. (2018) of r = 0.27. One study assessed test–retest reliability of N − 2 repetition costs in both a task-switching and a language-switching paradigm, with about one week between test and retest (Timmer et al., 2018). These authors observed a retest reliability of N − 2 repetition costs of r ≈ 0.40 (both in the task-switching and the language-switching paradigm). Taken together, the available data on the reliability of N − 2 repetition costs is scarce and ranging from poor to moderate reliabilities.

Apart from task-switch costs and N − 2 repetition costs, another important cognitive-control measure that can be assessed in cued task-switching paradigms is the time-based task-preparation effect. Here we define this effect as the performance difference between trials with short versus long time intervals between task cue and task-specific stimulus (cue-stimulus interval, CSI). For instance, Lawo et al. (2012) observed substantial task-preparation effects that differed between younger and older adults, suggesting that task-preparation ability deteriorates with older age (on a group level). Other aging and developmental studies confirm that the efficiency of task preparation is an important aspect when assessing age-related differences in cognitive control (e.g., Cepeda et al., 2001; Crone et al., 2006; Schuch, 2016; Schuch & Konrad, 2017; Wild-Wall et al., 2007; for reviews, see Gajewski et al., 2018; Kray & Doerrenbaecher, in press; Kray & Ferdinand, 2014). Assuming that the relevant task set becomes activated during the CSI, the performance difference between short and long CSI conditions can be interpreted as reflecting the degree of cue-based activation of the relevant task-set, especially in N − 2 repetition cost paradigms where usually every trial is a task switch (e.g., Lawo et al., 2012; Schuch & Grange, 2019; Schuch & Koch, 2003). It is often assumed that task preparation involves activation of the relevant attentional settings and task rules in working memory and builds up gradually over time, such that a longer CSI leads to better task preparation (for reviews, see Kiesel et al., 2010; Koch et al., 2018).

Beyond the general task-preparation effect discussed here (i.e., the reduction of mean RT in trials with long vs. short CSI), considerable research has been carried out focusing on the specific task-preparation effect, denoting the reduction of task-switch costs with long vs. short CSI (see Kiesel et al., 2010; Koch et al., 2018, for reviews). The latter measure is often interpreted as a marker of “advance reconfiguration of task set” (Meiran, 1996; Monsell, 2003; Vandierendonck et al., 2010). Whether such a specific task-preparation effect also occurs with N − 2 task repetition costs to date is an unresolved issue. While earlier studies did not find a reduction of N − 2 task repetition costs with longer as compared to shorter CSI (e.g., Mayr & Keele, 2000; Schuch & Koch, 2003; see Koch et al., 2010, for review), more recent studies do sometimes report reduced N-2 repetition costs with longer task-preparation time (e.g., Gade & Koch, 2014; Scheil & Kleinsorge, 2014; Schuch & Grange, 2019). The design of the present Experiment 1 allowed us to contribute to this literature, by examining N − 2 repetition costs with short vs long CSI on a group level.

While CSI effects are well established on a group level, less attention has been paid to their reliability on the level of interindividual differences. Yet, the general task-preparation effect (i.e., performance improvement with long as compared to short CSI)—if it proves to be reliable—might be a good candidate for investigations of task switching processes from an interindividual-differences perspective. For instance, in the aging literature, age-related differences in task-preparation processes are widely discussed (e.g., Kray & Ferdinand, 2014, for review), but these studies typically compare task-preparation effects on a group level (i.e., comparing a group of younger adults with a group of older adults), such that reliability is usually not in the focus. Yet, the time-based task preparation effect may be suitable for correlational approaches, just as other behavioral indices of task preparation have been used in individual-differences studies (e.g., Wager et al., 2006). For instance, task-preparation effects related to the informativeness of the task cues have been correlated with electrophysiological and neuroimaging markers of task preparation (e.g., Brass & von Cramon, 2004; Karayanidis et al., 2009; see, e.g., Hsieh, 2012; Karayanidis et al., 2010, for reviews). Hence, assessing reliability of task-preparation measures in general, and of the time-based task-preparation effect in particular, might be useful for future investigations of cognitive control from an individual-differences perspective.

Regarding cognitive control measures in single-task context, perhaps the most popular effect is the color-word Stroop effect (i.e., saying the ink color of written color words that are either congruent or incongruent with the ink color they are presented in; see MacLeod, 1991; MacLeod & MacDonald, 2000, for reviews). The Stroop effect is a classic textbook example and popular classroom demonstration of a “conflict task”, where task-relevant and task-irrelevant features interfere, creating some kind of cognitive conflict (e.g., conflict between stimulus features, or conflict between competing responses). It is sometimes explained in terms of an inhibitory process, such as inhibition of distractor processing, or inhibition of inappropriate response tendency (e.g., Friedman & Miyake, 2017; Gärtner & Strobel, 2021; Miyake et al., 2000; Pettigrew & Martin, 2016). Others have argued that the Stroop effect and other conflict tasks do not necessarily reflect inhibitory control (e.g., Paap et al., 2020). Here, we will use the more descriptive terms “distractor interference control” or “control of cognitive conflict”. The Stroop effect has been reported to be quite reliable (with Spearman-Brown corrected split-half reliability often between r = 0.80 and r = 0.90, see, e.g., Friedman & Miyake, 2004; Rey-Mermet et al., 2018).

Interestingly, despite the high robustness of this experimental effect, when examining the sequential modulation of the Stroop effect (sequential congruency effect, e.g., Egner, 2007), which is typically used to examine conflict adaptation, the split-half reliability of this sequential measure has been found to be very poor, ranging between r =  − 0.12 and r = 0.08 across three experiments reported by Whitehead et al. (2019). This drop in reliability is at least partly due to the fact that the sequential congruency effect is computed as the difference of a difference score, and therefore has lower reliability than the congruency effect, which is computed as a simple difference score (see Kopp, 2011; Miller & Ulrich, 2013; Whitehead et al., 2019, for considerations on the reliability of difference scores).

While a considerable number of studies assessed split-half reliability of Stroop-like interference effects and task-switching effects, only few studies investigated the retest reliability of such effects. In one recent study, Hedge et al., (2018b) assessed retest reliability of a number of interference effects, including the Stroop effect, with a temporal separation of three weeks between test and retest. They found a retest reliability of r = 0.60 and r = 0.66 for the Stroop effects in two studies (they did not report retest reliability of the sequential congruency effect). In another study, Paap and Sawi (2016) examined retest reliability of effects in four different tasks, including task switching, over a period of one week and found only moderate reliabilities. For example, for color-shape switching, they found a retest reliability of r = 0.62. For the Simon task (which is often considered a conflict task, similar to the Stroop task), they found a retest reliability of only r = 0.43.

To summarize, several measures of cognitive control that are highly robust when analyzed on the group level in standard experimental paradigms have surprisingly low reliability when taken as a measure of interindividual differences in correlational approaches, for instance, in structural equation modeling. Therefore, more studies are needed that assess the split-half and retest reliability of standard cognitive control measures, to elucidate which of these measures are suitable for individual-differences approaches, and which are not.

In the present study, we assessed the reliability of four standard cognitive control measures. In Experiment 1, we focused on N − 2 repetition costs, which are a measure of task-level inhibition (see Koch et al., 2010, for review), and the time-based task-preparation effect (i.e., CSI effect, denoting the finding of improved performance with long as compared to short CSI), which may be considered as a marker of cue-based task-set activation (especially in paradigms with task switches only; e.g., Lawo et al., 2012; Schuch & Grange, 2019; Schuch & Koch, 2003). The design of Experiment 1 also allowed us to explore the potential preparatory modification of N − 2 repetition costs by task-preparation time (on a group level).

In Experiment 2, we examined a variant of the Stroop effect. The family of Stroop-like effects is a marker for distractor interference processing, and is sometimes taken as a marker for inhibitory processing; moreover, the sequential modulation of Stroop-like effects has been taken as a hallmark of conflict-triggered adjustments of cognitive control (Botvinick et al., 2001; see also Egner, 2007, 2017; Paap et al., 2019; Schuch et al., 2019, for more recent reviews). Here we used a face-name interference paradigm that resembles paradigms often used in the neuroimaging literature (e.g., Egner & Hirsch, 2005; Gazzaley et al., 2005; O’ Craven, et al., 1999), and has been used in our own lab before (Schuch & Koch, 2015; Schuch et al., 2017).

For these four measures of cognitive control, we report the group-level effects (i.e., the average effects across all participants), as well as their split-half and retest reliability. In both experiments, the respective effects were measured using standard experimental paradigms in a first and second session on the same day, which were separated by a short unrelated filler task. Then, participants performed the same experiment again in a second session (i.e., on the same day). We first report the group-level effects as obtained with a standard analysis of variance (ANOVA), with first vs. second session as an independent within-subjects variable. Then, we report split-half reliability (correlation between odd and even trials) and retest reliability (correlation between first and second session) for each of the effects.

To get reliable estimates for correlations, two issues are important: first, there needs to be a large enough number of participants—for instance, to reliably detect medium-sized correlations, a sample of N = 85 or larger is necessary (Cohen, 1992). With smaller sample sizes, correlation estimates are very variable (Schönbrodt & Perugini, 2013).

Second, and perhaps even more importantly, the number of experimental trials that provide the basis for computation of the experimental effects play a crucial role (Green et al., 2016; Rouder & Haaf, 2019). With small trial numbers, the estimates of the experimental effects are variable, which leads to attenuated correlations between the experimental effects from different conditions. One remedy to this issue is to apply the Spearman-Brown correction formula (Spearman, 1904), which corrects for a reduction of test length (i.e., of trial numbers in the case of experimental effects).¹ The estimates of split-half reliabilities of experimental effects are often Spearman–Brown corrected, to compensate for halving the “test length” by splitting trials into odd versus even trials. Note, however, that “test length” may vary considerably across experimental paradigms. When assessing, e.g., the Stroop effect, some researchers might use a paradigm with as little as 20 trials per condition, while others might use a different paradigm with, say, 100 trials per condition. Usually, researchers do not pay much attention to the number of trials that provide the basis for computing the experimental effect. Rouder and Haaf (2019) therefore suggested to calculate reliabilities of experimental effects for the case of infinitely large trial numbers. They did so by applying linear mixed models, and including trial-by-trial variability as an additional random factor in the model. They re-analyzed the data from Hedge et al., (2018b), and found retest reliabilities of around r = 0.70 for both Stroop and Flanker effect (as opposed to retest reliabilites of r = 0.55 and r = 0.50 when correlating the effects from first and second session without accounting for trial-by-trial variability). In a similar vein, Whitehead et al. (2020) re-analyzed data from Whitehead et al. (2019), and observed slightly larger split-half reliabilities for Stroop, Flanker, and Simon effects when using linear mixed models that account for trial-by-trial variability (split-half reliabilities ranging between r = 0.57 and r = 0.65) than when correlating the effects between odd and even trials without accounting for trial-by-trial variability (split-half reliabilities ranging between r = 0.31 and r = 0.61). Hence, it is important to always consider the number of trials per condition (or to extrapolate to the large-trial limit) when estimating split-half and test–retest reliabilities of experimental effects. Here, we considered the number of trials per condition when comparing reliability scores of different kinds (retest, split-half), and when comparing reliability measures across different studies.

The large-trial limit might be regarded as the “ideal case” for computing reliabilities; however, there are assets and drawbacks for designing experiments with large trial numbers. A potential disadvantage is that the longer the experiment, the more pronounced the influence of practice effects, and the more likely the cognitive tasks become highly overlearned and “automatized”. When investigating cognitive control functions, however, researchers might want to avoid too much automaticity and overlearning of task-specific associations or stimulus–response rules, as these cognitive processes might alter or even substitute the cognitive control processes the researcher is interested in (see, e.g., Grange & Juvina, 2015; Scheil, 2016, for practice effects on N − 2 repetition costs; Davidson et al., 2003, for practice effects on the Stroop effect in young versus old adults; Strobach et al., 2014, for review).

The first and second trial per block were excluded, because these could not be classified as ABA or CBA task sequence. Outliers were defined as trials with RT deviating more than three standard deviations from an individual’s overall mean RT, computed separately for first and second session, and were excluded as well (0.95% of the trials in the first session, 1.18% of the trials in the second session). Moreover, the two trials following an error were excluded, to eliminate influences of post-error processing. For RT analysis, error trials were excluded as well.

We will first discus the group-level effects as obtained in the ANOVAs, and then turn to the reliability analyses. This experiment showed very robust main effects of N − 2 task repetition and of CSI. That is, there were overall quite sizeable N − 2 task repetition costs of about 40 ms, and the general preparation effect was also very substantial and showed a RT benefit of 284 ms with long CSI relative to short CSI. These two effects replicate established basic effects. Moreover, we observed that N − 2 repetition costs became smaller with longer preparation time.

Effects of task-preparation time are of high theoretical relevance in task switching research. The finding of better performance with long than short CSI (i.e., main effect of CSI) is probably one of the most robust findings in task-switching research (see Kiesel et al., 2010), and can possibly be related to activation of the cued task representation.

The finding of smaller N − 2 repetition costs with long than short CSI confirms findings from Gade and Koch (2014, Experiments 1 and 2), Scheil and Kleinsorge (2014, Experiment 2), and Schuch and Grange (2019, Experiment 2), and extends them to conditions with blocked CSIs. Notably, other previous studies did not find any modulation of N-2 repetition costs by CSI, despite substantial general preparation effects (i.e., main effect of CSI but no interaction with N − 2 repetition costs; e.g., Lawo et al., 2012; Mayr & Keele, 2000; Prior, 2012; Schuch & Koch, 2003). Given that the effect size of the modulation of N − 2 repetition costs by CSI was rather small in the present study (η_p² = 0.11 for the interaction of CSI and N − 2 repetition costs), it is possible that larger sample sizes are needed to observe this effect (e.g., sample size was N = 96 in the present study, , as opposed to Ns between 16 and 24 in the studies that did not find the modulation).

The finding of smaller N − 2 repetition costs with long than short CSI suggests that part of the persisting inhibition of a previously abandoned task set can be overcome when preparation time for the upcoming task is longer. This finding resembles the finding of smaller task-switch costs (i.e., smaller difference between task-switch and task-repetition trials) with long than short CSI, which has been interpreted as a marker of “endogenous” reconfiguration of task set, and as such a hallmark of endogenous cognitive control (e.g., Monsell, 2003; see Kiesel et al., 2010; Vandierendonck et al., 2010, for discussions).

Moreover, the design with two sessions that were separated by a short break allowed us to examine practice effects. We observed that performance improved generally (i.e., RT decreased) from first to second session, and both CSI effect and N − 2 repetition costs became significantly smaller in the second session. The latter finding is in line with previous studies observing a reduction of N − 2 repetition costs with practice (Grange & Juvina, 2015; Scheil, 2016). Grange and Juvina (2015) tested a small number of participants (N = 9) who practised extensively (over five sessions with more than 1200 trials each); here, we observed practice effects in a sample more than 10 times larger (N = 96), where participants received relatively little practice (only two sessions with 480 experimental trials each).

The focus of the present study was on the reliability of the cognitive-control measures. Regarding N − 2 repetition costs, we observed an odd–even split-half reliability of r = 0.38 (r = 0.25 when excluding one outlying data point), and bootstrapped split-half reliability of r_median = 0.47, which is at the lower end of the split-half reliabilites reported in previous studies, ranging between r ≈ 0.30 and r ≈ 0.60 (Kowalczyk & Grange, 2017; Pettigrew & Martin, 2016; Rey-Mermet et al., 2018). Note that such reliability scores depend on the number of experimental trials that provide the basis for the correlation, and that most earlier studies reported corrected split-half reliability scores using the Spearman–Brown formula (which corrects for halving trial numbers when splitting them into odd and even trials). For instance, Kowalczyk and Grange (2017) reported corrected (r ≈ 0.50) and uncorrected (r ≈ 0.33) split-half reliability from N − 2 repetition cost paradigms comprising 480 trials in total. For a direct comparison, it is warranted to compare their corrected scores (which are based on 480 trials in total) to the present uncorrected scores (which are based on 960 trials in total); we thus observed a somewhat lower split-half reliability of N − 2 repetition costs (r ≈ 0.30 to r ≈ 0.45) compared to the study by Kowalczyk and Grange (2017), who observed r ≈ 0.50.

An important new result of the present experiment is that we also calculated retest reliability as the correlation of the respective effect in the first session with that in the second session. Even though sessions were separated by only 10 min, the retest reliability of N − 2 repetition costs was low, with r = 0.21 (r = 0.06 when excluding one outlying data point). Retest reliability of N − 2 repetition costs was significantly lower than split-half reliability, even though the same participants performed the identical experiment in Session 1 and 2 on the same day, and the sessions were only separated by 10 min. The only systematic difference between Session 1 and Session 2 are practice effects, with somewhat smaller N − 2 repetition costs in the second than first session.

The low retest reliability of N − 2 repetition costs suggests that this measure is not suitable as a measure of a stable, trait-like cognitive ability of task inhibition, confirming recent doubts about the stability of inhibition as a psychometric construct in studies of interindividual cognitive differences (see also Kowalczyk & Grange, 2017; Rey-Mermet et al., 2018). We will return to this issue in the General Discussion.

Other than N − 2 repetition costs, the CSI effect yielded a good split-half reliability (odd–even split-half reliability of r = 0.83, again, the uncorrected score is reported; bootstrapped split-half reliability of r_median = 0.94), and reasonable retest reliability (r = 0.62 when excluding one outlying data point). The split-half reliability estimates are well above the value of r = 0.70, which is often taken as a lower limit for acceptable split-half reliability (Cronbach, 1951). The retest reliability score suggests that the CSI effect is suitable as a measure of interindividual differences in cue-based task preparation in task switching.

The first trial of each block was excluded. Outliers were defined as trials with RT deviating more than three standard deviations from an individual’s overall mean RT, computed separately for Session 1 and 2 and were excluded as well (1.85% of the trials in Session 1, 1.86% in Session 2). To minimize episodic retrieval effects that might compromise the measurement of sequential congruency effects (Duthoo et al., 2014; Egner, 2007, 2017; Hommel et al., 2004; Mayr & Awh, 2009; Mayr et al., 2003; Whitehead et al., 2019), trials with repetitions of pictures and repetitions of names from trial N − 1 to trial N were excluded (9.25% of the trials). Moreover, trials immediately following an error were excluded to eliminate influences of post-error processing. For RT analysis, error trials were excluded as well.

In Experiment 2, we found the typical congruency effect and the sequential congruency effect that is known from the literature (see Egner, 2017; Duthoo et al., 2014) and we replicated our own previous findings using the same experimental Stroop-like paradigm (e.g., Schuch & Koch, 2015). The novel aspect of the present experiment was the calculation of split-half and retest reliability of these experimental effects.

Regarding the size of the reliability scores, previous studies have reported quite high reliability of the Stroop effect (e.g., Friedman & Miyake, 2004, using a color-word Stroop effect with vocal responses). In the present study, the split-half reliability of the Stroop-like effect was moderate at best (r = 0.34 and r_median = 0.27 in RT data, r = 0.53 and r_median = 0.51 in error data for odd–even and bootstrapped reliabilities, respectively). The finding of lower reliability, along with an overall smaller size of the Stroop-like effect, might be due to several reasons. First, we used manual instead of vocal responses, and the task required a categorization of the stimulus (i.e., categorizing the name as female or male) instead of an identification of stimulus (i.e., identifying the ink color). Second, we applied strict filtering of the data, excluding all trials that constituted partial feature repetitions with the preceding trial, to eliminate any trial-to-trial effects of episodic interference. This filtering procedure is especially important for the sequential congruency effect (to obtain a “pure” measure of conflict adaptation), but might also attenuate the size of the congruency effect.

Notably, when we restricted our reliability analyses to trials preceded by congruent trials (i.e., to those trials in which the congruency effect is larger), then we found a higher correlation (odd–even split-half reliability was r_c = 0.69 in RT data, r_c = 0.62 in error data), indicating that these trials are probably better suited to assess the split-half reliability of the Stroop effect (in RT data, the uncorrected split-half reliability scores were significantly larger for trials preceded by congruent vs incongruent trials; in error data, this difference did not reach statistical significance).

Retest reliability was low for the RT effect (r = 0.16), but was similar to split-half reliability for the error data (r = 0.45 in error data). When only trials after congruent trials were considered, retest reliability of the Stroop-like effect in error data was good (r_c = 0.71), suggesting that Stroop-like interference effects should be assessed in error data in addition to RT data, and should be restricted to trials after congruent trials when used for interindividual-difference approaches.

Compared to the Stroop-like congruency effect itself, the sequential modulation of the congruency effect did not show a significant retest reliability, confirming recent demonstrations of lacking reliability of sequential modulations of the “family of conflict effects”, such as the Stroop effect, the Simon effect, and the Flanker effect (see Whitehead et al., 2019, for a recent discussion). Note that we carefully controlled for episodic retrieval effects, excluding all trials with partial feature repetitions from analysis. This rather strict way of data filtering was applied to get a measure of conflict adaptation that is not (or only minimally) “contaminated” by episodic interference effects. This rather strict way of filtering leads to smaller congruency effects and sequential congruency effects (cf. Whitehead et al., 2019), which in turn might have contributed to the rather low reliability scores in the present experiment.

The finding of lacking reliability of sequential modulations of Stroop-like congruency effects has recently been thoroughly investigated by Whitehead et al. (2019). Our data confirm their findings and further extend them to lacking retest reliability (which does not come as a surprise given that split-half reliability was not significant in the first place).

The cue-based task preparation effect showed the highest reliability scores of the four cognitive-control measures investigated here, suggesting it is a good candidate for future investigations of cognitive control using correlational approaches. As the present task-switching paradigm consisted of task switches only, we suggest that the task-preparation effect as measured here may be taken as a measure of participants’ efficiency of cue-based activation of task set.

Another important result of the present study is the lacking retest reliability of N − 2 task repetition costs in Experiment 1, while at the same time the N − 2 repetition costs were highly robust as an experimental effect in both sessions. This might be a case of the “reliability paradox” as discussed by Hedge et al., (2018b), who argued that very strong effects at the level of group means are often counterintuitively less reliable at the level of interindividual differences, because if all participants display an effect of similar size (i.e. in very homogeneous groups), already small intraindividual variations can substantially attenuate the correlation across participants (see also Paap & Sawi, 2016, for a discussion).

One possibility is that inhibitory control is not stable within an individual, but is highly state-dependent, and is applied in a context-sensitive way (cf. Rey-Mermet et al., 2018). If so, it would be quite plausible to assume a substantial fluctuation of inhibitory control across trials. The case could be similar as with sequential fluctuation of Stroop-like effects: Stroop-like effects are usually more pronounced after no-conflict (i.e., congruent) than after conflict (i.e., incongruent) trials, which might reflect trial-to-trial fluctuations of attentional selectivity. Specifically, after no-conflict trials, the cognitive system is in a state of unfocused attention, so that the irrelevant stimulus dimension is processed up to the level of response activation, which in the case of incongruent stimuli results in a response conflict and triggers an upregulation of selective attention. In contrast, after conflict trials, the system is in a state of highly selective attention, so that no irrelevant response is activated, hence there is no response conflict, and no further upregulation of selective attention occurs. That is, for Stroop-like effects, the demand for an upregulation of cognitive control differs greatly across trials depending on the conflict level of the preceding trial. A similar scenario is conceivable for task-level inhibition: The demand for inhibitory task-level control might depend on the degree of task conflict in a particular trial. There is evidence that the amount of task-level inhibition (as measured by the size of N − 2 repetition costs) depends on the degree of task-set competition. Conditions with low task-set competition, such as when previous task-set activation has decayed (Gade & Koch, 2005) or when the current task set has been very well prepared (e.g., Scheil & Kleinsorge, 2014), have been shown to attenuate N − 2 task repetition costs. That is, fluctuations of activation level of the current task and of competing tasks jointly determine the degree of inhibitory control. In that sense, inhibition is not a process that is mandatory in every trial and always to the same degree but is a highly context-dependent, adaptive process. Hence, if this “inhibition on demand” account were correct, we would expect low intraindividual stability, which is reflected in low reliability, even though, when aggregated across many trials, there is clear evidence for such an effect at the individual and group level. This “inhibition on demand” idea could explain the discrepancy between reliability measures and group-level effects; however, it cannot account for the observed discrepancy between moderate split-half reliability and near-zero retest reliability of N − 2 repetition costs observed in the present study.

Of the four cognitive-control measures investigated here, the largest effect (i.e., the task-preparation effect) showed the best reliability scores. This might be due to methodological reasons. First, the higher reliability of the task-preparation effect as compared to that of the other effects might be partly due to its larger effect size. For instance, in the present Experiment 1, the overall mean CSI effect was 284 ms, with a standard deviation [SD] of 98 ms, corresponding to Cohen’s d effect size of 2.89; N − 2 repetition costs, on the other hand, were 43 ms, with SD of 37 ms, corresponding to Cohen’s d of 1.14. When only looking at Session 1, the variability of the CSI effect across participants (SD = 112 ms in Session 1) relative to its size (mean CSI effect = 305 ms in Session 1) is smaller than the variability of N − 2 repetition costs across participants (SD = 47 ms in Session 1) relative to their size (mean N − 2 repetition costs = 50 ms in Session 1). Therefore, the correlation of the CSI effect between Sessions 1 and 2 can be higher than the correlation of N − 2 repetition costs between Sessions 1 and 2.

Second, apart from effect size, the absolute size of an effect may also play a role for reliability. For instance, the CSI effect observed in the present study amounted to about 280 ms, whereas N − 2 repetition costs were in the order of magnitude of 40 ms. The larger absolute size of an effect could contribute to its reliability scores. This is because the unsystematic measurement error inherent to computer hardware and software constitutes a lower limit to the reliability of cognitive effects that are small in absolute size (e.g., 50 ms or smaller). The unsystematic error of a single-trial RT measurement depends on hardware and software settings, and may be in the order of 20 ms or larger (e.g., Plant & Turner, 2009; Plant et al., 2004). This means that also the average scores per condition, and the difference scores between conditions, are subject to unsystematic measurement error (the more trials are included for averaging, the smaller the unsystematic error). The measurement error is also reflected in the variability of the difference scores across participants. This constitutes a lower limit to the reliability with which smaller effects can be measured (such as N − 2 repetition costs, which were about 40 ms in the present experiment). For larger effects (such as the CSI effect, which was about 300 ms in the present experiment, or task-switch costs, which are often around 200 ms to 300 ms), the variability due to technical timing inaccuracy is proportionally smaller (given the same number of trials included for averaging), such that effects that are larger in absolute size (e.g., 200 ms or larger) can be measured with higher reliability. Consistent with this reasoning, for instance, Whitehead et al. (2019) observed good reliability of the post-error slowing effect (an effect of around 100 ms, 160 ms, and 300 ms in their Experiments 1, 2, and 3, respectively), but poor reliability of the sequential congruency effect (which was about 10 ms in their Experiments 1 and 2, and about 14 ms in their Experiment 3). Likewise, Friedman and Miyake (2004) observed good split-half reliability of large effects (e.g., task-switch costs that were in the range of 200–500 ms), moderate reliability of smaller effects (e.g., residual switch costs of their category switching paradigm that were about 70 ms), and poor reliability of very small effects (e.g., negative priming effects that were in the range of 3–8 ms). Future research would need to investigate systematically the relationship between the absolute size of an effect and the maximum reliability that can be achieved for this effect.

Another issue that complicates the measurement of trait-like cognitive control abilities are recent observations that difference scores in cognitive-control tasks are not “pure” measures of control (e.g., of conflict processing), but also reflect settings of individual speed-accuracy tradeoff (SAT) and general processing speed (Hedge et al., 2018a, 2021).

Using computer simulations and computational modeling, Hedge and colleagues (2018a, 2021) demonstrated that correlations between different cognitive-control measures (e.g., between Stroop effect and Flanker effect) can be observed even if there are no correlations between the model parameters reflecting conflict processing, but only correlations between model parameters reflecting SAT setting and/or general processing speed. Moreover, if there are correlations between the conflict-related model parameters across the different tasks, this does not necessarily lead to correlations between the behavioral difference scores in the two tasks. These findings call into question the widespread assumption in the literature that computing difference scores would “cancel out” interindividual differences in SAT settings and processing speed.

Together, the data suggest that all four cognitive control measures investigated here (task-inhibition effect, task-preparation effect, Stroop-like effect, and conflict adaptation effect) are well-suited for assessing group-level effects of cognitive control. Yet, except for the task-preparation effect, these measures do not seem suitable for reliably assessing interindividual differences in the strength of cognitive control. Therefore, they do not seem suited for correlational approaches, such as structural equation modeling, assessment of correlations with psychophysiological data (EEG, fMRI, etc.), or correlations with psychometric constructs (e.g., questionnaire data assessing rumination tendency). In line with recent claims in the field (Parsons et al., 2019), we suggest that researchers should assess and optimize reliability of their behavioral measures before subjecting them to correlational analyses.