Modality compatibility in task switching depends on processing codes and task demands

For the single-task blocks, we ran a repeated-measures ANOVA with the variables modality compatibility (incompatible vs. compatible) and processing code (spatial-verbal vs. spatial-location). For the analysis of mixed-task blocks, we calculated two ANOVAs each for RT and error rates, involving the independent variables modality compatibility (incompatible vs. compatible), processing code (spatial-verbal vs. spatial-location), and transition ([mixed repeat vs. single task {which is repetition by definition} for the mixing-cost contrast]; [switch vs. repetition for the switch-cost contrast]). To follow up significant interactions, paired-sample t-tests were calculated. All analyses were conducted at α = 0.05.

In Experiment 1, we tested the prediction that modality-compatibility effects in task switching should be larger with spatial-verbal stimuli because these should strengthen particularly the vocal-auditory modality mapping. We found a consistent influence of spatial-verbal stimuli in terms of a larger modality-compatibility effect on mixing costs compared to spatial-location stimuli. We also found increased switch costs with modality incompatibility with spatial-verbal stimuli (even though the trend was non-significant in RT), but for spatial-location stimuli we found it only for error rates, whereas this effect was even non-significantly reversed in RT, hinting at a specific speed-accuracy trade-off. Note that the effect of increased RT switch costs in modality-incompatible conditions has been replicated several times with spatial-location stimuli (Fintor, Stephan et al., 2018a; Stephan & Koch, 2010, 2011, 2015, 2016), so that its absence in the present experiment should not be overemphasized.

Note also that the single-task blocks showed even worse performance in modality-compatible conditions, but this was confined to spatial-verbal conditions. While we have no explanation for this effect, methodologically it rules out that more “difficult” tasks also lead to larger costs in multitasking (Stephan & Koch, 2011) because the data pattern is the opposite.

Using a dual-task setting, Göthe et al. (2016) had performed a modality-compatibility study with spatial-verbal and spatial-location stimuli in a between-subjects design with bimodal stimulation. They found higher dual-task costs not only for modality-incompatible mappings, but also for location-vocal and verbal-manual mappings of processing code and response, compared to the mappings spatial-manual and verbal-vocal. Notably, the highest costs were observed in the group that faced both an incompatible modality mapping and a location-vocal + verbal-manual feature mapping. In our task-switching setup, all factors were varied within subjects. Note also that none of the mappings in the study by Göthe et al. (2016) met the narrow definition of ideomotor compatibility (Greenwald, 1972), whereas our setup included one ideomotor-compatible condition: the auditory-verbal condition, that is, hearing the word “left” or “right” and responding vocally by saying the same word.

Taken together, the data suggest that spatial-verbal stimuli create greater between-task crosstalk than spatial-location stimuli when two tasks with modality-incompatible stimulus–response mappings have to be maintained in working memory at the same time. Note though that the spatial-verbal and the spatial-location task we used in Experiment 1 were similar to each other because both types of stimuli contained a spatial meaning (left vs. right).

Data analysis proceeded as in Experiment 1. Trials excluded as RT outliers amounted to 0.002%.

In Experiment 2, we found an influence of temporal-duration task demands, in form of a trend for a stronger effect of modality compatibility on mixing costs compared to spatial-location task demands. However, the data suggested a numerically larger influence of modality compatibility on mixing costs with the temporal-duration task only for RT. Again, similarly to Experiment 1, for spatial-location task demands the switch cost contrast showed an unclear speed-accuracy trade-off. The predicted interactions of modality compatibility and transition revealed a modality-compatibility effect on mixing costs for RT, and a modality-compatibility effect on switch costs for errors.

For Experiment 1, the interaction of mixing and processing code was significant for RT, F(1, 23) = 6.708, p = 0.016, η _p ²  = 0.226, revealing larger mixing costs with spatial-verbal codes compared to spatial-location codes (159 ms vs. 117 ms); in error rates, the interaction was not significant (p = 0.263), but the numerical trend pointed into the same direction (1.0% spatial-verbal vs. − 0.1% spatial-location). The interaction of switching and processing code was neither significant for RT, F(1, 23) = 1.845, p = 0.188, η _p ²  = 0.074, nor for errors, F(1, 23) = 1.230, p = 0.279, η _p ²  = 0.051, but there were numerically larger switch costs with spatial-verbal processing codes than with spatial-location processing codes (84 ms vs. 65 ms for RT and 3.8% vs. 2.1% for errors).

For Experiment 2, the interaction of mixing and task demand was neither significant for RT, F(1, 23) = 0.746, p = 0.397, η _p ²  = 0.031, nor for errors, F(1, 23) = 1.905, p = 0.181, η _p ²  = 0.076. The interaction of switching and task demand however was significant in both RT, F(1, 23) = 40.251, p < 0.001, η _p ²  = 0.636, and error rates, F(1, 23) = 15.091, p = 0.001, η _p ²  = 0.396, with larger switching costs for the temporal-duration task demand than for the spatial-location task demand (122 ms vs. 52 ms for RT and 5.4% vs. 0.8% for errors).

For Experiment 1, the interaction of mixing and processing code was neither significant for RT, F(1, 23) = 0.147, p = 0.705, η _p ²  = 0.006, nor for errors, F(1, 23) = 0.001, p = 0.970, η _p ²  = 0.000. Switching and processing code, however, revealed a non-significant interaction trend in RT, F(1, 23) = 3.836, p = 0.062, η _p ²  = 0.143 (120 ms spatial-location vs. 88 ms spatial-verbal), and a significant interaction in error rates, F(1, 23) = 6.416, p = 0.019, η _p ²  = 0.218 (8.2% spatial-location vs. 3.9% spatial-verbal), demonstrating consistently higher switch costs for spatial-location codes than for spatial-verbal codes.

For Experiment 2, mixing and task demand did not interact significantly, neither for RT, F(1, 23) = 0.294, p = 0.593, η _p ²  = 0.013, nor for error rates, F(1, 23) = 1.438, p = 0.243, η _p ²  = 0.059. Switching and task demand neither showed a significant interaction for RT, F(1, 23) = 0.003, p = 0.960, η _p ²  = 0.000, nor for errors, F(1, 23) = 0.868, p = 0.361, η _p ²  = 0.036.

Taken together, these results reveal that mixing costs and switch costs with the modality-incompatible auditory-manual mapping were indeed negatively affected by both spatial-verbal processing codes and temporal-duration task demands compared to spatial-location processing codes/task demands. Specifically, with the auditory-manual mapping spatial-verbal processing codes predominantly affected mixing costs, whereas temporal-duration task demands mainly influenced switch costs. Meanwhile, mixing costs and switch costs with the competing visual–vocal mapping either did not differ based on processing code/task demand, or they were enlarged with the spatial-location processing code/task demand compared to its respective counterpart. This suggests that the increased influence of modality compatibility on mixing costs (and, to a lesser extent, switch costs) which we found in the main analysis with both spatial-verbal processing codes and temporal-duration task demands can be attributed mainly to their effect on the auditory-manual mapping, probably because auditory stimuli are particularly strongly coupled with vocal responses.

We can only speculate why the auditory-manual mapping was only affected in terms of mixing costs by the spatial-verbal processing codes and only in terms of switch costs by the temporal-duration task demands. The absence of an effect of temporal-duration task demands on mixing costs for the auditory-manual mapping might be explained by the smaller similarity between the auditory temporal-duration stimulus and the auditory effect of the response, which was still spatial-verbal (the spoken word “left” or “right”)—compared to Experiment 1, where the effect of the vocal response and the auditory stimulus were ideomotor-compatible. This smaller similarity in Experiment 2 might have led to less between-task crosstalk.

We found a consistently larger modality-compatibility effect on mixing costs for both spatial-verbal codes compared to spatial-location codes (Experiment 1) and for temporal-discrimination task demands compared to spatial-location task demands (Experiment 2). The findings with respect to switch costs are less consistent in the spatial-discrimination paradigm because of an unexpected speed-accuracy trade-off, but for both spatial-verbal processing codes and temporal-duration tasks switch costs were always at least numerically larger in the modality-incompatible condition than in the modality-compatible condition.

The analysis of the individual modality mappings supported our account that, compared to spatial-location discrimination tasks (i.e. requiring spatial-location codes), both spatial-verbal codes and temporal-duration task demands indeed strengthened the auditory–vocal coupling. This is suggested by the analysis of the respective counterparts to those couplings, i.e. the individual modality-incompatible mappings: The auditory-manual mapping showed larger mixing costs with spatial-verbal processing codes and larger switch costs with temporal-duration task demands, as well as numerical trends into the same direction for larger switch costs with spatial-verbal processing codes in Experiment 1 and larger mixing costs with temporal-duration task demands in Experiment 2. Meanwhile, the overall impact of processing codes/task demands on the visual-manual coupling was much less pronounced when assessed in the modality-incompatible counterpart, that is, the visual–vocal mapping.

Embedding our findings on mixing costs into the literature, our results are in line with Schacherer and Hazeltine (2019), who also found larger mixing costs with modality-incompatible mappings than with modality-compatible mappings. Our study suggests, more specifically, that both spatial-verbal processing codes and temporal-discrimination task demands increase the impact of modality compatibility on mixing costs.

When two tasks with modality-compatible mappings need to be maintained in working memory, working-memory load should be lower because both tasks can be processed more or less independently in distinct subsystems (Baddeley, 1992, 2010). For example, the auditory–vocal task requires the phonological loop and the visual-manual task requires the visuospatial sketchpad, whereas in the modality-incompatible visual–vocal task, the visual stimulus would refer to the visuospatial sketchpad, but the vocal response (and its anticipated auditory effect) to the phonological loop. Hence, with both tasks referring to distinct subsystems, between-task crosstalk should be lower (see Maquestiaux et al., 2018; Schacherer & Hazeltine, 2019).

The supplemental analysis showed that mixing costs for auditory-manual trials were particularly affected by spatial-verbal processing codes and temporal-discrimination task demands (each compared to spatial-location codes/task demands). That is, when sensory input in different modalities needs to be processed and also reactions in different modalities are required, auditory stimuli tend to require vocal responses (as it is common in conversations in everyday life) and this tendency is even increased when auditory input was also verbal in nature. This suggests that, as predicted, the auditory–vocal coupling was indeed strengthened by spatial-verbal codes. This pattern is consistent with Wickens (1984), and we can specify that the definition of compatibility of mappings can be derived from ideomotor theorizing proposing a strong role of anticipation of sensory action effects in action selection generally (Greenwald, 1972; Shin, Proctor, & Capaldi, 2010).

While there is a similar connection between spatial-location tasks and the visual-manual mapping (Wickens et al., 1984), the similarity in the present study between a manual keypress and a white diamond-shaped stimulus is obviously much smaller than between an auditory stimulus and a vocal response which both produce the exact same word. Thus, the ideomotor linkage between the spatial-location task and the visual-manual coupling should be considerably weaker than the ideomotor linkage between the spatial-verbal task and the auditory–vocal coupling. Hence, spatial-verbal stimuli can be assumed to have strengthened the auditory–vocal coupling to a greater extent than the spatial-location stimuli may have strengthened the visual-manual coupling.

Regarding switch costs, however, the prediction that interference due to spatial-verbal codes or temporal-duration task demands should be particularly strong on switch trials could not be confirmed: While the modality-compatibility effect was still numerically present in the expected direction for both spatial-verbal codes and temporal-duration task demands, spatial-location codes/task demands yielded a speed-accuracy trade-off that was not observed in earlier studies (Fintor, Poljac et al., 2018; Fintor, Stephan et al., 2018a, 2018b; Schacherer & Hazeltine, 2019; Stephan & Koch, 2010, 2011, 2015, 2016; Stephan, Koch, Hendler, & Huestegge, 2013). Specifically with regard to spatial-verbal codes, our results are also not completely in line with the study by Schäffner et al. (2018), who had indeed found larger switch costs with verbal compared to nonverbal stimulus material, and with incompatible modality mappings, while we found larger switch costs with spatial-location (nonverbal) stimuli and compatible modality mappings in RT, and larger switch costs for incompatible modality mappings in error rates, but with both types of processing codes. The important distinction though, as mentioned earlier, is that in the spatial-verbal condition in the study by Schäffner et al. (2018), only the responses actually featured a spatial component, whereas the stimuli referred to semantic categories; in our Experiment 1, both the spatial-verbal stimuli and the required responses were conceptually overlapping, since both referred to the spatial dimension.

Schacherer and Hazeltine (2019) demonstrated how differences in switch costs between compatible and incompatible modality mappings could be made to disappear by reducing conceptual overlap between the tasks. However, since our participants only switched between modality mappings, that is, there were no trial-to-trial switches between spatial-verbal codes/temporal-duration task demands and spatial-location codes/task demands, we would argue that the amount of conceptual overlap within one processing code or task demand was equal: It either always referred to a spatial code (with spatial-verbal and spatial-location stimuli) or duration (with temporal-duration). Note that in the context of dual-task training, Maquestiaux et al. (2018) explained the beneficial effects of modality compatibility in terms of Baddeley’s (e.g. 1992, 2010) model of working memory, suggesting that response selection should be easier with compatible modality mappings because both stimulus and response codes referred to the same subsystem. However, this more general approach can explain the effect on mixing costs (see previous section); it is less clear how it could explain the specifically increased between-task crosstalk in task switches relative to repetitions.

Based on our ideomotor approach, we can integrate the notion of distinct working memory subsystems, so that these are not mutually exclusive accounts, even though we believe that our account is more specific. In particular, we propose that in Experiment 1, spatial-location codes may be processed in the visuospatial sketchpad, while spatial-verbal codes would be held primarily by the phonological loop (because their “spatial” component is merely semantic; their physical location was central/binaural, i.e. neutral). Hence, it would be more difficult to select the proper response code from one working-memory subsystem if both the code for the stimulus modality and the spatial-verbal/spatial-location code are processed in the respective other subsystem. In the supplemental analyses, the auditory-manual mapping showed larger interference with spatial-verbal codes (because both the auditory stimulus and the spatial-verbal code point towards the phonological loop, while the anticipated effect of the correct manual response should be processed in the visuospatial sketchpad), whereas the visual–vocal mapping showed increased interference, if at all, with spatial-location codes (because both the visual stimulus and the spatial-location code point towards the visuospatial sketchpad, while the anticipated effect of the correct vocal response should be contained in the phonological loop). In turn, auditory-manual spatial-location trials and visual–vocal spatial-verbal trials are “easier” because two of the three relevant codes are contained in the subsystem that features the correct response (spatial-location and manual in the visuospatial sketchpad, spatial-verbal and vocal in the phonological loop). This idea could also be transferred to Experiment 2: While Baddeley (2010) described how a visual verbal stimulus can be transformed into the phonological loop like a spoken word, it is by no means obvious that an analogous translation into the duration of some type of sound would happen to a visual temporal-duration stimulus. If indeed this does not occur, it would explain why, with the visual–vocal mapping, the temporal-duration task was not sufficiently easier than the spatial-location task to yield a significant difference in switch costs, as it was found for the same modality mapping between spatial-verbal and spatial-location processing codes.

As noted in the previous section, both multiple resource accounts (Wickens, 1984, 2008) as well as working memory subsystem accounts (Maquestiaux et al., 2018; see also Schacherer & Hazeltine, 2019) and the between-task crosstalk account that we propose in line with previous studies (e.g. Stephan & Koch, 2011, 2015, 2016) are not mutually exclusive but differ in their degree of specificity. Both could explain modality-compatibility effects in mixing costs (or dual-task costs in dual-task contexts, e.g. Göthe et al., 2016), whereas our ideomotor approach, based on anticipation of response effects (see also Wirth, Steinhauser et al., 2018), can also explain specifically increased interference in switch trials relative to repetition trials.

The present study adds to the body of evidence strengthening the claim that “central” processes, as put forward by both bottleneck accounts (e.g. Pashler, 1994) and resource-sharing approaches (e.g. Kahneman, 1973; Navon & Miller, 2002; Tombu & Jolicœur, 2003), are affected by modality-specific influences. Specifically, the increased mixing-costs with modality-incompatible mappings, as well as increased switch costs with such mappings in error rates, further strengthen the support for the between-task crosstalk account (Stephan & Koch, 2011; see also Göthe et al., 2016), in line with ideomotor approaches to action control (e.g. Greenwald, 1970, 1972; Shin et al., 2010). In contrast, modality-specific accounts that assume generally preferred mappings, such as visual-spatial-manual and auditory-verbal-vocal, meaning that these combinations of modalities and processing codes should be beneficial even in a single-task setting (Wickens, Sandry, & Vidulich, 1983; Wickens et al., 1984), would consequently expect incompatible modality mappings to lead to worse performance in general. However, as shown by the higher RT and error rates for the compatible condition in the single-task analysis of both experiments, this can be ruled out as an explanation for the influence of modality compatibility in multitasking situations. Our findings further extend previous research by demonstrating that the interference in form of between-task crosstalk can be further modulated by varying processing codes and task demands; specifically, such processing codes and task demands that strengthen the auditory–vocal coupling can further increase the already-present interference between mappings. Keep in mind that this between-mapping interference occurs in addition to the central interference in terms of general mixing costs and switch costs, which were still present with modality-compatible mappings. However, if both task sets feature incompatible modality mappings, spatial-verbal codes and temporal-duration task demands seem to interfere predominantly with the concurrent maintenance of two competing task sets in working memory, and only to a lesser extent with the updating and shifting from one task set to another, as it is required on switch trials.

In sum, our study provides further evidence for a modality-specific influence on cognitive control processes in the form of task-confusion due to crosstalk between two incompatible mappings. Our research extends the earlier work in this field by demonstrating that this influence can be further modulated by introducing both spatial-verbal processing codes and temporal-duration task demands compared to the previously employed nonverbal spatial-location paradigm. We attribute this to a strengthening of the auditory–vocal coupling, resulting from the tight ideomotor binding between audition and verbal content, as well as the modality appropriateness of audition for temporal discrimination.