Exploring the impact of stimulus–stimulus and stimulus–response conflicts on computer mouse trajectories: continuous flow of information from stimulus encoding to response preparation to motor action

Being able to operate efficiently in daily life requires the ability to orient our attention and behaviour towards goal-relevant information under conditions of conflict (‘cognitive control’). The mechanisms behind cognitive control have been extensively explored with tasks that involve some form of conflict. For instance, in the Simon task, a stimulus is classified into one of two responses (e.g., by colour: blue → left; red → right) and the stimulus is presented to the left or right of the screen. The irrelevant stimulus position is compatible or incompatible with the response side, and reaction times are typically slowed on incompatible compared to compatible trials (see Hommel, 2011, for review). Cognitive control is required to resolve the incompatibility on incongruent trials, and subtracting reaction times in the congruent trials from reaction times in the incongruent trials provides a measure of the effectiveness of an individual’s ability to exert cognitive control.

In conflict tasks, relevant and irrelevant stimulus dimensions and the response can engage in various forms of congruency. The ‘dimensional overlap’ (DO) model by Kornblum (1994) systematically organises various experimental tasks into a taxonomy of ‘ensembles’ depending on their dimensional overlap, with the Simon task a member of pure stimulus–response (S–R) (or ‘Ensemble 3’) tasks. However, congruency effects can also arise from instances in which the response dimension is not involved, but instead relevant and irrelevant stimulus dimensions overlap. An instance of a stimulus-stimulus (S–S, or ‘Ensemble 4’ type) task is the classic colour-word Stroop task with verbal responses, in which the conflict in the ‘incongruent’ condition (e.g., naming the colour of the word BLUE printed in red) arises from two overlapping stimulus dimensions. It is worth highlighting that even in tasks characterised as involving ‘stimulus-stimulus’ conflicts here and elsewhere, it is still the case that responses might compete against each other. Even in the Stroop task, co-activated words compete with each other in the mental lexicon (e.g., Roelofs, 2003). Hence, any task which involves some sort of decision can be characterised as having a ‘response dimension’. Here we use the term stimulus–stimulus conflict in the narrower sense that relevant and irrelevant dimensions overlap regarding their stimulus but not their response characteristics (Kornblum, 1994). This issue will be unfolded in greater detail in the General Discussion.

Kornblum’s classification of stimulus/response compatibility constellations have had considerable impact because it relates experimental tasks to theoretical notions of perceptual and cognitive processing. His DO model (Kornblum, 1994) joins a tradition of models prevalent at the time (e.g., Sternberg, 1969) according to which information processing consists of sequence of stages, with strict thresholding in between them. Hence, information processing at a particular stage is not initiated until processing at the previous stage has been completed to a point where a critical threshold has been reached. For instance, in Kornblum et al.’s (1999) computational implementation of the DO model, processing is divided between an input layer (concerned with stimulus encoding) and an output layer (in charge of response preparation), with a thresholding mechanism in between the two stages which gives the model serial processing characteristics. As a theoretical alternative to such strictly serial models, information may continuously ‘leak’ (or cascade) from one stage to the next, a notion which is embedded in numerous frameworks arising from the ‘parallel distributed processing’ framework (e.g., McClelland, 1979). In recent years, these continuous models have been favoured over thresholding models.

Pre-screening revealed that one participant exhibited an error rate larger than 25%, and this participant was removed from the analysis, with 29 participants remaining. Reaction times faster than 150 ms and slower than 1500 ms (1.5%), as well as latencies on error trials (4.4%) were removed from the analysis. Results are shown in Fig. 2.

Errors were analysed via a two-way Analysis of Variance (ANOVA) with the within-participants factors conflict type (S–R vs S–S) and congruency (congruent vs incongruent). Results showed a highly significant main effect of conflict type, F(1, 28) = 19.13, MSE = 9.92, p < 0.001, of congruency, F(1, 28) = 54.67, MSE = 17.35, p < 0.001, and a highly significant interaction between congruency and conflict type, F(1, 28) = 31.29, MSE = 11.96, p < 0.001. The effect of congruency was significant for the S–R conflict type (9.3%), t(28) = 7.50, p < 0.001; as well as for the S–S conflict type (2.1%), t(28) = 3.08, p = 0.005. A parallel ANOVA conducted on the response latencies revealed a significant main effect of conflict type, F(1, 28) = 9.33, MSE = 273, p = 0.005 and congruency, F(1, 28) = 110.84, MSE = 582, p < 0.001, and a significant interaction, F(1, 28) = 4.86, MSE = 1030, p = 0.036. The effect of congruency was significant for the S–R type (60 ms), t(28) = 7.75, p < 0.001, as well as for the S–S type (34 ms), t(28) = 4.78, p < 0.001.

This study exhibited highly significant S–R and S–S congruency effects both in response latencies and error rates. Regarding latencies, the S–S effect (34 ms) reported in our study compares well with previous studies using a similar arrow task, for example, Li et al., (2014; S–S: 33 ms; S–R: 37 ms) and Wang et al., (2014; S–S: 30 ms; S–R: 36 ms). However, in comparison to these studies, the S–R effect (60 ms) in our study is noticeably larger. The reason for this discrepancy is presently unclear, nonetheless we reliably captured S–S and S–R congruency effects in key press responses in the arrow task. We now tackled the central question of how these effects emerge in mouse tracking. As outlined in the Introduction, numerous mouse tracking studies have used S–R manipulations and observed more curved response trajectories for incongruent than congruent trials. The central question is whether this is also the case for S–S conflicts which are generated via conflict between relevant and irrelevant stimulus dimensions but do not involve the response dimension.

Data were processed in R (R Core team, 2021) using the package mousetrap (Kieslich, Henninger, Wulf et al., 2019). For each trial, initiation time was calculated as the time at which participants began moving the mouse, calculated relative to the onset of the target arrow (in milliseconds). Response latencies were determined as the time between target onset and the time at which participants clicked on one of the response boxes (in milliseconds). Errors were calculated as responses toward the incorrect response region. As a measure of curvature of a movement trajectory, we used “Area under curve” (AUC), the geometric area between the trajectory and a straight line proceeding directly from the start to the response region, calculated in pixels (Kieslich, Wulff, Henninger & Haslbeck, 2019). To avoid very large numbers, these were subsequently divided by 1000.

Trials with response latencies faster than 0 ms and slower than 2000 ms (1.6%) were excluded from the analysis, as were trials with initiation times slower than 500 ms (2.6%). All trajectories were time-normalised into 101 time steps and flipped to appear pointing toward the left response box. Finally, in previous experiments using MouseTracker we observed that participants clicking on the “Start” region often generate unintentional very small cursor movements which can be mistaken for genuine movement initiation times. Hence, in the current study we implemented an initiation threshold such that the first time sample with a cursor movement larger than 36 pixels was taken as the initiation time. This specific value was chosen because upon clicking on “Start”, MouseTracker moves the cursor to the centre of the start region which has a height of 72 pixels. Hence, initiation was conceptualised as the time at which the cursor left a virtual circle around the starting position with a radius of 36 pixels (cf. Ye & Damian, 2023).

Figure 3 shows average mouse movement trajectories on the left side, and inset panels on the right side show the four dependent measures (error rates, initiation times, response latencies, curvature) which were analysed via two-way ANOVAs with conflict type (S–S vs S–R) and congruency (congruent vs incongruent). The inset panels on the right side also show the ANOVA results, with the p values corresponding to conflict type, congruency, and the interaction reported below each panel (detailed statistics can be found in Appendix A). Average trajectories are less straight in the incongruent condition, when compared to the congruent condition, and this was the case not only for the S–R conflict type, but also (and critically) for the S–S conflict type.

An interaction between type of conflict and congruency was found in response latencies, and AUC but not in initiation times and errors. Simple effects of congruency carried out for each conflict type separately showed that for the S–R task, the congruency effect was significant in the errors, 1.48%; t(27) = 2.64, p = 0.0137, response latencies, 127 ms; t(27) = 11.86, p < 0.001, initiation times, 9 ms, t(27) = 3.21, p = 0.003, and curvature, t(27) = 14.65, p < 0.001. For the S–S task, congruency was not significant in the errors,  – 0.14%; t(27) = 0.29, p = 0.776, and initiation times, 2 ms, t(27) = 0.50, p = 0.621, but it was significant in response latencies, 15 ms; t(27) = 2.53, p = 0.017, and curvature, t(27) = 2.68, p = 0.012.

As can be seen in Fig. 3, the average trajectory corresponding to the S–R ‘incongruent’ condition points to the incorrect response in its early stage, then diverts course and finally arrives at the correct response. On a broad level, this ‘change of mind’ (CoM) indicates that the decision had not been completed when movement was initiated but was revised during action execution, and that the incorrect response was previously activated. By contrast, for the S–S ‘incongruent’ condition it is less clear whether CoM took place, and hence whether incongruency results in an attraction towards the incorrect response. To explore this issue, we performed a ‘cluster analysis’ in which each raw trajectory was mapped onto one of five ‘prototypes’, with two of them (‘straight’ and ‘curved) involving no temporary diversion toward the wrong response, and the three remaining ones involving various forms of CoM (see Wulff et al., 2019, for details). An ANOVA conducted on the proportion of CoM trials showed main effects of type of conflict, congruency, and an interaction (all ps < 0.001). For the S–R trials we observed a dramatic increase of CoM trials in the incongruent relative to the congruent condition (0.07 vs 0.55, respectively; t(27) = 14.23, p < 0.001); by comparison, for S–S trials the proportion of CoM trials differed less strongly and the statistical comparison was not significant (0.17 vs 0.20; t(27) = 1.87, p = 0.072). These results highlight the powerful attraction towards the incorrect response on S–R incongruent trials (which is exactly as expected given that the conflict involves the response dimension); by contrast, the results regarding whether incongruent S–S trials also involve activation of the incorrect response are not particularly clear.

In our experiments, we used an integrated task previously used in the literature (e.g., Li et al., 2014) which allows the generation of stimulus–response and stimulus–stimulus compatibility effects. Experiment 1 used key press responses and confirmed that the integrated task allowed us to reliably capture both S–S and S–R effects, in line with previous studies which had used a similar manipulation (e.g., Li et al., 2014; Paap et al., 2020; Wang et al., 2014). In Experiment 2, responses were made via dynamic computer mouse movements, and the main aim was to compare and contrast the way in which S–R and S–S effects emerge in responses of this type. As highlighted in the Introduction, S–R manipulations have been widely used in conjunction with mouse tracking and typically emerge not only in response latencies, but also in the curvature of the movement trajectories. The critical question was whether S–S manipulations would emerge in a similar fashion to S–R conflicts. The answer was positive: S–S effects also emerged in the curvature of response movements, although in reduced magnitude compared to S–R effects.

A methodological implication of our results is that mouse tracking can clearly be used to investigate S–S conflict. However, our findings are theoretically informative as well. As described in the Introduction, the fact that an S–R conflict consistently affects the characteristics of ‘dynamic’ responses such as mouse tracking or reaching suggests that cognition and action are not ‘staged’, i.e., motor execution begins before a decision has been fully completed. This finding is broadly in line with cognitive theories in which mind, body and environment dynamically interact (e.g., Spivey & Dale, 2004). More specifically, if there is a ‘threshold’ between cognition and action (e.g., Calderon et al., 2015) it is evidently set relatively low in our mouse tracking task such that motor activation is released at a quite early stage of the decision-making process. Our novel finding is that S–S compatibility affects dynamic responses. For an effect such as our current S–S manipulation (which resides at the cognitive stage of stimulus encoding) to emerge in response characteristics, it has to be assumed that stimulus encoding and response preparation are ‘cascaded’ and processing is not staged: a S–S conflict cascades into response preparation, and finally emerges in motor execution. This assumption is at odds with theoretical models such as Kornblum’s (1999) DO model which propose thresholding between stimulus encoding and response preparation. If this was true, the prediction would have been that only S–R effects appear in curvatures. This is because proponents of a model of this type would argue that a S–S conflict is resolved at the stimulus encoding stage before movement initiation while S–R conflict would be resolved during the response preparation stage. By contrast, our findings indicate that both S–S and S–R effects can be observed in mouse trajectories. Therefore, our results are in line with ‘continuous flow’ models which bridge the gap between cognition and action (Erb et al., 2021; Eriksen & Schultz, 1979).

A potential counterpoint that can be made is that curved trajectories do not reflect online corrections in movement. For example, Song and Nakayama (2008) observed no difference in response latencies (note, the authors used the term “total time”) between curved and straight trajectories. As a result, trajectories appeared to be comparable in terms of efficiency and in turn, the authors concluded that corrective movements plans are generated before movement is initiated. This conclusion would be at odds with “continuous flow” models which hold that movement can be executed before a decision has been completed. However, our data in Experiment 2 are not in line with Song and Nakayama’s findings since it appears that curved trajectories are less efficient (indexed by response latencies) than straight trajectories. To demonstrate, on Fig. 3, when comparing the average trajectories of S–R congruent trials with S–R incongruent trials, the former is noticeably straighter than the latter. If curved trajectories reflect corrective movement plans that were made before a participant started moving the mouse, then we should expect to observe similar response latencies between these two trial types. However, as can be seen on the response latencies graph on Fig. 3, average responses latencies are significantly longer for S–R incongruent than S–R congruent trials. Therefore, for our data, curved trajectories or larger AUCs do appear to reflect online corrections and thus imply that participants are moving their mouse before fully committing to a decision.

It is important to embed the results of the current study in a wider literature on decision making and action. Imagine a hypothetical result of our Experiment 2 in which stimulus–stimulus incompatibility resulted in a movement trajectory which compared to the compatible condition, was simply shifted in time (with slower initiation and response times) but generated trajectories of identical shape. As outlined in the Introduction, every situation which requires a decision via action involves some sort of competition between response alternatives. The hypothetical scenario would require that (a) this competition took place, and was resolved, in an abstract decision space rather than between response alternatives, and that (b) action commenced only after the decision was completed. Although such a scenario is not impossible and could in fact be predicted by ‘serial’ models such as those by Kornblum et al. (1999), more recent theorising tends to conceptualise decision making and action as a graded continuum. For instance, Wispinski et al. (2020) provide an extensive review and interpretation of a wealth of behavioural and neural evidence and suggest that the relationship between decision making and action is best characterised as a continuous and graded process which traverses a ‘landscape’ of behavioural options, from presentation until movement has been completed. This relationship can more clearly be captured using dynamic responses as opposed to ballistic ones. Under this theoretical perspective, the central finding of our second experiment which involved a dynamic rather than ballistic response method (mouse tracking), the results regarding the stimulus–stimulus compatibility condition were maybe predictable. Indeed, perhaps in this view, the surprising aspect of our findings is that stimulus–stimulus incompatibility had merely a rather subtle effect on movement trajectories. As highlighted in the Introduction, the bulk of relevant work on mouse movement trajectories has involved variations of stimulus–response compatibilities, and here their consequences for trajectories are well-documented in numerous studies. The results of our Experiment 2 suggest that indeed, the bulk of effects on movement curvatures derives from directly competing response options, with stimulus–stimulus conflicts generating much smaller (but still reliable) consequences for trajectories. Considering the results of the change-of-mind (CoM) analysis, it is unclear whether stimulus–stimulus conflict results in an attraction towards the incorrect response or in a delay in decision making where participants move the mouse forward between the two response options for a longer time. This differs from stimulus–response conflict where it appears to be clear that conflict results in an increase in attraction towards the incorrect response.

Given that our mouse tracking experiment yielded evidence that stimulus–stimulus conflict can affect the characteristics of response movements, future research may investigate whether our findings generalise to other tasks and stimuli. For instance, in a related task used in the neuroscientific literature to disentangle stimulus from response-based conflicts (e.g., Marinkovic et al., 2012), participants classify four colour patches into two responses (e.g., press ‘left’ if green or red; press ‘right’ if blue or yellow). Target colour patches are ‘flanked’ by two distractor patches of the same colour, which can either be congruent (e.g., a red target flanked by red distractors), stimulus incongruent (a red target flanked by green distractors), or response incongruent (a red target flanked by yellow distractors). Response latencies show a gradient, with slower stimulus incongruent than congruent responses, and slower response incongruent than stimulus incongruent responses. The prediction based on our current findings is that in a mouse tracking version of this task, average trajectories associated with the three conditions will follow the same pattern, with trajectories for the response incongruent condition showing a sizeable deflection towards the incorrect response, and trajectories for the stimulus incongruent condition still showing some deviation from the congruent condition (but probably less so than in the stimulus incongruent condition). The same logic applies to the Eriksen task with letter flankers (Eriksen & Eriksen, 1974). Here, participants classified four target letters into two response categories (H and K, or S and C). Flanking distractor letters could be stimulus-incongruent (e.g., KKKKHKKK) or response-incongruent (SSSHSSS), and relative to a condition in which targets and flankers were identical, latencies showed a gradient with somewhat slower latencies for stimulus-incongruent flankers, and substantially slower latencies for response-incongruent flankers. Again, we predict that in a mouse tracking study, the same gradient should appear in the curvature of average response trajectories.

As summarised in the Introduction, Li et al. (2014) factorially crossed the S–R and S–S manipulation with the same spatial arrow task used in the current study, and found strict additivity in the conflict scores. A potential further experiment would be to implement this manipulation in a mouse tracking task; our prediction, based on our findings, is that interactivity rather than additivity would be observed in curvatures. This is because we found both S–S and S–R effects in curvatures, with the inference that stimulus encoding and response preparation are closely related and both cascade into motor execution. Having said that, considering that the magnitude of the S–S effects in response latencies for Experiment 2 (15 ms) was relatively small, it is possible that no interaction would be found in latencies and/or curvatures, because according to Hommel (1997) and Sanders (1980) interactions require relatively large base effects to be reliably observed.

In conclusion, our current study is to our knowledge the first to demonstrate ‘pure’ S–S effects emerging in the curvature of response trajectories in a mouse tracking task. A major theoretical implication is that stimulus encoding and response preparation are closely related (i.e., processing of the former ‘cascades’ into the latter) and both cross the gap between cognition and action (i.e., characteristics of decision making emerge in motor characteristics). Future research should use the mouse tracking paradigm to further investigate cognitive control via the exploration of S–S and S–R manipulations.