The experiential basis of compatibility effects in reading-by-rotating paradigms

Traditional approaches to cognition see language as a mental system manipulating abstract, amodal and arbitrary symbols by means of grammar rules (Fodor, 1975; Kintsch, 1988; Pylyshyn, 1984). In this perspective, no relevance is given to connections between mental representations of linguistic meaning and representations of our bodily experiences.

Theories of embodiment on the other hand postulate a representational overlap between perception, action and cognition: sensorimotor traces formed during our physical interaction with the environment are reactivated and newly combined to yield mental simulations that make up cognition. These traces are experiential, multimodal and interconnected via co-occurrence learning mechanisms. As part of cognition, language processing itself might be grounded in perception and action, and exploit the same kind of representations as those involved in perception/action (Barsalou, 1999; Glenberg, 1997; Zwaan, 2004). Specifically, embodied-simulation views of language comprehension suggest that understanding words and sentences involves mentally simulating the actions, events and referents being described through the reactivation of experiential traces (Zwaan & Madden, 2005). For example, understanding a sentence like John pets the dog would reactivate perceptual traces of direct experience with dogs, as well as motor traces of petting.

Whereas the limitations of the functional role of such simulations are still under discussion (Fischer & Zwaan, 2008; Kaup et al., 2015), the literature includes evidence claiming support for embodied-simulation views. fMRI studies have highlighted physical overlap of brain patterns active during language comprehension with the ones active during direct experience of the linguistic content. For example, González et al. (2006) showed increased brain activity in the olfactory region when participants read words with referents associated with a strong olfactory experience, such as cinnamon. Additional support is presumably provided by behavioural studies dealing with so-called action-sentence compatibility effects (Glenberg & Kaschak, 2002), although there is some debate about the extent to which the original effect can be replicated (see, e.g., Papesh, 2015). These studies show that the comprehension of linguistic material describing actions affects actual physical movements of the reader that are relatable to the direct performance of these action, in a way that hints at the reactivation of the corresponding motor traces. One variant of this paradigm is the reading-by-rotating paradigm introduced by Zwaan and Taylor (2006), who uncovered a compatibility effect between the content of a sentence and the actual movement participants performed during reading. In Zwaan and Taylor (2006) participants read sentences such as (1) frame by frame, proceeding from one frame to the next by rotating a knob in either a clockwise or counterclockwise direction. Each sentence contained a critical frame implying a hand rotation movement (e.g., turned down in (1)) that is usually performed in either a clockwise or counterclockwise direction (e.g., counterclockwise for turned down). Reading times for the critical frames were shorter when the described actions involved a direction of rotation matching the required direction of rotation of the knob, compared to when the two directions were opposite. The finding of a match advantage was reproduced in further studies employing the reading-by-rotating paradigm (Taylor & Zwaan, 2008; Taylor et al., 2008; Zwaan et al., 2010).

However, a study applying the reading-by-rotating paradigm on verb gapping with sentences such as (2) produced the opposite pattern of results, with a reading time advantage for mismatching rotation directions in the frame with the overt verb (e.g., opens a lemonade bottle) as well as in the frame with the gapped verb (e.g., a juice bottle) (Claus 2015). This inconsistency seems to count against an embodied simulation view of language comprehension, as it might lead to the suspicion that compatibility effects arise arbitrarily and inconsistently across studies.

Claus (2015) discussed how the opposite findings can be reconciled within the embodiment framework by considering differences in the experimental task demands. One account pertains to differences in the specific knob turning procedures. Specifically, in Zwaan and Taylor (2006) participants were instructed to rotate the knob to trigger each frame change and to keep holding it in the current position before proceeding to the next frame. Only at the end of each item could they release the knob, which due to the presence of springs would come back to the original central position. In Claus (2015) the knob (also containing springs) was to be released back to the central position after each frame turn. The two different procedures could potentially explain the mismatching results, as they might have led to differences in the relative temporal dynamics of mental simulation and motor planning. Possibly, the necessity to keep holding the knob in Zwaan and Taylor (2006) resulted in temporal overlap of the two processes, whereas in Claus (2015) motor planning might have succeeded mental simulation. This could be decisive if temporal succession of mental simulation and motor planning, as opposed to overlap, inhibits rather than facilitates a matching action. The idea is supported by studies pointing to the lack of a compatibility effect when motor instructions are given after sentence presentation (e.g., Borreggine & Kaschak, 2006; Diefenbach et al., 2013; Kaschak & Borreggine, 2008; but see de la Vega et al., 2013 for different findings). An alternative account of the opposite patterns of results relates to a difference in the materials. Each experimental item of Zwaan and Taylor (2006) involved only one main character, whereas there were two main characters in each experimental item of Claus (2015). This difference in number of characters might have led to a difference in perspective adoption (Franklin et al., 1992). With one character, the default might be an internal, character-centered perspective, yielding a simulation of performing the described actions. However, with two characters, comprehenders might adopt an external, “en face” perspective (cf. Sato et al., 2012) and simulate the observation, rather than the execution, of described actions.

The current study aims at exploring the validity of the two accounts of the conflicting findings. We used the exact same device as in Claus (2015), but our materials always described only one character. Thus, if the device is the critical factor, a mismatch advantage should be observed as in Claus (2015). If on the other hand the number of characters is crucial, a match advantage should be observed as in Zwaan and Taylor (2006).

However, there is another major difference in the materials between the studies by Zwaan and Taylor (2006) and Claus (2015). Whereas all the target actions in Claus (2015) were opening/closing actions of objects with a lid (e.g., bottles, jars, tubes), the sentences in Zwaan and Taylor (2006) were more mixed (e.g., sharpening a pencil, screwing in a light, opening a jar, dimming the lights; with only three out of 16 items involving objects with a lid). This difference in the materials may have been crucial when considering the possibility of individual differences in actually performing the described actions. Most of the target actions in the material of Zwaan and Taylor can be assumed to be uniformly performed in one direction with one hand (or with both hands, e.g., turning left while driving). This is different for the actions in the materials of Claus: opening/closing objects with a lid often involves the simultaneous rotation of both hands in opposite directions. For instance, if the right hand is on the lid of an object and the left hand is on the base of the object, then the right hand turns clockwise for closing and counterclockwise for opening, whereas the left hand—positioned on the base of the container—turns counterclockwise for closing and clockwise for opening; and vice versa if the left hand is on the lid (see Fig. 1). In this matter, it seems conceivable that individual preferences for the use of a hand over the other in rotating lids (independently of the participant’s handedness) might be reflected on a match or a mismatch advantage.

In the study by Claus, all participants were right-handed and assumed to use the right hand on the lid when opening/closing objects with lids: the levels match and mismatch were assigned to the experimental conditions (described action: open/close x knob turning direction: counterclockwise/clockwise) according to the manual rotation of the hand on the lid (match: open + counterclockwise and close + clockwise; mismatch: open + clockwise and close + counterclockwise) and the participants were explicitly instructed to rotate the knob only with the right hand. However, the above considerations suggest that this assignment of the match and mismatch levels might not be generally appropriate. Only for participants who preferably have their right hand on the lid does the right-hand knob turning in the match/mismatch conditions of the experiment indeed match/mismatch their right hand rotations when actually performing the described opening or closing actions. However, for participants who preferably have their right hand on the base, the mapping of match and mismatch is actually reversed, because the direction of the rotation of the hand on the base is opposite to that of the hand on the lid (match: open + clockwise and close + counterclockwise; mismatch: open + counterclockwise and close + clockwise). Thus, for these participants, the right-hand knob turning in the match conditions of the experiment mismatches rather than matches their right hand rotations when actually performing the described actions, and vice versa for the mismatch conditions: for example, when reading about an opening action, their simulation might involve a counterclockwise turning of the left hand but a clockwise turning of right hand, such that the mismatch condition, requiring a clockwise turning of the knob, mismatches the simulation for the left hand but actually matches the simulation for the right hand (the hand that performs the actual knob turning) resulting in a mismatch advantage. Hence, for the former group of participants (right hand on lid), a match advantage is expected, reflecting the pattern of movements of the right hand on the lid, whereas for the latter group of participants (right hand on base), a “mismatch” advantage is expected, reflecting the pattern of movements of the right hand on the base of the containers.

According to this line of reasoning, the mismatch advantage observed by Claus (2015) might be due to the possibility that the majority of participants in her study preferably use their right hand on the base when opening/closing containers with a lid. To pursue some insight with regard to this explanation, we included an assessment of participants’ hand preferences in the present experiments.

In Experiment 1, to advance from frame to frame, participants rotated the knob in the same direction throughout each of the two experimental blocks. This might have led to a rather automated performance of the movement. In Experiment 2 the direction of knob turning changed on a frame-by-frame basis, as participants had to choose between the two directions of turning depending on the font color in which the frame was presented.

The initial aim of the current study was to explore the role of different knob turning procedures and different number of characters involved in the narratives in explaining conflicting findings of studies employing the reading-by-rotating paradigm. Specifically, Zwaan and Taylor (2006) found shorter reading times for sentence frames describing hand rotations, when the implied direction of rotation (clockwise or counterclockwise) matched the direction of hand rotation that the participants were instructed to perform immediately afterwards. In contrast, Claus (2015) found a mismatch advantage.

In our study we used the exact same knob device that was used in Claus (2015), and we presented items describing the opening and closing of the same objects as employed in the materials of Claus (2015). We changed the number of characters described in the texts, such that the actions would involve only one character (vs. two characters in, Claus, 2015). If the inconsistency in the two studies were attributable to the difference in number of characters, an overall match advantage was to be expected. If, instead, the difference in the knob turning procedure was crucial, an overall mismatch advantage was to be expected. The overall results show no significant difference for the match vs. mismatch condition on the critical frame.

An additional difference in the materials of the two studies appeared to be potentially relevant: all target actions of Claus (2015) were actions of opening/closing containers with a lid, which often involve the simultaneous rotation of both hands in opposite directions, whereas the target actions of Zwaan and Taylor (2006) were more heterogeneous and mostly implied only one direction of rotation. One of the fundamental assumptions of embodied simulation views of language comprehension is that mental simulations involve the reactivation of individual experiential traces. If the experience of the participants differs in terms of being accustomed to rotate lids with the right vs. left hand, then differential patterns of motor trace reactivation should be expected. By proceeding along this line of reasoning, we explored the experiential basis of action-sentence compatibility effects by inquiring individual motor experiential differences.

Experiment 1 indeed showed a different pattern of results depending on participants’ hand preferences when opening/closing bottles. A match advantage was found for participants who use the right hand on the lid, and a mismatch advantage for those that use the left hand on the lid. This translates in a compatibility effect between the action described in the sentence and the action performed by the participants, as they are accustomed to perform it; when defining match and mismatch according to participants’ self-assessed hand preferences, both groups showed a match advantage. Experiment 2 replicated the finding for the right-hand-on-lid group, but the compatibility effect did not reach significance level for the left-hand-on-lid group. The non-significant compatibility effect for the group that uses the left hand on the lid is maybe attributable to reduced statistical power with respect to this group. However, more in general, the compatibility effect elicited in the two studies might simply be more stable for right-handers who use the right hand on the lid. In fact, right-handers who prefer the left hand on the lid might be less consistent in the use of the preferred configuration, the right hand still being their dominant one (they might sometimes still use the right hand on the lid).¹ This possibly raises the need for more fine grained measures of hand preference. Gijssels and Casasanto (2020) indeed point out that individual actions vary continuously, rather than categorically, either in the extent of the involvement of each hand (for bimanual actions), or in the extent to which people consistently prefer one hand over the other (for unimanual actions). These kinds of considerations have important implications for experiments on embodied cognition, in that categorical measures might lead to higher rates of type II errors. In the case of the current study, the actions described in the experimental items might have varied continuously both in the strength of the individual preference for a hand configuration, and in the relative workload of the dominant and non-dominant hand. Another explanation for the absence of a mismatch advantage for the left-hand-on-lid group in Experiment 2 lies in the misalignment of motor dimensions in the different movements: one might argue that the lid-turning movement and the container-turning movement do not occur along the same motor dimension, with the lid turning being carried out by means of adduction–abduction movements, and the container-turning via flexion–extension movements; in contrast, turning the experimental knob requires only adduction–abduction movements. The divergence of the container-turning and the knob-turning movements in terms of motor dimensions involved might support the conclusion of a null effect for the left-hand-on-lid participants (as found in Experiment 2, the one with the more robust left-hand-on-lid sample), with the mismatch advantage in Experiment 1 simply reflecting random variation. If the divergence really excludes any kind of overlap between the two movements, they need to be brought under a common denominator at a more abstract level to justify a mismatch advantage: for example, they might have a common directionality marking. Bub and Masson (2010) consider the possibility that the motor resonance in Zwaan and Taylor (2006) reflects the correspondence of only more generic action features, such as direction of motion. There is nonetheless no sufficient evidence available to lean towards the one or the other experiment in terms of credibility of the results for the left-hand-on-lid subgroup, and the issue requires further investigation to reach conclusiveness.

It should be further noted that the results reported in this study are fruit of a rather nonconservative analysis and should, therefore, be interpreted with caution. The factor Frame was dropped from the analyses due to power concerns. We are aware that the embodied account would predict a differential behavior across frames in the first place, and a more conservative analysis including the factor Frame would allow to draw stronger conclusions. At the same time, a test interested in detecting a potentially very small interaction is likely to require a sample size that is simply too large for us to collect. The reported results should, therefore, be seen as providing compatible evidence within the limits of the available resources.

With regard to the issue of the conflicting results of the studies by Claus (2015) and Zwaan and Taylor (2006), the present results do not support the assumption that either the number of characters described nor the different devices are decisive. In fact, they are inconclusive in this respect. However, they suggest that it is necessary to take into account individual differences in action execution. For the study by Claus (2015), the information whether participants have a preference for the right or the left hand on the lid when opening/closing bottles is not available. An account of her mismatch advantage finding in terms of individual differences would presuppose a stable left-hand-on-lid preference for the majority of her participants, which seems unlikely considering our data.

Whereas contrasting versions of the embodied-simulation view of language comprehension differ in their claims, mostly with respect to the strength attributed to the functional role of the simulations for comprehension (Kaup et al., 2015), they all share the underlying assumption that experiential traces are reactivated during language processing. The questions of whether modal experiential traces are the only meaning representations available for language processing and, if not, to what extent they are functionally relevant to comprehension, are still open (Dove, 2009; Zwaan, 2014) and are not tackled in our study. In its most radical interpretation, embodied simulation views assume an equivalence between comprehension and mental simulation, whereby modal traces are the only meaning representations available. One implication of this account is that simulations should be found not only at word-, but also at phrase- and sentence-level. Evidence in this sense remains limited (e.g., Taylor & Zwaan, 2008; Taylor et al., 2008), as it is not straightforwardly possible to disentangle whether the reactivation of sensorimotor traces is driven by experiential higher level mental simulations or mediated by more general linguistic/conceptual associations. In the case of the present study for example, we cannot safely assume the effects to reflect sentence- (or phrase-) level simulations. In fact, although it is unlikely that the effects stem from any single word employed in the sentences, associations might still hold between combinations of words and experiences (such as open and bottle and the action of rotating the lid counterclockwise) and this could explain our results without the need for modal meaning composition (as discussed in Kaup et al., 2012). On the other hand, studies such as Taylor and Zwaan (2008) and Taylor et al. (2008) point to compatibility effects resulting from modal meaning composition. Taylor and Zwaan (2008) found an action-sentence compatibility effect for action-modifying adverbs (e.g., On the shelf, he found a closed jar which he opened rapidly) but not for agent-modifying adverbs (hungrily instead of rapidly). Taylor et al. (2008) found a compatibility effect for adjectives that allowed for inferences with respect to to the direction of an action that was mentioned earlier in the text (e.g., He approached the stereo and adjusted the volume. The music was too loud). At the very least, these results are difficult to explain with simple associations of words and experiences.

As for the assumption of the experiential nature of the traces reactivated during language processing, such an assumption would predict that, if people differ in their experience, accordingly will their simulations (as supported by the findings of Casasanto, 2009; Holt & Beilock, 2006; Lyons et al., 2010). Despite the evidence for the activation of sensorimotor traces during language processing, the majority of the studies do not allow to conclude that these sensorimotor traces are based on the experiences of the individual at hand (Öttl et al., 2017; Casasanto, 2011). Although not providing conclusive evidence, the present findings seem to reflect differences in the reactivation of experiential traces during the simulation process which are brought about by differences in individual experience. The different patterns of results found for the two groups seem to be related to the experiential differences, as self-assessed by the participants. Nevertheless, whereas the interaction effect strongly supports the experiential basis of compatibility effects, it does not guarantee the direct reactivation of motor experiences: the results are additionally compatible with, e.g., the hypothesis that movements might have a directionality marking, as well as with the idea that participants might be simulating the perceptual effects of the movements rather than more straightforwardly reactivating the motor experience itself.

To sum up, the current study addressed the linking of compatibility effects with individual motor experiences. Altogether, the results are suggestive of the potential need to take into account individual experiences in studies on embodied simulation views , which may help truly test the assumptions underlying these views.