Language and vertical space: On the automaticity of language action interconnections

Abstract

Grounded models of language processing propose a strong connection between language and sensorimotor processes (Barsalou, 1999, 2008; Glenberg & Kaschak, 2002). However, it remains unclear how functional and automatic these connections are for understanding diverse sets of words (Ansorge, Kiefer, Khalid, Grassl, & König, 2010). Here, we investigate whether words referring to entities with a typical location in the upper or lower visual field (e.g., sun, ground) automatically influence subsequent motor responses even when language-processing levels are kept minimal. The results show that even subliminally presented words influence subsequent actions, as can be seen in a reversed compatibility effect. These finding have several implications for grounded language processing models. Specifically, these results suggest that language-action interconnections are not only the result of strategic language processes, but already play an important role during pre-attentional language processing stages.

Introduction

Grounded models of language comprehension suggest a close connection between language understanding and sensorimotor processes (Barsalou, 2008, Glenberg and Gallese, 2012). Diverse empirical evidence supports a close relationship between language, perception and action. For example, Hauk, Johnsrude, and Pulvermüller (2004) have shown that the neural activation during reading action verbs (e.g., kick) resembles the neural activation during the actual performance of the accordant actions. Additionally, studies have demonstrated that language processing influences subsequent motor responses (e.g., Borghi et al., 2004, Borreggine and Kaschak, 2006, Boulenger et al., 2006, Glenberg et al., 2008, Scorolli and Borghi, 2007, Taylor and Zwaan, 2008, Zwaan and Taylor, 2006). For example, reading sentences such as “He opens the drawer” results in faster arm movements towards one's own body, than away from one's body (Glenberg & Kaschak, 2002). These language-action compatibility effects highlight the potential interconnections between language understanding and motor processes, and are often cited as important evidence in favor of the grounded language-processing model (Barsalou, 2008). However, despite substantial evidence that language and sensorimotor processes are closely interconnected and even share neural substrates, it is still unclear how fundamental these connections are for language understanding and whether they are automatically activated during comprehension (Fischer & Zwaan, 2008).

Further evidence supporting grounded language processing models stems from research that investigated direction-associated words. For example, words referring to entities with a typical location in the vertical space (e.g., hat = up, shoe = down) influence subsequent visual target processing in compatible or incompatible screen locations (Dudschig et al., 2012b, Estes et al., 2008, Gozli et al., 2013, Zhang et al., 2013). Similar results have been reported during verb processing (e.g., rise, fall) (Verges & Duffy, 2009) and during sentence comprehension (Bergen, Lindsay, Matlock, & Narayanan, 2007). Analog to the findings in studies investigating the effect of linguistic stimuli on perceptual processing, it has been shown that words referring to entities with a typical location also influence subsequent response-related processing (Lachmair et al., 2011, Thornton et al., 2012). In these studies participants were required to respond with either an upward or downward arm movement to word font color. Responses were faster if the arm movement was towards the compatible location (e.g., sun followed by an upward arm movement). Subsequent studies have shown that eye movements are similarly influenced by word processing (Dudschig, Souman, Lachmair, de la Vega, & Kaup, 2013) and that these language-action associations can also be observed during second-language processing (Dudschig, de la Vega, & Kaup, 2014). In addition, such language-action compatibility effects have also been reported for verbs (e.g., rise vs fall) (Dudschig, Lachmair, de la Vega, De Filippis, & Kaup, 2012a) and in studies implementing sentences (Kaup, De Filippis, Lachmair, de la Vega, & Dudschig, 2012). These compatibility effects have been attributed to automatic re-activation of experiential traces during language processing (e.g., Barsalou, 2008, Fischer and Zwaan, 2008, Zwaan and Madden, 2005). For example, when we hear the word bird, this often occurs in situations in which we look up to the sky, or in which someone points up to the sky. Thus, when later hearing the word bird, these perceptual and motoric experiences become automatically reactivated (Zwaan & Madden, 2005). Pulvermüller, 1999, Pulvermüller, 2005 proposed that Hebbian associative learning underlies these connections between language and motor activation, as frequently co-activated neurons strengthen their connections resulting in the development of functional cell assemblies. Thus, according to this view, word processing becomes closely connected to sensorimotor processing, and these connections are automatically reactivated when processing language.

The semantic processing demands in the studies summarized above vary with respect to the level of language processing required for the task. For example, in some paradigms, participants had to actively read the words or sentences and perform sensibility judgments by deciding whether a visually presented word was a real word or a pseudoword, or whether a sentence was sensible or not (e.g. Glenberg & Kaschak, 2002). In other studies, word meaning was task-irrelevant and participants responded to stimuli features such as color (e.g., Lachmair et al., 2011). Language-action compatibility effects in tasks where word meaning is task-irrelevant (e.g., Stroop, 1935) have been interpreted in favor of a highly automated connection between language and action. It was argued that automatic access to word meaning, as typically reported in a Stroop paradigm, is sufficient to trigger compatibility effects. However, there is an ongoing debate regarding the automaticity of reading within the Stroop paradigm (Besner, Stolz, & Boutilier, 1997), and it cannot be excluded that participants strategically access word meaning within the Stroop paradigm. Thus, it remains unclear whether the reported language-action compatibility effects are automatic in nature, or whether strategic processes underlie these compatibility effects. For example, it is possible that participants recognized regularities in the experimental stimuli and automatically categorized the words into up- versus down words. This categorization might subsequently result in voluntary or involuntary activation of the compatible motor response. For basic directional words (e.g., above, below), there is evidence that these words automatically activate motor processing, even if no strategic word processing takes place, such as when words are presented subliminally (Ansorge, Kiefer, Khalid, Grassl, & König, 2010). However, studies investigating less direct language-action interconnections provide evidence that these language-action compatibility effects presuppose rather deep linguistic processing. In line with the findings regarding pictures (e.g. picture of a mug) facilitating motor responses (e.g., Vainio & Mustonen, 2011), Bub, Masson, and Cree (2008) showed that words (e.g., mug) facilitate appropriate motor responses (e.g., grasping gesture) if the task demanded deeper linguistic processing (e.g., lexical decision task). If the task did not demand linguistic processing, with participants simply responding to word color, no compatibility effects were reported. This suggests that some language-action associations are driven by high-level or strategic language processing, rather than automatic language-action associations. It is of great importance for grounded language processing models to establish whether perceptual features of the entities to which words refer, influence motor responses even when strategic reading or strategic mapping of words' referent dimensions to response dimensions can be excluded as the cause of the language-action compatibility effects.

Previous studies investigating the influence on motor responses by stimuli that are not consciously accessible or influenced by strategic processing demands have typically implemented masked-priming paradigms. For example, Eimer and Schlaghecken (1998) presented a subliminal arrow (pointing to the left or right) that was followed by a target arrow (pointing to the left or right). Participants responded to the target arrow with left or right key-presses, respectively. Motor inhibition was observed in compatible prime-target conditions (e.g., masked arrow pointing left followed by target arrow pointing left) when the target followed the prime by more than 60 msec. In contrast, responses to incompatible prime-target pairs were facilitated (for a review see Eimer & Schlaghecken, 2003). The authors attributed this phenomenon to a self-inhibitory motor control system stopping our behavior being controlled by task-irrelevant stimuli. In their view, an initial automatic activation of the motor system by the masked stimulus is instantly suppressed by this inhibitory control system. Importantly, these motor inhibition effects were only reported if the prime was masked, preventing continuous updating of the stimulus information provided by the prime. In the case of supra-threshold non-masked stimuli, this suppression mechanism typically fails to inhibit the activation from the prime arrow, resulting in facilitation effects. Additionally, reversed compatibility effects have also been reported in other tasks, such as the Simon task, if the prime stimuli were masked (Treccani, Umiltà, & Tagliabue, 2006). In all of these tasks the reversed compatibility effect did not critically depend on the masked prime being subliminal. Rather, the interruption of the prime information by a mask seemed to be critical in causing the inhibitory effects (e.g., Klapp and Hinkley, 2002, Schlaghecken et al., 2008). Interestingly, these findings have been recently extended, and it has been shown that even briefly presented pictures can result in motor inhibition processes (Vainio et al., 2011, Vainio and Mustonen, 2011). Vainio et al. (2011) showed that 30 msec picture presentations showing manipulable objects (e.g., a mug with a handle pointing to the left side) interfere with subsequent motor responses. Specifically, compatible motor actions were slowed down after brief picture presentations (e.g., left hand responses were slower after the picture of a mug with a handle pointing to the left side). The authors concluded that during picture processing, motor inhibition mechanisms become active in a similar fashion to the motor inhibition effects triggered by briefly presented symbolic cues (Eimer and Schlaghecken, 1998, Eimer and Schlaghecken, 2003). In a recent study investigating the influence of briefly presented masked action words (50 msec) on motor responses, it was shown that action verbs resulted in decreased action preparation as reflected in a diminished readiness potential (Boulenger et al., 2008). The readiness potential is an electrophysiological potential that can be measured over the motor cortex during phases of movement preparation, reflecting movement planning processes in the brain (Kornhuber & Deecke, 1965). In the study by Boulenger et al. (2008) action words such as throw resulted in a diminished readiness potential during the movement preparation phase and in smaller wrist acceleration in the response execution phase than control words without any motor association. Surprisingly, random letter strings resulted in similar effects on movement kinematics and the readiness potential as action verbs, thus leaving open the question whether specific word meaning was the cause of these action influences, or whether other associations were causing these effects (for discussion see Boulenger et al., 2008).

As summarized above, direction-associated nouns referring to entities typically located in the lower or upper visual space (e.g., sky, ground), activate motor responses even in tasks that do not demand semantic processing (Lachmair et al., 2011, Thornton et al., 2012). However, it remains open whether these language-action interactions are automatically triggered during word processing, or whether they are the result of more strategic language processing. Here, we use direction-associated nouns to investigate the automaticity of language-action associations, as it has been suggested that language-action associations grounded in space are particularly strong. Vertical space is one of the most important organizational structures, and it has been argued that experiential knowledge about vertical space is already available to pre-linguistic babies (e.g. Bowerman, 1996, Lakoff and Johnson, 1980, Levinson, 2003, Needham and Baillargeon, 1993, Vosniadou and Brewer, 1992). In the present study we decreased the level of strategic language processing and investigated whether language-action compatibility effects are observed in a masked presentation paradigm where participants cannot actively access word meaning. Importantly, such a masking procedure reduces strategic language processing to a minimum (Ansorge et al., 2010, Dehaene et al., 2001, Diaz and McCarthy, 2007). Thus, if direction-associated words influence motor responses similarly to symbolic cues even under masked conditions, this would be evidence for rather close language-action interconnections that are automatically activated during very early word processing stages. In contrast, if only clearly visible words influence motor responses, this suggests that active processing of word meaning is demanded for the observation of language-action compatibility effects. In that case, the language-action connections might be less automatic than typically claimed, and rather the result of strategic language processing.