Creating number semantics through finger movement perception
Introduction
How conceptual knowledge emerges, is represented in the brain, and exchanged with other members of the species has long been a matter of debate. After several decades dominated by the computational metaphor in which the nature of cognitive processes and conceptual knowledge was discussed independently from the brain and its functioning, the theory of embodied semantics proposed that concepts are built and represented within the same sensory–motor cerebral systems on which the enactment of these concepts rely (Gallese and Lakoff, 2005, Lakoff and Johnson, 1999, Martin, 2007). For example, the concept of “grasping” would be represented in the sensory–motor areas underlying grasping actions (Glover & Dixon, 2002). This theory is supported by neurophysiological data showing that “mirror” neurons originally discovered in the ventral premotor cortex (area F5c) of monkeys discharge not only when a monkey performs a motor action but also when it observes another monkey (or human being) performing the same action (Di Pellegrino, Fadiga, Fogassi, Gallese, & Rizzolatti, 1992). Rostral to this mirror system lies a cortical area controlling the orofacial musculature and involved in communicative acts, that may be considered as the homologue of Broca’s area in the human brain (Petrides, Cadoret, and Mackey, (2005) for a discussion, see Toni, de Lange, Noordzij, and Hagoort (2008)). Together with this area, the mirror motor mechanism is viewed as the substrate on which communication and language evolve, for it represents the link between the sender and the receiver of a message (Arbib, 2005, Rizzolatti and Arbib, 1998). A similar motor resonance system exists in the human premotor cortex and constitutes the neural substrate of a corresponding mechanism recording the observed actions in the observer’s motor repertoire (Rizzolatti & Craighero, 2004). As a part of this system, the posterior parietal cortex is also involved in observing object-related actions; interestingly, observing an action mimicked without an object also induces a motor resonance in the premotor cortex (Buccino et al., 2001). Finally, it has been shown that observing a motor action fully engages the mirror system, especially the left premotor area, when the acting model is human but not when a non-biological actor performs the same action (Tai, Scherfler, Brooks, Sawamoto, & Castiello, 2004).
Sensory–motor networks could thus support conceptual knowledge building, which could give rise to semantic-to-motor interactions. Motor programs associated with objects are indeed accessed by language, and naming a feature of an object with a direct relation to a sensory–motor transformation can influence an action (Glover & Dixon, 2002). For example, the meaning of words printed on objects being reached for influences the kinematics of reaching and grasping movements (e.g., largest movements when the word “long” is printed on the object, irrespective of its real size; Gentilucci & Gangitano, 1998), processing action verbs impairs motor performance (e.g., by delaying the onset of concurrent reaching movements) whereas processing nouns does not (Boulenger et al., 2006), and processing action verbs but not concrete nouns is impaired in Parkinson’s disease (Boulenger et al., 2008). There is also an increasing body of findings showing that processing language triggers brain activity in sensory–motor areas (Aziz-Zadeh et al., 2006, Pulvermüller, 2005, Tettamanti et al., 2005). To date, semantic-to-motor interactions have mainly been described when processing images, words, sentences and concepts referring explicitly to concrete objects and actions or, at least, one of their parameters, and there have only been very few reports of implicit semantic knowledge of objects affecting hand movements (e.g., an increase in grip aperture following the presentation of words referring to large objects; Glover, Rosenbaum, Graham, & Dixon, 2004). That the sensory–motor system may be invoked for accessing or generating the semantics of words referring to actions (Hauk, Jonhsrude, & Pulvermüller, 2004) or manipulable objects (Almeida, Mahon, Nakayama, & Caramazza, 2008) is thus largely admitted, even by opponents of a general embodiment of cognition (e.g., Toni et al., 2008), but how it could account for abstract concept creation and communication remains highly controversial. Interestingly, semantic-to-motor interactions have been observed with the abstract concept of numerical magnitude. For example, processing the magnitude of Arabic numbers was found to interfere with finger movements, leading to compatibility effects when responses required a grip closing or opening: grip closing was initiated faster in response to small numbers, whereas grip opening was initiated faster in response to large numbers (Andres, Davare, Pesenti, Olivier, & Seron, 2004). Similar results have been reported for object-directed hand grips (Lindemann, Abolafia, Girardi, & Bekkering, 2007); in this case, the effect of number magnitude is higher during the first stages of object reaching movements and decreases as the hand gets closer to the object to grasp (Andres, Ostry, Nicol, & Paus, 2008). Recently, we showed that processing numerical information affected the judgment of actions toward objects, even when no actual action was required, but not a perceptual judgment per se (Badets, Andres, Di Luca, & Pesenti, 2007). Similarly, numerical magnitude was found to interact with an action judgment made on objects but not with a color judgment on the same objects (Chiou, Chang, Tzeng, & Wu, 2009).
So far, interactions have mainly been described from semantics to sensory–motor processes. From an evolutionary and developmental perspective, the functional link should, however, be in the other direction, from the sensory–motor system to semantics. Moreover, the previously reported numerical-to-motor interactions may emerge from a general magnitude system processing not only numbers but also any kind of magnitude (e.g., time and space; Dormal and Pesenti, 2007, Dormal et al., 2006, Walsh, 2003). In this study, we first established the effect of processing an abstract semantic concept on grip movement perception. We then reversed the interaction, and tested whether the perception of motor actions would influence the processing of an abstract semantic concept. We did this by assessing whether the perception of a movement mimicking an object-related action would influence number processing. We also tested the specificity of this putative functional link between the sensory–motor system and semantics by contrasting biological hand movements and identical non-biological fake hand movements. If number semantics truly relates to specific hand movements, we predict that motor-to-semantic interactions should only appear when a biological hand movement mimics a meaningful object-directed action.
To test these hypotheses, we designed two pairs of experiments. The first pair of experiments assessed semantic-to-motor interactions; the participants were presented with two Arabic digits to remember and had to recall one of them as soon as they perceived a hand movement. The second pair of experiments tested motor-to-semantic interactions; the participants had to name one of two Arabic digits after the presentation of hand movements. The hand movements were biological (Experiments 1 and 3) or non-biological (Experiments 2 and 4), and mimicked either a precision or a power grip closing or opening. Precision and power grips were chosen as they represent the main classes of grasping behavior in humans (Castiello, 2005, Napier, 1956, Susman, 1998). If space, numbers, and motor processes are represented within a general common magnitude system, interactions between hand-movement magnitude and number magnitude should be present in all experiments and conditions, irrespective of the nature of the movement (biological or fake hand), the type of grip (precision or power grip), and the type of object-directed action (opening or closing). Participants should answer faster to a small number after the presentation of a closing movement (because a closing movement would activate the meaning “small”), and to a large number after the presentation of an opening movement (because an opening movement would activate the meaning “large”). In contrast, if a common representation for number processing and biological movement perception is restricted to an object-related action, then interferences should only be observed for specific biological hand-movement conditions. When numerical magnitude is presented first, processing a large number should interfere with precision grip perception and processing a small number with power grip perception, as suggested by previous studies on semantic-to-motor interactions (Andres et al., 2004, Andres et al., 2008, Badets et al., 2007, Chiou et al., 2009; Lindemann et al., 2007). However, when movement perception comes first, both precision and power grips should interfere with large number processing but only when they mimic a closing movement, as observed during usual grasping, and not when they mimic an opening movement that is actually not related to object prehension; moreover, non-biological grip movements should never affect number processing.
Section snippets
General design
Two pairs of experiments were designed; the first pair (Experiments 1 and 2) tested semantic-to-motor interactions, and the second pair (Experiments 3 and 4) motor-to-semantic interactions. In Experiment 1, a pair of small or large Arabic digits, one odd and one even, was presented and followed by an imperative stimulus consisting of a finger movement mimicking an opening or closing movement. The participants had to recall the odd or the even number depending on the type (i.e., opening vs.
Global analysis
This analysis revealed the main effects of directions [semantic-to-motor: 568 ± 103 ms, motor-to-semantic: 521 ± 70 ms; F(1, 152) = 11.97, p < .001], hands [biological: 529 ± 73 ms, fake: 561 ± 105 ms; F(1, 152) = 5.54, p < .02], and magnitude [small: 542 ± 94 ms, large: 548 ± 95 ms; F(1, 152) = 5.26, p < .02]. Directions and hands interacted [F(1, 152) = 6.27, p < .01], with the latencies being significantly slower for the non-biological hands in the motor-to-semantic experiment compared to all the other conditions (about 75 ms
Discussion
The present study constitutes the first evidence that the perception of specific finger movements interferes with numerical magnitude processing. It shows that response latency increases when a grip closing preceded a response to a large but not a small number. This was observed with biological hands only. We suggest that this effect stems from the automatic activation of a code system sharing a dimension of magnitude across numbers, finger movements and, implicitly, objects, because fingers
Competing interests
The authors declare that they have no competing financial interests.
Acknowledgements
This study was supported by a Marie Curie Research Training Network from the European Community (MRTN-CT-2003-504927, Numbra project), and by Grant 01/06-267 from the Communauté Française de Belgique – Actions de Recherche Concertées (Belgium). We thank Nathalie Lefevre for her statistical advice. A.B. is a tenured researcher at the National Center for Scientific Research (France), and M.P. is a research associate at the National Fund for Scientific Research (Belgium).
References (46)
- et al.
Time course of number magnitude interference during grasping
Cortex
(2008) - et al.
Congruent embodied representations for visually presented actions and linguistic phrases describing actions
Current Biology
(2006) - et al.
Cerebral activations during number multiplication and comparison: A PET study
Neuropsychologia
(1996) - et al.
Numerosity–duration interference: A Stroop experiment
Acta Psychologica (Amst)
(2006) - et al.
Somatotopic representation of action words in human motor and premotor cortex
Neuron
(2004) - et al.
Language within our grasp
Trends in Neuroscience
(1998) Hand function and tool behavior in early hominids
Journal of Human Evolution
(1998)- et al.
The human premotor cortex is ‘mirror’ only for biological actions
Current Biology
(2004) - et al.
Language beyond action
Journal of Physiology – Paris
(2008) A theory of magnitude: Common cortical metrics of time, space and quantity
Trends in Cognitive Science
(2003)
Unconscious processing dissociates along categorical lines
Proceedings of the National Academy of Sciences
Number magnitude and grip aperture interaction
Neuroreport
From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics
Behavioral and Brain Sciences
Number magnitude potentiates action judgment
Experimental Brain Research
A fronto-parietal circuit for object manipulation in man: Evidence from an fMRI-study
European Journal of Neuroscience
Word processing in Parkinson’s disease is impaired for action verbs but not for concrete nouns
Neuropsychologia
Cross-talk between language processes and overt motor behavior in the first 200 ms of processing
Journal of Cognitive Neuroscience
Action observation activates premotor and parietal areas in a somatotopic manner: An fMRI study
European Journal of Neuroscience
The neuroscience of grasping
Nature Reviews Neuroscience
The common magnitude code underlying numerical and size processing for action but not for perception
Experimental Brain Research
FMRI evidence for a “parietal reach region” in the human brain
Experimental Brain Research
Where and when to pay attention: The neural systems for directing attention to spatial locations and to time intervals as revealed by both PET and fMRI
The Journal of Neuroscience
Cited by (66)
A common neural substrate for number comparison, hand reaching and grasping: A SDM-PSI meta-analysis of neuroimaging studies
2022, CortexCitation Excerpt :Secondly, we shed light on the issue of whether number and the hand action of reaching and grasping recruit overlapping brain areas (Aim 2). On the basis of previous behavioural studies showing the interplay between symbolic number processing and processes related to goal-directed hand actions (e.g., Andres et al., 2004; Andres et al., 2008; Badets et al., 2016; Badets & Pesenti, 2010, 2011; Ranzini et al., 2011), we embrace the idea of (partially) shared mechanisms underlying the hand action of reaching and grasping and number processing, and we argue that observing spatial overlap provides a stronger case for this hypothesis. In line with the ATOM theory (Walsh, 2003) and its recent extention (Cona, Weiner, Scarpazza, 2021), we predicted to observe such overlap primarily within the parietal lobes, in those areas which have been systematically involved in both the hand action of reaching and grasping and in number processing (e.g., IPS, PSPL).
The effect of multimodal information on children's numerical judgments
2019, Journal of Experimental Child PsychologyCitation Excerpt :This work demonstrates that young infants can track small sets cross-modally, whereas adults can make numerical judgments abstractly and independent of modality. However, developmental studies focusing on cross-modal numerical abilities during childhood have produced mixed results (Badets & Pesenti, 2010; Barth et al., 2005, 2007; Cote, 2015; Jordan & Baker, 2010; Mix et al., 1996). These studies have used critically different paradigms, resulting in noninterpretable findings leaving it unclear whether children’s numerical abilities are actually compromised in the context of cross-modal numerical stimuli or whether these performance costs are task specific.
Finger versus pencil: An eye tracking study of learning by drawing on touchscreens
2024, Journal of Computer Assisted LearningNear-optimal integration of the magnitude information of time and numerosity
2023, Royal Society Open ScienceMulti-dimensional sensorimotor grounding of concrete and abstract categories
2023, Philosophical Transactions of the Royal Society B: Biological Sciences