The multisensory nature of human action imagery

Motor imagery refers to the deliberate mental simulation of a movement or an action without actually executing it (Decety, 1996; Jeannerod, 1994, 2001). In the literature, traditionally the term motor imagery (e.g., Decety, 1996), rather than action imagery has been used. The term motor especially refers to the internal rehearsal of movements implying that the participant imagines her- or himself executing a given action or movement what inherently requires a representation of the body as the generator of acting forces, and not only of the effects of these forces on the external world (Jeannerod, 1994, 2001).

Regarding its sensory aspects, motor imagery often focuses on kinesthesis of a movement/action. Kinesthetic imagery requires one to ‘‘feel the movement’’ and to perceive muscle contractions (Roberts et al., 2008), e.g., how it feels to move the arms while swimming. In addition to kinesthetic imagery (KI), visual imagery (VI) is another major modality. Visual imagery requires visualization of a movement from a first-person perspective (internal VI), e.g., watching the arms moving ahead and to the side in front of oneself while swimming, or third-person perspective (external VI), e.g., watching the whole body, from above. The first-person perspective corresponds to the representation of a movement as if one is taking part in the action oneself; hence, suggesting that the movement is visualized as if one had a camera on one’s head. In contrast, the third-person perspective corresponds to the representation of the movement as if one was a spectator watching somebody (oneself or another person) perform the action.

However, those aspects commonly covered under the term motor imagery do not cover the whole experience during imagery of an action. First, additional sensory modalities may also be part of the imagery of actions, e.g., tasting the salt in the mouth while swimming in the sea. Second, action imagery not only includes the simulation of a movement itself, i.e., the motor part. Rather, it includes the simulation of concurrent sensations in different modalities. The sensations may be distal to the movement like the intended action effects in the environment e.g., imagery of the resulting melody while one imagines to play the piano, or they may be proximal sensory consequences of movement such as feeling the nubby surface of a basketball while practicing imaginary of free throws. Third, imagery of actions may also include aspects of the environment that are not directly related to the movement itself, but nevertheless important for it, e.g., the sound of music to which one is dancing. On the basis of these considerations, we prefer the term action imagery over the term motor imagery as it emphasizes that the imagined movement is accompanied by related and very specific sensory impressions in several modalities. Thus, it becomes necessary to expand the definition on motor imagery above, to capture what we mean by action imagery: Action imagery refers to the (deliberate) mental simulation of a movement without actually executing it. It can be experienced in several sensory modalities and may also address sensations which are rather indirectly related to the movement.

Over the past 30 years, the results of numerous studies have led to a picture of action imagery as a process deeply rooted in the human motor system using neural structures devoted to motor control to run off-line simulations of imagined actions (Gallese, 2005; Hardwick et al., 2018; Munzert et al., 2009; Svensson et al., 2008). There is consensus that the neural substrate of action imagery is organized around several core regions: the supplementary motor area (SMA), different sections of the premotor cortex (dPMC, vPMC), the primary motor cortex (M1), posterior parietal regions, such as the inferior (IPL) and the superior parietal lobe (SPL), the basal ganglia (BG), and the cerebellum (for reviews, see Lotze & Halsband, 2006; Munzert et al., 2009; for meta-analyses, see Hardwick et al., 2018; Hetu et al., 2013). With regard to the role of the SMA during action imagery, Kasess et al., (2008) showed a strong suppressive influence of the motor imagery condition on the forward connection between SMA and M1 what highlights the importance of the SMA for suppressing movements that are represented in the motor system but not to be performed.

When examining the activation pattern during the generation of a mental action image in more detail, it becomes apparent that the detected neural activation sites within the aforementioned brain regions are not stable, but rather modulated by several factors including the imagined perspective (Lorey et al., 2009; Ruby & Decety, 2001), the imagined effector (Lee et al., 2019; Lorey et al., 2014; Piefke et al., 2009; Stippich et al., 2002), the actual body position (Lorey et al., 2009; Vargas et al., 2004), the environmental requirements of a task (Lorey et al., 2010), an individual’s imagery capacity (Guillot et al., 2008; Lorey et al., 2011; Zabicki et al., 2019), imagery strategy and instructions (Guillot et al., 2009; Lorey et al., 2009), as well as motor expertise with the imagined action (Orlandi et al., 2020).

Most of the above-mentioned studies analyzed fMRI data by assessing overall group activity changes in brain regions in response to a stimulus or a cognitive task (Decety et al., 1994; Deiber et al., 1996; Ehrsson et al., 2003; Guillot et al., 2009; Lorey et al., 2014; Lotze et al., 1999; Porro et al., 1996; Stephan et al., 1995). In more recent years, multivariate pattern analysis (MVPA) in the form of pattern classification techniques (Haynes, 2011; Kamitani & Tong, 2005) or representational similarity analysis (RSA) (Kriegeskorte & Kievit, 2013; Kriegeskorte et al., 2008) were applied to investigate the representational content of neuronal population codes. Studies using MVPA and RSA in the context of action imagery research have shown that specific features of the imagery process can be decoded from population activity within the sensorimotor system. For example, it has been demonstrated that distributed neural response patterns in (pre-)motor and posterior parietal areas can be used to distinguish the execution of a movement from its imagery (Filimon et al., 2015; Park et al., 2015; Zabicki et al., 2017), the specific imagined action (i.e., different imagined hand actions) (Pilgramm et al., 2016; Zabicki et al., 2017), or the individual impression of vividness of the imagery experience (Zabicki et al., 2019). Zabicki et al (2017), for example, demonstrated that different imagined hand actions could be decoded significantly above chance from the spatial patterns of BOLD signals in premotor and posterior parietal cortices thereby elucidating the distinctiveness of the neural codes underlying imagery of different hand actions that are characterized by different task requirements. In a follow-up study, they also showed that spatial patterns of neural activity within the premotor and parietal area reflect the perceived vividness of imagined actions (Zabicki et al., 2019).

Although these studies show that specific features of an action image, like the imagined task or the experienced vividness, are forming specific neural patterns, there is still a lack of work that describes how sensory qualities of imagined actions that differ with respect to a number of action characteristics (e.g., action requirements, the effector used, action kinematics, or individual experience) as well as with respect to an individual’s motor experience might emerge through brain activity.

One may instruct participants to perform action imagery in very different ways. Most often, participants are instructed to perform either kinesthetic or visual imagery. Here, the imager’s attentional focus is placed on one sensory quality at a time. Instructed imagery strategies relate specifically to the sensory modality of the imagined action: for example, kinesthetic (feeling of the movement) or visual (from a first- or third-person perspective). Neurophysiological research indicates that brain activation differs between visual and kinesthetic action imagery (Guillot et al., 2009; Kuhtz-Buschbeck et al., 2003; Lorey et al., 2009). Guillot et al. (2009) showed that occipital regions (including the primary visual area and the extrastriate cortex) as well as superior parietal regions were recruited during instructed visual imagery; whereas increased activity in the inferior parietal lobe, the ventral premotor cortex, as well as the supplementary motor area were observed during instructed kinesthetic imagery. Thus, alongside overlapping activation patterns, there are also partially distinct networks for the different imagery modalities. This is supported by a study that showed differential activation patterns depending on the imagery strategy used while imagining hand movements. When participants were instructed to imagine movements from a first-person perspective including kinesthetic feelings (as if they were performing it), they showed stronger activation in left-hemisphere sensorimotor and posterior parietal structures, especially in the inferior parietal lobe, than they did during imagery trials using a third-person perspective (as if they were watching another person performing it) (Lorey et al., 2009). Thus, both studies indicate that an instructed sensory modality of an action image is associated with a specific neural activation pattern. An instruction seems to be a viable tool to focus the inner spotlight on a specific sensory quality and triggers a simulation on the basis of its neuronal representation.

Modality instructions also play an important role in mental practice. A common assumption is that imagery practice is most effective when multiple sensory modalities are employed (Cumming & Williams, 2012). For instance, a golfer may imagine feeling the club in the hands, seeing the movement of the club, hearing the club hit the ball, and smelling the grass. In most studies, however, visual and/or kinesthetic action imagery practice is investigated, as vision and kinesthesis are the most important modalities for most actions. The type of instruction makes a difference for skill acquisition, and further, modality instruction may interact with task characteristics (Féry, 2003). In a visual-spatial drawing task performance was better after visual action imagery practice than after kinaesthetic action imagery practice (Féry, 2003, Exp. 1). However, in a bimanual coordination task participants’ performance was better after kinaesthetic action imagery practice than after visual action imagery practice (Féry, 2003, Exp. 2). This might indicate that kinaesthetic imagery is particularly suited to acquire ordering and timing of movement elements (Féry, 2003) or for tasks with a strong motor component whereas visual imagery might be suited to acquire tasks with higher environmental precision needs.

Research manipulating imagery instructions shows that participants are able to flexibly perform imagery of the same action in different ways using different sensory modalities. An alternative to instructing participants is to solely instruct the action but leave the modalities they attend to up to them. Later, participants can be asked which modalities they attended to. This provides a situation more similar to the performance of actual actions in which people are rarely instructed which modalities they should attend to (an exemption may be specific instructions when learning a new skill in sports). Studies in which imagery modalities can be freely chosen show that the preferred modality is highly task-specific, may depend on expertise and that individual differences exist. This will be outlined below.

When participants are not provided with specific modality instructions, they presumably choose the modalities that are best suited to imagine the task at hand. Those modalities may be different and task-dependent. For instance, whereas participants spontaneously focused more on vision than on kinesthesis in a coloring task (Rieger & Massen, 2014), no significant differences between the focus on vision and kinesthesis was observed in a reaching task (Dahm & Rieger, 2016), and higher focus on kinesthesis than on vision was observed during proximal action elements of a dart throwing task (finger grip, the arm movement, and the release of the dart, Dahm & Rieger, 2019). The overall pattern of focus on modalities is thereby similar in imagination and execution of the same action, though sometimes the focus is weaker in imagination than in execution, at least for some modalities (Dahm & Rieger, 2016; Rieger & Massen, 2014). This might indicate that the imagined sensory experience may not be as complete as the actual experience, or it might be related to individual differences in imagery ability. Nevertheless, these results support the notion that the representation of different modalities may be highly task-specific in both, imagination and in execution of actions. In coloring, vision may be particularly important to monitor the progress of an action. In reaching, visual feedback from the hands might be particularly important when the hands reach into the area of the targets at the end of the movement but kinesthesis may be important for other parts of the movement. In darts, the hands do not reach the target of the action (the bullseye) and additionally, a specific position in which the hand movement ends does not exist. Therefore, kinesthetic/tactile feedback may be more important in darts than in reaching, which is consequently represented in imagery of playing darts. A recent published study investigated the sensory impression of free imagery of a broad variety of actions regarding to their action characteristics as well as the individual experience with performing the specific action (Krüger et al., 2020a). The results demonstrate that the sensory impression of the action image can be systematically explained by properties of the imagined action (e.g., the required precision and goal orientation of a movement or the required force) underpinning the presumption of a task-related tuning of the sensory focus. For example, goal-oriented actions that require a certain but also varying degree of precision to hit a target are imagined more visually than actions that can be described as more force-related or rhythmic.

The notion that the imagined action might be a driver of the sensory quality of the action image is also underpinned by training studies, which revealed that the learning of movements with different task demands is enhanced by either visual or kinesthetic imagery (Féry, 2003; Hardy & Callow, 1999; White & Hardy, 1995). Regarding the impact of task characteristics on the sensory modality, it was demonstrated that task characteristics interact with the imagery modality: In a visual-spatial drawing task performance was better after visual action imagery practice than after kinaesthetic action imagery practice (Féry, 2003, Exp. 1). In a bimanual coordination task, however, participants’ performance was better after kinaesthetic action imagery practice than after visual action imagery practice (Féry, 2003, Exp. 2).

Evidence from neurophysiological studies that investigated mental practice of different actions indicate differential activation depending on action characteristics. In a motor learning study, Krüger et al. (2020b) asked participants to learn different sequences of a manual pointing task either physically or mentally. This sequence-learning task required that participants generate an image of the target grid, the targets as well as their moving hand on the target grid. The task, therefore, is related to an external action goal and requires a certain precision for the successful execution. The results revealed strong activation sites in the posterior parietal and visual cortices while imagining the pointing task after training. These cortical sites are associated with visual rather than kinaesthetic imagery processes (cf. Guillot et al., 2009). Another study that investigated training outcomes (Lebon et al., 2018) examined associations between fMRI activation measured prior to action imagery practice and kinesthetic or visual imagery strategies and their effect on motor performance in a finger sequence task without any external target that had to be hit. Here, participants were instructed to move or to imagine moving the fingers of the right hand in a specific order without any target. They observed that especially high kinesthetic imagery vividness assessed before training (compared to visual imagery) and the IPL activation during imagery predicted high motor performance after training, thereby demonstrating the importance of high kinesthetic vividness for improving motor execution in the respective task. Both studies deliver initial indications regarding the relation between characteristics of the task (e.g., target vs. no target), imagery modality and a specific neural activation pattern. Thus, task-dependent usefulness of sensory information might also shape neural activation during imagery practice. However, a systematic investigation of such a relationship is still pending.

Most studies that investigate sensory features of action images focus on visual and kinesthetic sensory qualities. However, studies investigating performing music in one's mind and the use of imagery techniques to learn musical instruments include the acoustic modality. In general, many professional musicians use imagery techniques to rehearse various aspects of a musical piece, e.g., difficult parts of an already executed musical passage (Lotze et al., 2003). Several studies underpinning the usefulness of mental imagery in music are available on learning the trombone (Ross, 1985), piano (Bernardi et al., 2013), guitar (Theiler & Lippmann, 1995) and singing (DeSantis et al., 2021; Theiler & Lippmann, 1995). In these studies, the created action image usually includes the auditory modality (e.g., Ross, 1985). On a neurophysiological level, work that examined the singing of an Italian aria (Kleber et al., 2007) showed an involvement of the premotor cortex, SMA, and secondary auditory areas lying within the superior temporal gyrus during imagined singing. Especially the involvement of superior temporal gyrus in auditory imagery has been suggested to reflect a simulation of sound contributing to the subjective auditory experience.

In sum, all presented studies suggest that the imagined action, its characteristics and its effects mediate to a certain extent the sensory quality of the imagery process. The visual, kinesthetic and acoustic imprint are determined to a certain degree by action characteristics and are also reflected in the neural representations of the imagined actions.

In the former sections, it became apparent that the given instruction as well as the imagined task are potent drivers for differential sensory imagery experiences. However, individual determinants such as motor expertise have also a substantial influence on the quality of the imagination process. In this vein, it has been shown that high- compared to low-level athletes reported more sensory vivid imagery (Eton et al., 1998; Isaac & Marks, 1994).

Action imagery is based on contents of (motor) memory (Annett, 1996) or internal models for the action (Davidson & Wolpert, 2005). Experts for an action should, therefore, be able to use those pre-existing internal representations for action imagery but novices should not. Several differences distinguish experts and novices. For example, hierarchically structured representations are similar between experts, but not between novices (Schack & Mechsner, 2006), indicating that expertise influences the structure of action representations. Experts in typing develop task-specific representations that are not observed in non-experts (Beilock & Holt, 2007; Rieger, 2004) and presumably have more precise internal models (Rieger, 2012), which might allow for a richer and more (sensory) detailed imagination of the action. Consequently, the duration of action imagery compared to action execution depends on expertise (Reed, 2002). Familiarity with an action may not only be relevant in actions, in which training is highly specialized, but also with simple everyday actions. When participants are asked to imagine actions, which only slightly differ from everyday actions, they may not be imagined adequately. For instance, effort applied in executed actions to compensate for additional weight may not be imagined spontaneously (Cerritelli et al., 2000; Decety et al., 1989) and familiarity with an action influences imagination durations (Rieger, 2012). One may therefore assume that differences in the imagination of different sensory modalities exist depending on experience and expertise. However, there is little research investigating this issue. Some studies indicate that the use of different modalities in imagination between experts and novices is not as different as one might think. For instance, no difference in the focus on kinesthesis and vision was found between experts and novices in imagery of dart throwing (Dahm & Rieger, 2019). Here, one explanation might be that the use of modalities in imagery is largely determined by characteristics of the task itself and not on one’s experience with it. A recent study that investigated a broad set of different actions (Krüger et al., 2020a) in a larger collective of people, however, revealed that the sensory impression of the action image can be systematically explained by a person’s idiosyncratic experience: each participant generates a more kinesthetic and a more vivid action image especially for those actions with which she or he has the most experience. In this vein, Lotze et al. (2003) investigated experienced musicians compared to amateurs during imagined violin play and demonstrated that the vividness of movement imagery was higher in the expert group and that rhythm and pitch imagination scores correlating positively with lifetime and weekly training. These results underpin the notion the sensory modality and vividness of the image is not just a matter of instruction and task, but has its roots in the imager’s individual memory that encodes for specific actions and the related sensory effects representing the individual experience with a specific action that enriches action representations.

Supporting this, a study by Fourkas et al. (2008) investigated the corticospinal excitability in forearm and hand muscles via single-pulse transcranial magnetic stimulation during mental imagery of a tennis forehand, table tennis forehand, and a golf drive in expert tennis players and novices. In tennis experts, they showed increased corticospinal facilitation during imagery of tennis, but not golf or table tennis. Corticospinal facilitation of novices was not modulated across sports. On a subjective level, the tennis experts differed only in the tennis imagery condition from novices in the ability to form proprioceptive images and to consider the tool as an extension of the hand suggesting a key role of long-term experience in modulating sensorimotor action representations that drive the (sensory) quality of the formed image.

High imagery ability is characterized by easy generation, maintenance, and control of the imagined action. It is also associated with the subjective experience of vividness, that is, clearer, richer and entailing several sensory modalities (e.g., McAvinue & Robertson, 2008). In the nineteenth century, Galton (1880) already stated that the “detail and clarity with which individuals experience mental imagery” (p. 304) involves a difference gradient across individuals. Thus, the capacity to generate mental images is not an undifferentiated ability. The literature demonstrates variability in the individual capacity to generate action images (Cumming & Eaves, 2018; Pearson, 2019): Imagery performance differs inter- and intra-individually with respect to not only the preferred imagery perspective but also the imagery modality used (see, for a review, Moran et al., 2012). Furthermore, there are differences in imagery ability across the lifespan: young adults were significantly more accurate and rated their imagery significantly more vivid than children indicating that action imagery ability continues to develop into adulthood (Fuchs et al., 2020) and declines again in adults 70 years and older (Schott, 2012). Vividness of imagery has also a moderating effect on motor learning (e.g., Isaac, 1992): Participants who report more vivid imagery show greater performance improvements. Thus, individual differences in imagery ability relate to the effectiveness of action imagery practice (Isaac & Marks, 1994; Ruffino et al., 2017).

On the neurophysiological level, Guillot et al., (2009) compared via functional magnetic resonance imaging the pattern of cerebral activations of skilled and unskilled imagers during execution and imagery of a sequence of finger movements. Between-group comparisons revealed that participants with high imagery ability activated more the parietal and ventrolateral premotor regions, which are known to play a critical role in the generation of mental action images. By contrast, participants with low imagery abilities recruited the cerebellum, orbito-frontal and posterior cingulate cortices. The authors concluded that participants with high imagery abilities activate posterior parietal and premotor regions to a greater extent than those with low imagery abilities.

Inter-individual differences in imagery ability and imagery experience, not only in action imagery ability, relate to differences in neuronal activation pattern but also to differences in brain structure. Regarding visual imagery (not in the action context), Bergmann et al. (2016) revealed inter-individual differences in the neuronal substrate that co-varies with different aspects of the imagery experience. They observed a negative relationship between primary visual cortex (V1) surface size and sensory imagery strength, but found positive relationships between V1 surface size and imagery precision. Hence, individuals with a smaller V1 tended to have stronger, but less precise imagery. Their findings revealed the importance of V1 layout in shaping visual imagery vividness. A further study, that demonstrated inter-individual differences in imagery vividness also focused on the question whether differences in brain structure predict differences in subjective imagery vividness, here, however, in the context of auditory images. The results revealed that auditory imagery varies considerably across the investigated individuals and this variability relates to differences in the local structure of gray matter. Vividness of auditory imagery correlated with gray matter volume in the supplementary motor area (SMA), parietal cortex, medial superior frontal gyrus, and middle frontal gyrus (Lima et al., 2015). All studies underpin that brain substrate on a functional and structural level relates to the subjective ability to form rich and vivid (action) images (cf. Pearson, 2019) and highlight the role of perceptual–motor interactions for processing internally generated sensory information (cf. Lima et al., 2015; Lotze et al., 2003).

Besides inter-individual differences, imagery capacity and perceived vividness might also vary intra-individually from time to time and from image to image. Lorey et al. (2011) investigated the perceived imagery vividness during imagery of hand movements in a trial-by-trial approach with parametric functional magnetic resonance imaging. The results showed that that the perceived trial-to-trial vividness of action imagery is positively associated with neural activity within sensorimotor areas. In addition to vividness-related amplitude modulations, vividness of action images is also reflected by neural pattern in sensorimotor areas (Zabicki et al., 2019). Thus, it could be shown that similarly rated trials evoked more similar neural patterns in the left vPMC and right IPL. Furthermore, imagined actions accompanied by higher vividness ratings were significantly more distinguishable and elicited an action-specific neural pattern within the left ventral section of the premotor area as well as the right superior parietal lobe. Less vividly imagined movements, however were noisier and elicited a less action-specific noisier neural pattern, resulting in lower levels of decoding accuracy in these areas. In light of these results, the idea emerged that vividness ratings may reflect the multisensory (i.e., kinesthetic and visual) distinctiveness of an imagined action as an increased vividness rating is accompanied by a more distinct neural representation for different imagined actions. Both studies imply that functional measurements of neural activity in sensorimotor areas represent differences in subjective experience of action imagery processing. Increased vividness is accompanied by higher neural activation in sensorimotor areas (Lorey et al., 2011) but also by more distinct neural motor representation for imagined actions (Zabicki et al., 2019) reflecting the degree of success regarding the kinesthetic (and possibly visual) retrieval of action representations.