Elsevier

Neurocomputing

Volume 70, Issues 13–15, August 2007, Pages 2194-2203
Neurocomputing

EEG evidence for mirror neuron activity during the observation of human and robot actions: Toward an analysis of the human qualities of interactive robots

https://doi.org/10.1016/j.neucom.2006.02.024Get rights and content

Abstract

The current study investigated the properties of stimuli that lead to the activation of the human mirror neuron system, with an emphasis on those that are critically relevant for the perception of humanoid robots. Results suggest that robot actions, even those without objects, may activate the human mirror neuron system. Additionally, both volitional and nonvolitional human actions also appear to activate the mirror neuron system to relatively the same degree. Results from the current studies leave open the opportunity to use mirror neuron activation as a ‘Turing test’ for the development of truly humanoid robots.

Introduction

A great deal of robotics research has recently investigated issues surrounding the creation of robots that are able to socially interact with humans and other robots [7], [8], [36]. Some recent work has focused on developing anthropomorphic “humanoid” robots for optimal social-communicative interactions with humans. Models have been developed stressing the importance of different processing strategies (i.e., purely sensory vs. embodied vs. a combination of both) [6], [24], [38], [43]. As a result, several of these models draw on findings in the fields of developmental psychology [23], [35] and cognitive neuroscience [7], [32], [37], [38]. Perhaps the most influential contribution to this area derives from the discovery of an action observation/execution network known as the ‘mirror neuron’ system.

Single unit studies indicate that neurons in area F5 of the macaque premotor cortex, which are indistinguishable from neighboring neurons in terms of their motor properties, also discharge in response to observed actions [17], [57]. That is, when a monkey observes another individual performing an action that is part of its own motor repertoire, these ‘mirror neurons’ fire, creating the basis for a neural observation/execution matching system. These single unit studies also show that the macaque mirror neuron system is selective for object-directed actions [21]. Functionally, it has been suggested that this system may allow the monkey to perform both an on-line automatic execution of the action and an off-line internal simulation of the observed action. Such a simulation may play a critical role in one's ability to understand the movements of other individuals, an ability that is critical for social interaction [22] and particularly relevant for the development of “humanoid” interactive robots.

Individual neurons have not been directly studied in the same way in humans. However, the existence of an analogous system in the homologous brain region (Broca's area/Brodmann's area 44) has received strong support from multiple indirect population-level measures, including transcortical magnetic stimulation (TMS) [18], functional magnetic resonance imaging (fMRI) [30] and electroencephalography (EEG) [15], [39], [44], [51], [52]. Though this system seems to be functionaly and anatomically analogous, one notable difference between the monkey F5 system and the human mirror system is that the monkey system only responds to actions with target objects whereas the human system will respond, albeit to a lesser extent, to pantomimed [9], [40] and intransitive actions [18]. Additionally, recent work has uncovered populations of neurons with similar properties in the parietal cortex [9], [40], as well as the superior temporal sulcus [12], [47]. These results suggest that the frontal mirror neuron system may be part of a broader action observation/execution network [19], [45].

Previous studies in our laboratory [1], [44], [51] and those of others [39], [15] have investigated the mirror neuron system in humans through analysis of EEG mu frequency band oscillations. At rest, sensorimotor neurons spontaneously fire in synchrony [23], leading to large amplitude EEG oscillations in the 8–13 Hz (mu) frequency band. When subjects perform an action, these neurons fire asynchronously, thereby decreasing the power of the mu-band EEG oscillations [49]. Over the past 50 years there have been several theories relevant to the function of the mu rhythm [for a review 50]. Most recently, results of several studies have uncovered various properties of mu suppression that directly link it to the frontal mirror neuron system. First, mu power recorded from electrodes at scalp locations C3 and C4 is reduced by self-initiated movement and observed movement [3], [15], [23], [45]. Importantly, similar to mirror neuron activity, the mu wave does not respond to non-biological directional motion such as bouncing balls [1], [44]. Furthermore, analogous to previous fMRI studies of the mirror neuron system [9], the presence of an object increases mu wave suppression as compared to pantomimed actions [40].

Since the mu rhythm is generated by activity in sensorimotor areas [23], and mirror neurons are located in premotor cortex [17], it has been hypothesized that the mu rhythm may specifically index downstream modulation of primary sensorimotor areas by mirror neuron activity [40]. Additionally, the frontal mirror neuron system is the only network in the region of sensorimotor cortex that has been identified as responding to observed hand actions. Taken together, these results suggest that mu wave suppression to observed actions can be used as a selective measure of activity of this system.

The discovery of mirror neuron activity in humans has resulted in extensive experimental research as well as theoretical papers on the role of the mirror system in human imitative [57], social [22], emotional [13], and cognitive [54], [56] behaviors. Mirror neuron activity has also been implicated in disorders of social cognition (i.e. autism spectrum disorders) [16], [42], [44], [62], [64].

Based on the extant findings, researchers have begun to develop dynamic neural network models based on the human mirror neuron system to be used with interactive robots. Of particular relevance to the current study is research conducted by Ito and Tani [32], who developed a neural network model for deferred imitation for use by an entertainment robot (Sony's ‘QRIO’). The purpose of this neural network was to create a more naturalistic human–robot interaction by having the robot mimic the synchronization patterns naturally present in human interactions and, therefore, increase the time humans spent interacting with the robot. A recent review [7] describes the development of a “socially intelligent” robot (Leonardo). The robot's physical and cognitive architecture are based on previous research in simulation and the mirror neuron system, which enable it to not only imitate, but also to understand the human interactant's emotions, based on facial expressions. Although not directly evaluated in these experiments, the implication is that the human mirror neuron system may be activated as a result of the human interactant anthropomorphizing these robots. Indeed, by activating the human mirror neuron system humanoid robots could potentially tap into the powerful social motivation system inherent in human life, which could lead to more enjoyable and longer lasting human–robot interactions.

Ito and Tani's as well as Breazeal's work address two important aspects of a robot that might influence the activation of the mirror neuron system in a human interactant: (1) the temporal aspects of social interaction (e.g. simple dance-like imitation sequences); (2) the cues used by humans to determine when and how spontaneous switching of roles (e.g. modeler vs. imitator) can and should occur during a social interaction. Research with human subjects has shown that these are important aspects of behavior during social interactions [2], [11], [58]. Other aspects of a stimulus shown to be important for the perception of human or biological movement include physical shape [27], [46], the temporal properties of the physical body [53], the temporal and spatial properties of its motion [4], [5], [14], [25], [28], and the topography of the motion [10], [41]. Furthermore, the volitional nature of the actions may also be an important factor in the perceived ‘humanness’ of a stimulus [55]. In fact, a recent study by Iacoboni and colleagues [29] indicated that the activity in the mirror neuron system was modulated based on both whether the observed action was embedded in a context and the specific intention of that action. It has been proposed by Gallese [20] that the observation/execution matching system provided by the mirror neuron system may allow humans to have a “shared manifold” for more internal states such as goals and intentions.

The goal of this study was to characterize the specific properties of stimuli that produce activation of the human mirror neuron system, with an emphasis on properties that are critically relevant for the perception of humanoid robots. Since it is unclear whether robot actions suppress the mu wave, Experiment 1 determined whether the observation of an action performed by a robot hand with human-like characteristics (i.e., four fingers and opposable thumb) is sufficient to activate this biological action perception system. Since the presence of a target object appears to modulate the mu rhythm when observing human actions [40], Experiment 1 also examined the capacity for robot actions with and without target objects to differentially activate the human mirror neuron system. This study has clear implications for both the neural basis of robot action perception and the flexibility of the mirror neuron system. Experiment 2 examined the influence of volition during the observation of human actions. If the mu wave were differentially modulated by volitional actions, this would provide further support for the role of mirror neurons in the understanding of the intentions of others and thus the creation of a theory of other minds. This study may also lead to a better understanding of the critical features necessary for successful development of an interactive humanoid robot.

Section snippets

Subjects

Our original sample consisted of 20 undergraduate students, recruited through the UCSD Psychology Department Subject Pool, who received class credit for their participation. Three subjects’ data were excluded from analysis due to technical problems with the EEG apparatus, resulting in a final sample of 17 subjects (13f, 4 m) ages 18–23 (M=19.6, SD=1.50). Four subjects were left-handed based on a self-report measure. The project protocol was reviewed and approved by the UCSD Human Research

Subjects

Our sample consisted of 20 subjects (13f, 7 m) ages 18–39 (M=21.2, SD=4.43). Three subjects were left-handed. Subjects were recruited through the UCSD Psychology Department Subject Pool and received class credit for their participation. This project was reviewed and approved by the UCSD Human Research Protections Program, and all subjects gave written consent.

Procedure

EEG data were collected during three conditions: (1) watching video of a hand moving volitionally (Fig. 1, Experiment 2a). Subjects viewed

Acknowledgement

We would like to thank Brian P. Jacoby for his assistance with data analysis. J.P.M. was supported by a research fellowship from the M.I.N.D. Institute and a traineeship from NSF grant DGE-0333451 to GW Cottrell. In addition to being presented at the 3rd International Conference for Development and Learning the data presented in this manuscript were presented in San Diego at the 34th annual meeting of the Society for Neuroscience, October 2004.

Lindsay M. Oberman is a doctoral student in Experimental Psychology at the University of California, San Diego. She earned her bachelor's degree with a double major in Neuroscience and Behavioral Biology, and Psychology at Emory University in 2002, and her master's degree in Experimental Psychology at UCSD in 2004. Her research focuses on behavioral and electrophysiological investigations of the mirror neuron system in typical adults and children as well as individuals diagnosed with autism

References (64)

  • S.D. Muthukumaraswamy et al.

    Primary motor cortex activation during action observation revealed by wavelet analysis of the EEG

    Clin. Neurophysiol.

    (2004)
  • S.D. Muthukumaraswamy et al.

    Mu rhythm modulation during observation of an object-directed grasp

    Cogn. Brain Res.

    (2004)
  • S. Nolfi

    Power and limits of reactive agents

    Neurocomputing

    (2002)
  • L.M. Oberman et al.

    EEG evidence for mirror neuron dysfunction in autism spectrum disorders

    Cogn. Brain Res.

    (2005)
  • G. Pfurtscheller et al.

    Evaluation of event-related desynchronization (ERD) preceding and following self-paced movement

    Electroencephalogr. Clin. Neurophysiol.

    (1979)
  • G. Pfurtscheller et al.

    Foot and hand area mu rhythms

    Int. J. Psychophysiol.

    (1997)
  • J.A. Pineda

    The functional significance of mu rhythms: translating “seeing” and “hearing” into “doing.”

    Brain Res. Rev.

    (2005)
  • L. Pizzamiglio et al.

    Separate neural systems for processing action- or non-action-related sounds

    Neuroimage

    (2005)
  • G. Rizzolatti et al.

    Language within our grasp

    Trends Neurosci.

    (1998)
  • A.P. Saygin et al.

    Pragmatics in human–computer conversation

    J. Pragmat.

    (2002)
  • Y.F. Tai et al.

    The human premotor cortex is mirror only for biological actions

    Curr. Biol.

    (2004)
  • H. Theoret et al.

    Impaired motor facilitation during action observation in individuals with autism spectrum disorder

    Curr. Biol.

    (2005)
  • M.E. Villalobos et al.

    Reduced functional connectivity between V1 and inferior frontal cortex associated with visuomotor performance in autism

    Neuroimage

    (2005)
  • E.L. Altschuler, A. Vankov, V. Wang, V.S. Ramachandran, J.A. Pineda, Person see, person do: human cortical...
  • J.B. Asendorpf et al.

    Self-awareness and other-awareness. II: mirror self-recognition, social contingency awareness, and synchronic imitation

    Dev. Psychol.

    (1996)
  • B.I. Bertenthal et al.

    Perception of biomechanical motions by infants: implementation of various processing constraints

    J. Exp. Psychol. Hum. Percept. Perform.

    (1987)
  • A.E. Booth et al.

    Perception of the symmetrical patterning of human gait by infants

    Dev. Psychol.

    (2002)
  • C. Breazeal et al.

    Learning from and about others: towards using imitation to bootstrap the social understanding of others by robots

  • G. Buccino et al.

    Action observation activates premotor and parietal areas in somatotopic manner: an fMRI study

    Eur. J. Neurosci.

    (2001)
  • G. Buccino et al.

    Neural circuits involved in the recognition of actions performed by nonconspecifics: an fMRI study

    J. Cogn. Neurosci.

    (2004)
  • J.N. Cappella

    Behavioral and judged coordination in adult informal social interactions: vocal and kinesic indicators

    J. Pers. Soc. Psychol.

    (1997)
  • D.P. Carey et al.

    Recognizing, understanding, and producing action

  • Cited by (182)

    • Scoping review of the neural evidence on the uncanny valley

      2023, Computers in Human Behavior Reports
    • Being watched by a humanoid robot and a human: Effects on affect-related psychophysiological responses

      2022, Biological Psychology
      Citation Excerpt :

      Ascribing such agency to a robot could, in turn, be an indication of people’s tendency to ascribe mental states to robots (Gray, Gray, & Wegner, 2007). Indeed, previous studies in HRI have provided evidence that people tend to perceive humanlike characters, sometimes even mind, in robots (Airenti, 2015; De Graaf & Malle, 2019; Gazzola, Rizzolatti, Wicker, & Keysers, & 2007; Krach et al., 2008; Oberman, McCleery, Ramachandran, & Pineda, 2007; Thellman, Silvervarg, & Ziemke, 2017). In the present study, we wanted to directly investigate whether the observers’ experience of the state of a humanoid robot’s “mind”, specifically, the experience of the robot watching the observer, has an impact on the psychophysiological responses to the robot’s direct and averted gaze.

    • Addressing joint action challenges in HRI: Insights from psychology and philosophy

      2022, Acta Psychologica
      Citation Excerpt :

      This general strategy of taking advantage of human preferences, inclinations, or curiosity is not restricted to their physical features. Several labs (see, for instance, Breazeal, 2002; Craig et al., 2010; Kishi et al., 2013; Oberman et al., 2007; Wendt et al., 2008; Wendt & Berg, 2009) are, for instance, equipping robots with different emotional expressions in order to prompt empathy or pro-social attitudes into the human agent, so robots “could potentially tap into the powerful social motivation system inherent in human life, which could lead to more enjoyable and longer-lasting human-robot interactions” (Oberman et al., 2007, p. 2195). For instance, Craig et al. (2010) have demonstrated that facial expressions of the BERT humanoid robots can elicit the same neuronal responses as human facial expressions in an emotion recognition task.

    View all citing articles on Scopus

    Lindsay M. Oberman is a doctoral student in Experimental Psychology at the University of California, San Diego. She earned her bachelor's degree with a double major in Neuroscience and Behavioral Biology, and Psychology at Emory University in 2002, and her master's degree in Experimental Psychology at UCSD in 2004. Her research focuses on behavioral and electrophysiological investigations of the mirror neuron system in typical adults and children as well as individuals diagnosed with autism spectrum disorders. She also conducts related research aimed at understanding the relationship between physiological measures of skin conductance and the perception of pain in oneself and others. She has been awarded research fellowships from the Division of Social Science at UCSD as well as the UCSD Department of Psychology. Her research was recently featured on the PBS series ‘NOVA Science Now’ and the November, 2006 issue of Scientific American. She currently is first author on five other publications.

    Joseph P. McCleery is a doctoral candidate in Experimental Psychology with an emphasis in Developmental Psychology, at the University of California, San Diego. He earned his bachelor's degree in Psychology at Rutgers University in 2000 and his master's degree in Experimental Psychology at UCSD in 2002. His research focuses on brain and behavioral development in children with autism and their families. He also conducts related research on the perception of human action and the development of face processing expertise. His graduate education and research are supported by a traineeship from the National Science Foundation (Vision and Learning in Humans and Machines), a research fellowship from the M.I.N.D. Institute, and a research grant from the Kavli Institute for Brain and Mind.

    Vilayanur S. Ramachandran is Director of the Center for brain and cognition and professor with the neuroscience program and psychology department at the University of California, San Diego, and adjunct professor of biology at the Salk Institute. Ramachandran trained as a Physician obtaining an M.D. from Stanley Medical college and subsequently a Ph.D. from Trinity College at the University of Cambridge, Ramachandran is world-renowned brain researcher. His early research was on visual perception but he is best known for his work in behavioral neurology. Two topics that were once regarded largely as curiosities—phantom limbs and synesthesia—have now become part of “mainstream” research in neuroscience as a result of his work. He has received many honors and awards including a fellowship from All souls college, Oxford, an honorary doctorate from Connecticut college, an honorary doctorate from IIT, Madras the Ariens Kappers gold Medal from the Royal Nederlands Academy of Sciences, for landmark contributions in neuroscience, a Gold medal from the Australian national university, the Ramon Y Cajal award from the international neuropsychiatry association, and the presidential lecture award from the American Academy of Neurology. In 2003 he gave the annual Reith lectures on the BBC, broadcast internationally. In 1995 he gave the Decade of the Brain Lecture at the 25th annual (Silver Jubilee) meeting of the Society for Neuroscience and more recently the Inaugural keynote lecture at the Decade of the brain conference held by NIMH at the Library of Congress. He has also been elected a fellow of the National Academy of Sciences (India), a fellow of the neurosciences Institute La Jolla, a Fellow of the center for advanced studies in behavioral sciences, Stanford, and Hilgard visiting Professor at Stanford university. Ramachandran has published over 130 papers in scientific journals (including six invited review articles in the Scientific American), is Editor-in-chief of the Encyclopedia of Human Behaviour, the encyclopedia of the human brain and author of the critically acclaimed book “Phantoms in the brain” that has been translated into eight languages and formed the basis for a two part series on Channel Four TV UK and a 1 h PBS special in USA. His work is featured frequently in the major news media. NEWSWEEK magazine recently named him a member of “the century club”.one of the “hundred most prominent people to watch in the next century.”

    Jaime A. Pineda is currently an Associate Professor in the Departments of Cognitive Science and Neuroscience at the University of California, San Diego (UCSD) and has served as Director of the Cognitive Neuroscience Laboratory at UCSD for the past 14 years. His research interests include the etiology of addiction, the role of monoamines in behavioral arousal and attention, the role of motivation, and most recently the relationship between EEG mu rhythms, mirror neurons, and action comprehension. Dr. Pineda is founder and Chair of the Scientific Advisory Board for Otosonics, a biomedical therapeutics company involved in the treatment of tinnitus and for Zybernetix, a biotech company in the brain–computer technology field. He has served on a variety of National Institute of Health committees and is a reviewer for several scholarly journals. Dr. Pineda has been an author in over 62 peer-reviewed publications.

    View full text