Elsevier

Neural Networks

Volume 23, Issues 8–9, October–November 2010, Pages 1077-1090
Neural Networks

2010 Special Issue
“Artificial humans”: Psychology and neuroscience perspectives on embodiment and nonverbal communication

https://doi.org/10.1016/j.neunet.2010.06.003Get rights and content

Abstract

“Artificial humans”, so-called “Embodied Conversational Agents” and humanoid robots, are assumed to facilitate human–technology interaction referring to the unique human capacities of interpersonal communication and social information processing. While early research and development in artificial intelligence (AI) focused on processing and production of natural language, the “new AI” has also taken into account the emotional and relational aspects of communication with an emphasis both on understanding and production of nonverbal behavior. This shift in attention in computer science and engineering is reflected in recent developments in psychology and social cognitive neuroscience. This article addresses key challenges which emerge from the goal to equip machines with socio-emotional intelligence and to enable them to interpret subtle nonverbal cues and to respond to social affordances with naturally appearing behavior from both perspectives. In particular, we propose that the creation of credible artificial humans not only defines the ultimate test for our understanding of human communication and social cognition but also provides a unique research tool to improve our knowledge about the underlying psychological processes and neural mechanisms.

Introduction

The challenge to create convincing artificial social entities seems to hold a particular fascination for humans and in fact is older than psychology, computer science or cognitive neuroscience. Historical examples to build mechanic humans as well as recent scientific and technological endeavours to implement socially intelligent machines (Fong, Nourbakhsh, & Dautenhahn, 2003), although differing in their starting intuitions, methodologies and goals reveal a common denominator: the urge to unravel the secrets of human communication and social information processing. Be it for the aim of impressing an audience through a skillful machine, the development of useful and acceptable computer agents or robots, or the experimental control of social cues in psychological studies, basically the success of all these efforts relies on the understanding of the most complex and still in many respects enigmatic social cognitive processes related to the production and perception of social behavior and their underlying neural mechanisms.

Research in social cognition, evolutionary psychology and more recently also in social cognitive neuroscience has provided ample evidence that the human mind in contrast to other species holds particular capacities to process and to adapt to complex affordances emerging from our social environment (Moll and Tomasello, 2007, Tomasello et al., 2005). Equipped by nature with unique prerequisites for social cognition the human cognitive system already in early childhood develops the capability to differentiate self and others (Decety & Chaminade, 2003), to infer emotional and cognitive states of other minds (Frith & Frith, 2003), to form social impressions and to adjust actions and communicative behavior accordingly (Decety and Chaminade, 2003, Frith and Frith, 2003, Vogeley and Roepstorff, 2009). As adults we refer to this ability in everyday life with great ease forming spontaneous impressions of complex psychological matters hardly ever reflecting on the information causing our inferences or the rational of the underlying processes: “Though the full significance of man’s relation to man may not be directly evident, the complexity of feelings and actions that can be understood at a glance is surprisingly great” (Heider, 1958, p. 2). Although social cognitive processes such as social perception and interpersonal communication are seemingly automatic in many instances, humans are also prone to reflect on their actions and capabilities, what not only creates an inferential, reflexive counterforce to intuitive, pre-reflexive processes in daily interactions, but also provides the universal cognitive basis for the development of culture, science and technology (Tomasello et al., 2005). Again we encounter a human particularity, i.e. our striving for understanding ourselves through reflections about regularities in the social world, through systematic psychological experimentation, through identifying relevant neural mechanisms and last not least, through simulating the complex reality of the human mind and behavior through technology.

The idea to simulate sensory, cognitive and motor functions of biological systems through technology has been already the key topic of cybernetics, systems theory and robotics and, more recently, of the emergent field of biorobotics (Webb, 2001). The aim to simulate biological systems by creating surrogates which are not only abstract models of the world but result in somehow materialized agents, which can be encountered in the real world or experienced via our senses, poses a particular challenge in theory and practice. Actually we might claim that it is the most critical test for a model of biological systems to put it into action and to expose it to the critical comparison with social reality as created by nature (Hut & Sussman, 1987). The requirements to meet this challenge have been sketched by Webb (2001), comprising different tasks levels which have to be addressed successfully before artificial systems can be expected to pass the real life test (see Fig. 1). These levels include reliable observation of the natural target behavior, appropriate theoretical assumptions about the biological mechanisms underlying this behavior, and efficient algorithms to implement a convincing simulation, which demonstrates the quality of the model.

While in its general form Webb’s model holds value as a taxonomy for a broad variety of technical simulations of biological systems, it has to be specified with regard to the topic of this article, which focuses on social behavior of humans, in contrast, for example, to the construction of a robot pet serving as a social toy or an industrial robot performing complex mounting tasks during the “ghost shift” of a car fabric. Specifications are thus required with regard to the behavioral domains and the cognitive processes to be addressed and the simulation approach to be chosen.

Section snippets

Simulating social interaction

The domain essentially addressed in this article is embodiment and nonverbal communication (NVC) in social virtual entities. Embodiment as a feature of artificial agents can be preliminarily defined as the presence of human-like physical properties, which enable the transmission of nonverbal signals (Bente, Rüggenberg et al., 2008, Ruttkay et al., 2002). Embodiment is a constituent of all face-to-face encounters, but can be minimized or might be even absent in mediated communication and in

The interpersonal perspective: meanings and functions of nonverbal behavior

Nonverbal cues including facial expressions, gaze behavior, gestures, postures and body movements have a deep impact on the process and outcome of our communication (Argyle et al., 1970, Mehrabian and Wiener, 1967, Schneider et al., 1979). Burgoon (1994) summarized relevant findings from NVB research concluding that approximately 60%–65% of social meaning is conveyed via NVC channels (Mehrabian & Ferris, 1967). This rough estimation however ignores the particular complexity (Bente and Krämer,

Processing embodied social cues: intrapersonal mechanisms

Social cognitive processes have recently become a key topic in cognitive neuroscience and social (cognitive) neuroscience has emerged as a new subdiscipline in neurosciences and recently developed into an autonomous scientific discipline (Adolphs, 2009, Cacioppo et al., 2004). Generally speaking, social neuroscience focuses on processes that are related to the adequate ascription of mental states to others for the purpose of successful communication or interaction between personal agents. One

Coordinating minds and actions: the case of gaze

Integrating both interpersonal functions and intrapersonal mechanisms social gaze defines a paradigmatic case, which allows to exemplify the multifunctionality of NVB and the multiple cognitive processes and neural mechanisms involved in social information processing. Everyday experience as well as extensive research in social psychology and social cognitive neuroscience confirm the crucial role of human gaze behavior in social interactions and its impact on cognitive, affective and

Conclusions and future prospects

The current paper aimed to demonstrate the complexity of nonverbal phenomena in social interaction both with regard to its functions as well as the psychological processes and neural mechanisms supporting its interpretation. Observing NVB of others involves information processing on various levels and recruits different cognitive and neural processes. What humans evidently learn in ontogeny during early interaction with seemingly great ease are not at all trivial which becomes apparent when

Acknowledgements

The study was supported by the German Research Foundation (“Deutsche Forschungsgemeinschaft, DFG”), the German Ministry for Education and Research (“Bundesministerium für Bildung und Forschung, BMBF”), and the Volkswagen Foundation, Germany.

References (247)

  • P. Fletcher et al.

    Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension

    Cognition

    (1995)
  • T. Fong et al.

    A survey of socially interactive robots

    Robotics and Autonomous Systems

    (2003)
  • C.D. Frith et al.

    Implicit and explicit processes in social cognition

    Neuron

    (2008)
  • H.L. Gallagher et al.

    Dissociable neural pathways for the perception and recognition of expressive and instrumental gestures

    Neuropsychologia

    (2004)
  • H.L. Gallagher et al.

    Reading the mind in cartoons and stories: an fMRI study of ‘theory of mind’ in verbal and nonverbal tasks

    Neuropsychologia

    (2000)
  • R. Adolphs

    Cognitive neuroscience of human social behavior

    Nature Reviews Neuroscience

    (2003)
  • R. Adolphs

    The social brain: neural basis of social knowledge

    Annual Reviews of Psychology

    (2009)
  • H. Aguinis et al.

    Effects of nonverbal behavior on perceptions of power bases

    Journal of Social Psychology

    (1998)
  • D.M. Amodio et al.

    Meeting of minds: the medial frontal cortex and social cognition

    Nature Reviews Neuroscience

    (2006)
  • J.F. Andersen et al.

    The measurement of nonverbal immediacy

    Journal of Applied Communication Research

    (1979)
  • I.A. Apperly et al.

    Frontal and temporo-parietal lobe contributions to theory of mind: neuropsychological evidence from a false-belief task with reduced language and executive demands

    Journal of Cognitive Neurosciences

    (2004)
  • J.M. Argyle et al.

    Gaze and mutual gaze

    (1976)
  • M. Argyle et al.

    The meaning of five patterns of gaze

    European Journal of Social Psychology

    (1974)
  • M. Argyle et al.

    The communication of inferior and superior attitudes by verbal and non verbal signals

    British Journal of Social and Clinical Psychology

    (1970)
  • J.N. Bailenson et al.

    Transformed Social Interaction, Augmented Gaze, and Social Influence in Immersive Virtual Environments

    Human Communication Research

    (2005)
  • J.N. Bailenson et al.

    Interpersonal distance in immersive virtual environments

    Personality and Social Psychology Bulletin

    (2003)
  • J. Bailenson et al.

    The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction

    Presence: Teleoperators and Virtual Environments

    (2006)
  • J.N. Bailenson et al.

    Equilibrium revisited: mutual gaze and personal space in virtual environments

    Presence: Teleoperators and Virtual Environments

    (2001)
  • A. Bandura

    Imitation

  • A. Bandura et al.

    Cognitive processes mediating behavioral change

    Journal of Personality and Psychology

    (1977)
  • J.A. Bargh

    The four horsemen of automaticity: Awareness, intention, efficiency, and control in social cognition

  • S. Baron-Cohen

    Mindblindness: an essay on autism and theory of mind

    (1995)
  • L.W. Barsalou

    Grounded cognition

    Annual Reviews of Psychology

    (2008)
  • J.N. Bassili

    Temporal and spatial contingencies in the perception of social events

    Journal of Personality and Social Psychology

    (1976)
  • J.B. Bavelas et al.

    Motor mimicry as primitive empathy

  • B. Beck

    Observation learning of tool use by captive Guinea baboons (Papio papio)

    American Journal of Physical Anthropology

    (2005)
  • G. Bente

    Facilities for the graphical computer simulation of head and body movements

    Behavior Research Methods, Instruments, and Computers

    (1989)
  • G. Bente et al.

    Virtual Gaze. A pilot study on the effects of computer simulated gaze in avatar-based conversations

  • G. Bente et al.

    Integrierte Registrierung und Analyse verbaler und nonverbaler Kommunikation

  • G. Bente et al.

    Computer animated movement and person perception: Methodological advances in nonverbal behavior research

    Journal of Nonverbal Behavior

    (2001)
  • G. Bente et al.

    Transcript-based computer animation of movement: evaluating a new tool for nonverbal behavior research

    Behavior Research Methods, Instruments and Computers

    (2001)
  • G. Bente et al.

    Avatar-mediated networking: increasing social presence and interpersonal trust in net-based collaborations

    Human Communication Research

    (2008)
  • G. Bente et al.

    Is there anybody out there? Analyzing the effects of embodiment and nonverbal behavior in avatar-mediated communication

  • P.M. Bentler

    Semantic space is (approximately) bipolar

    Journal of Psychology: Interdisciplinary and Applied

    (1969)
  • F.J. Bernieri et al.

    Interpersonal coordination: Behavior matching and interactional synchrony

  • Biocca, F., & Nowak, K. (1999). I feel as if I’m here, inside the computer”: Toward a theory of presence in advanced...
  • F. Biocca et al.

    Plugging your body into the telecommunication system: mediated embodiment, media interfaces, and social virtual environments

  • F. Biocca et al.

    Toward a more robust theory and measure of social presence

    Presence: Teleoperators and Virtual Environments

    (2003)
  • R. Blake et al.

    Perception of human motion

    Annual Reviews of Psychology

    (2007)
  • S.J. Blakemore et al.

    The detection of contingency and animacy from simple animations in the human brain

    Cerebral Cortex

    (2003)
  • Cited by (48)

    • Matter over mind? How the acceptance of digital entities depends on their appearance, mental prowess, and the interaction between both

      2020, International Journal of Human Computer Studies
      Citation Excerpt :

      In fact, cultural observations suggest that the Christian understanding of a uniquely human nature still informs the worldview of many inhabitants of Western industrial nations (Kaplan, 2004), even despite the ongoing decline of organized religion (Altemeyer, 2009). While the conceptualization of this distinctly human essence may vary with each person's specific philosophical stance, research has shown that most people consider aspects such as higher-order cognition (Leyens et al., 2001), emotional experience (Waytz et al., 2010), emotional warmth (Gray and Wegner, 2012), and sophisticated processing of social cues (Pagel, 2012; Vogeley and Bente, 2010) as explicitly human traits. In all probability, the validity of these criteria will keep experts from numerous research areas (including neuroscience, psychology, and zoology) occupied for the near future; nevertheless, they still provide a veritable basis for people's current acceptance of sophisticated technology.

    • Binding paradox in artificial social realities

      2023, Behavioral and Brain Sciences
    View all citing articles on Scopus
    View full text