2010 Special Issue“Artificial humans”: Psychology and neuroscience perspectives on embodiment and nonverbal communication
Introduction
The challenge to create convincing artificial social entities seems to hold a particular fascination for humans and in fact is older than psychology, computer science or cognitive neuroscience. Historical examples to build mechanic humans as well as recent scientific and technological endeavours to implement socially intelligent machines (Fong, Nourbakhsh, & Dautenhahn, 2003), although differing in their starting intuitions, methodologies and goals reveal a common denominator: the urge to unravel the secrets of human communication and social information processing. Be it for the aim of impressing an audience through a skillful machine, the development of useful and acceptable computer agents or robots, or the experimental control of social cues in psychological studies, basically the success of all these efforts relies on the understanding of the most complex and still in many respects enigmatic social cognitive processes related to the production and perception of social behavior and their underlying neural mechanisms.
Research in social cognition, evolutionary psychology and more recently also in social cognitive neuroscience has provided ample evidence that the human mind in contrast to other species holds particular capacities to process and to adapt to complex affordances emerging from our social environment (Moll and Tomasello, 2007, Tomasello et al., 2005). Equipped by nature with unique prerequisites for social cognition the human cognitive system already in early childhood develops the capability to differentiate self and others (Decety & Chaminade, 2003), to infer emotional and cognitive states of other minds (Frith & Frith, 2003), to form social impressions and to adjust actions and communicative behavior accordingly (Decety and Chaminade, 2003, Frith and Frith, 2003, Vogeley and Roepstorff, 2009). As adults we refer to this ability in everyday life with great ease forming spontaneous impressions of complex psychological matters hardly ever reflecting on the information causing our inferences or the rational of the underlying processes: “Though the full significance of man’s relation to man may not be directly evident, the complexity of feelings and actions that can be understood at a glance is surprisingly great” (Heider, 1958, p. 2). Although social cognitive processes such as social perception and interpersonal communication are seemingly automatic in many instances, humans are also prone to reflect on their actions and capabilities, what not only creates an inferential, reflexive counterforce to intuitive, pre-reflexive processes in daily interactions, but also provides the universal cognitive basis for the development of culture, science and technology (Tomasello et al., 2005). Again we encounter a human particularity, i.e. our striving for understanding ourselves through reflections about regularities in the social world, through systematic psychological experimentation, through identifying relevant neural mechanisms and last not least, through simulating the complex reality of the human mind and behavior through technology.
The idea to simulate sensory, cognitive and motor functions of biological systems through technology has been already the key topic of cybernetics, systems theory and robotics and, more recently, of the emergent field of biorobotics (Webb, 2001). The aim to simulate biological systems by creating surrogates which are not only abstract models of the world but result in somehow materialized agents, which can be encountered in the real world or experienced via our senses, poses a particular challenge in theory and practice. Actually we might claim that it is the most critical test for a model of biological systems to put it into action and to expose it to the critical comparison with social reality as created by nature (Hut & Sussman, 1987). The requirements to meet this challenge have been sketched by Webb (2001), comprising different tasks levels which have to be addressed successfully before artificial systems can be expected to pass the real life test (see Fig. 1). These levels include reliable observation of the natural target behavior, appropriate theoretical assumptions about the biological mechanisms underlying this behavior, and efficient algorithms to implement a convincing simulation, which demonstrates the quality of the model.
While in its general form Webb’s model holds value as a taxonomy for a broad variety of technical simulations of biological systems, it has to be specified with regard to the topic of this article, which focuses on social behavior of humans, in contrast, for example, to the construction of a robot pet serving as a social toy or an industrial robot performing complex mounting tasks during the “ghost shift” of a car fabric. Specifications are thus required with regard to the behavioral domains and the cognitive processes to be addressed and the simulation approach to be chosen.
Section snippets
Simulating social interaction
The domain essentially addressed in this article is embodiment and nonverbal communication (NVC) in social virtual entities. Embodiment as a feature of artificial agents can be preliminarily defined as the presence of human-like physical properties, which enable the transmission of nonverbal signals (Bente, Rüggenberg et al., 2008, Ruttkay et al., 2002). Embodiment is a constituent of all face-to-face encounters, but can be minimized or might be even absent in mediated communication and in
The interpersonal perspective: meanings and functions of nonverbal behavior
Nonverbal cues including facial expressions, gaze behavior, gestures, postures and body movements have a deep impact on the process and outcome of our communication (Argyle et al., 1970, Mehrabian and Wiener, 1967, Schneider et al., 1979). Burgoon (1994) summarized relevant findings from NVB research concluding that approximately 60%–65% of social meaning is conveyed via NVC channels (Mehrabian & Ferris, 1967). This rough estimation however ignores the particular complexity (Bente and Krämer,
Processing embodied social cues: intrapersonal mechanisms
Social cognitive processes have recently become a key topic in cognitive neuroscience and social (cognitive) neuroscience has emerged as a new subdiscipline in neurosciences and recently developed into an autonomous scientific discipline (Adolphs, 2009, Cacioppo et al., 2004). Generally speaking, social neuroscience focuses on processes that are related to the adequate ascription of mental states to others for the purpose of successful communication or interaction between personal agents. One
Coordinating minds and actions: the case of gaze
Integrating both interpersonal functions and intrapersonal mechanisms social gaze defines a paradigmatic case, which allows to exemplify the multifunctionality of NVB and the multiple cognitive processes and neural mechanisms involved in social information processing. Everyday experience as well as extensive research in social psychology and social cognitive neuroscience confirm the crucial role of human gaze behavior in social interactions and its impact on cognitive, affective and
Conclusions and future prospects
The current paper aimed to demonstrate the complexity of nonverbal phenomena in social interaction both with regard to its functions as well as the psychological processes and neural mechanisms supporting its interpretation. Observing NVB of others involves information processing on various levels and recruits different cognitive and neural processes. What humans evidently learn in ontogeny during early interaction with seemingly great ease are not at all trivial which becomes apparent when
Acknowledgements
The study was supported by the German Research Foundation (“Deutsche Forschungsgemeinschaft, DFG”), the German Ministry for Education and Research (“Bundesministerium für Bildung und Forschung, BMBF”), and the Volkswagen Foundation, Germany.
References (247)
- et al.
Do triangles play tricks? Attribution of mental states to animated shapes in normal and abnormal development
Cognitive Development
(2000) - et al.
A deficit in discriminating gaze direction in a case with right superior temporal gyrus lesion
Neuropsychologia
(2006) - et al.
Social perception from visual cues: role of the STS region
Trends in Cognitive Sciences
(2000) - et al.
Accurate judgments of intention from motion cues alone: a cross-cultural study
Evolution and Human Behavior
(2005) - et al.
The role of motor contagion in the prediction of action
Neuropsychologia
(2005) - et al.
Abnormalities of brain function during a nonverbal theory of mind task in schizophrenia
Neuropsychologia
(2003) - et al.
Movement and mind: a functional imaging study of perception and interpretation of complex intentional movement patterns
Neuroimage
(2000) - et al.
When eye creates the contact! ERP evidence for early dissociation between direct and averted gaze motion processing
Neuropsychologia
(2007) - et al.
When the self represents the other: a new cognitive neuroscience view on psychological identification
Consciousness and Cognition
(2003) - et al.
On signalling that it’s your turn to speak
Journal of Experimental Social Psychology
(1974)
Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension
Cognition
A survey of socially interactive robots
Robotics and Autonomous Systems
Implicit and explicit processes in social cognition
Neuron
Dissociable neural pathways for the perception and recognition of expressive and instrumental gestures
Neuropsychologia
Reading the mind in cartoons and stories: an fMRI study of ‘theory of mind’ in verbal and nonverbal tasks
Neuropsychologia
Cognitive neuroscience of human social behavior
Nature Reviews Neuroscience
The social brain: neural basis of social knowledge
Annual Reviews of Psychology
Effects of nonverbal behavior on perceptions of power bases
Journal of Social Psychology
Meeting of minds: the medial frontal cortex and social cognition
Nature Reviews Neuroscience
The measurement of nonverbal immediacy
Journal of Applied Communication Research
Frontal and temporo-parietal lobe contributions to theory of mind: neuropsychological evidence from a false-belief task with reduced language and executive demands
Journal of Cognitive Neurosciences
Gaze and mutual gaze
The meaning of five patterns of gaze
European Journal of Social Psychology
The communication of inferior and superior attitudes by verbal and non verbal signals
British Journal of Social and Clinical Psychology
Transformed Social Interaction, Augmented Gaze, and Social Influence in Immersive Virtual Environments
Human Communication Research
Interpersonal distance in immersive virtual environments
Personality and Social Psychology Bulletin
The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction
Presence: Teleoperators and Virtual Environments
Equilibrium revisited: mutual gaze and personal space in virtual environments
Presence: Teleoperators and Virtual Environments
Imitation
Cognitive processes mediating behavioral change
Journal of Personality and Psychology
The four horsemen of automaticity: Awareness, intention, efficiency, and control in social cognition
Mindblindness: an essay on autism and theory of mind
Grounded cognition
Annual Reviews of Psychology
Temporal and spatial contingencies in the perception of social events
Journal of Personality and Social Psychology
Motor mimicry as primitive empathy
Observation learning of tool use by captive Guinea baboons (Papio papio)
American Journal of Physical Anthropology
Facilities for the graphical computer simulation of head and body movements
Behavior Research Methods, Instruments, and Computers
Virtual Gaze. A pilot study on the effects of computer simulated gaze in avatar-based conversations
Integrierte Registrierung und Analyse verbaler und nonverbaler Kommunikation
Computer animated movement and person perception: Methodological advances in nonverbal behavior research
Journal of Nonverbal Behavior
Transcript-based computer animation of movement: evaluating a new tool for nonverbal behavior research
Behavior Research Methods, Instruments and Computers
Avatar-mediated networking: increasing social presence and interpersonal trust in net-based collaborations
Human Communication Research
Is there anybody out there? Analyzing the effects of embodiment and nonverbal behavior in avatar-mediated communication
Semantic space is (approximately) bipolar
Journal of Psychology: Interdisciplinary and Applied
Interpersonal coordination: Behavior matching and interactional synchrony
Plugging your body into the telecommunication system: mediated embodiment, media interfaces, and social virtual environments
Toward a more robust theory and measure of social presence
Presence: Teleoperators and Virtual Environments
Perception of human motion
Annual Reviews of Psychology
The detection of contingency and animacy from simple animations in the human brain
Cerebral Cortex
Cited by (48)
Matter over mind? How the acceptance of digital entities depends on their appearance, mental prowess, and the interaction between both
2020, International Journal of Human Computer StudiesCitation Excerpt :In fact, cultural observations suggest that the Christian understanding of a uniquely human nature still informs the worldview of many inhabitants of Western industrial nations (Kaplan, 2004), even despite the ongoing decline of organized religion (Altemeyer, 2009). While the conceptualization of this distinctly human essence may vary with each person's specific philosophical stance, research has shown that most people consider aspects such as higher-order cognition (Leyens et al., 2001), emotional experience (Waytz et al., 2010), emotional warmth (Gray and Wegner, 2012), and sophisticated processing of social cues (Pagel, 2012; Vogeley and Bente, 2010) as explicitly human traits. In all probability, the validity of these criteria will keep experts from numerous research areas (including neuroscience, psychology, and zoology) occupied for the near future; nevertheless, they still provide a veritable basis for people's current acceptance of sophisticated technology.
Binding paradox in artificial social realities
2023, Behavioral and Brain SciencesPsychiatric Interventions in Virtual Reality: Why We Need an Ethical Framework
2020, Cambridge Quarterly of Healthcare Ethics