Being with virtual others: Neural correlates of social interaction

https://doi.org/10.1016/j.neuropsychologia.2005.07.017Get rights and content

Abstract

To characterize the neural correlates of being personally involved in social interaction as opposed to being a passive observer of social interaction between others we performed an fMRI study in which participants were gazed at by virtual characters (ME) or observed them looking at someone else (OTHER). In dynamic animations virtual characters then showed socially relevant facial expressions as they would appear in greeting and approach situations (SOC) or arbitrary facial movements (ARB).

Differential neural activity associated with ME > OTHER was located in anterior medial prefrontal cortex in contrast to the precuneus for OTHER > ME. Perception of socially relevant facial expressions (SOC > ARB) led to differentially increased neural activity in ventral medial prefrontal cortex. Perception of arbitrary facial movements (ARB > SOC) differentially activated the middle temporal gyrus.

The results, thus, show that activation of medial prefrontal cortex underlies both the perception of social communication indicated by facial expressions and the feeling of personal involvement indicated by eye gaze. Our data also demonstrate that distinct regions of medial prefrontal cortex contribute differentially to social cognition: whereas the ventral medial prefrontal cortex is recruited during the analysis of social content as accessible in interactionally relevant mimic gestures, differential activation of a more dorsal part of medial prefrontal cortex subserves the detection of self-relevance and may thus establish an intersubjective context in which communicative signals are evaluated.

Introduction

Despite remarkable efforts having been made in the field of social cognitive neuroscience there still is a substantial debate about the neural processes underlying our ability to understand other minds (Gallagher, 2001). Much research has been conducted on what is called Theory of Mind or mentalizing referring to the capacity of constructing a meta-representation to explain and anticipate the behavior of others. From our standpoint, however, it is unclear to what extent human social understanding of others – particularly in everyday-life situations – relies on actual online involvement in a given situation or on inferential, offline modes of meta-representation. Regardless of whether those processes are to be thought of as theories or simulations as reflected in the debate on theory theory and simulation theory, Frith and Frith (2003) draw a distinction between implicit and explicit mentalizing corresponding to Theory of Mind online and offline. Alternatively a single common mentalizing ability has been suggested to underlie understanding mental states of both self and other (Gopnik & Meltzoff, 1994). Likewise implicit, reflexive mechanisms constituting forms of personal involvement are suggested to be involved in understanding social situations (Lieberman, Gaunt, Gilbert, & Trope, 2002).

Neuroimaging studies have only recently begun to target aspects of online interactions that require personal involvement in social communication (Gallagher, Jack, Roeppstorff, & Frith, 2002; McCabe, Houser, Ryan, Smith, & Trouard, 2001; Rilling et al., 2002; Sanfey, Rilling, Aronson, Nystrom, & Cohen, 2003). Thus far, however, the differential effects of self-involvement on the cerebral representation of mentalizing have not been investigated. This is emphasized by Ochsner (2004) who points out that no within-study comparison of self- and other-related activations (often found in MPFC) has been carried out.

How far are the neural mechanisms of perceiving social interactions altered by being personally engaged in social interaction versus being a passive observer who watches others interact? To address this question in an experimentally feasible way forms of virtual reality are useful instruments. Mediated environments have the potential to elicit a sense of being there, a sense of presence in a virtual reality (Heeter, 1992; Ijsselsteijn & Riva, 2003; Moore, Wiederhold, Wiederhold, & Riva 2002; Reeves & Nass, 1996; Steuer, 1992). Furthermore, it has been demonstrated that virtual characters not only convey social information to human observers, but are also perceived as social agents thus exerting social influence on human interactants (Bailenson, Blascovich, Beall, & Loomis, 2003; Pertaub, Slater, & Barker, 2001). Consequently this method has recently begun to be used in social and environmental psychological research (Blascovich, 2002; de Kort, Ijsselsteijn, Kooijman, & Schuurmans, 2003). Social cognitive neuroscience also benefits considerably by using mediated environments and dynamic stimuli of animated virtual characters (Adolphs, 2003; Pelphrey, Viola, & McCarthy, 2004). However, a crucial prerequisite for the employment of virtual characters in research settings is the assurance that these artificial entities actually do evoke the same experiences and reactions as human stimuli. Here, a considerable number of recent studies have yielded consistent evidence that not only experience (e.g. person perception, see Bente, Krämer, Petersen, & de Ruiter, 2001) but also social reactions are strikingly equivalent to human face-to-face interaction (Garau, Slater, Pertaub, & Razzaque, 2005). Virtual characters have for instance been shown to elicit social facilitation (Hoyt, Blascovich, & Swinth, 2003), proxemic behavior (Bailenson, Blascovich, Beall, & Loomis, 2001) and impression management (Sproull, Subramani, Kiesler, Walker, & Waters, 1996). In sum, they have been shown to evoke a sense of social presence (Biocca, Harms, & Burgoon, 2003)—especially if the appearance of the virtual characters is anthropomorphic (Nowak & Biocca, 2003). As the virtual character's morphology, outward appearance and movements in space and time can be varied systematically, virtual agents and environments have become a powerful tool for experimental psychology (Loomis, Blascovich, & Beall, 1999). Sanchez-Vives and Slater (2005) have recently re-emphasized the applicability of virtual reality as a tool in neuroscience and its value as an object of study in its own right.

We here put to use a much-needed and largely unexplored methodological tool in the realm of social cognitive neuroscience that allows the operationalized investigation of second-person phenomena, i.e. aspects of the dynamic interplay of being personally involved in social interactions (Reddy, 2003). We used this tool in a paradigm which enabled us to investigate the neural processes underlying the cognitive function known as mentalizing dependent upon whether the person who is deciphering someone else's mental state is either involved in social interaction (i.e. online) with that other or not (i.e. offline). For this purpose, we created an experimental situation in which subjects would be socially entrained by virtual others in a controllable mediated scene: In this desktop virtual reality (Tarr & Warren, 2002) virtual characters make an appearance and either turn towards and gaze at the human observer directly (ME; Fig. 1, top row) or look away from the human observer towards a third person not visible to the study participant situated at an angle of approximately 30° (OTHER; Fig. 1, bottom row) thereby systematically varying the study participant's state of self-involvement. Secondly, virtual characters exhibit dynamic facial expressions as they would appear in real-life approach situations to initiate social interaction (Cassell, Bickmore, Campbell, Vilhjalmsson, & Yan, 2001; Grammer, Schiefenhövel, Schleidt, Lorenz, & Eibl-Eibesfeldt, 1988; Kendon & Ferber, 1973). These facial expressions are regarded as socially relevant (SOC) in that they are indicative of someone's intention to establish interpersonal contact and elicit a sense of interactional engagement (Table 1). Alternatively facial movements are shown that are perceived as arbitrary and socially irrelevant (ARB) (Table 1). The resulting two factors (i) SELF-INVOLVEMENT (ME versus OTHER) and (ii) SOCIAL INTERACTION (SOC versus ARB), thus, constitute a two-factorial design (Table 2).

We implemented this design in an event-related fMRI study to investigate the neural substrates of both online and offline mentalizing. We hypothesized that differential neural activations would be associated with one's own personal involvement as compared to situations in which interaction between two others is perceived. We expected that SELF-INVOLVEMENT recruits medial prefrontal cortices whereas perception of the involvement of others in SOCIAL INTERACTION would be subserved by temporo-parietal activations as if taking a third-person-perspective (Vogeley & Fink, 2003). Additionally we assumed that the perception of SOCIAL INTERACTION regardless of self-involvement would also rely upon recruitment of medial prefrontal pathways (Damasio, 1996) while perception of arbitrary facial movements might be accomplished by more basic pathways subserving face perception (Haxby, Hoffman, & Gobbini, 2002).

Section snippets

Subjects

Eighteen right-handed, healthy male volunteers (mean age 25.9 years ± standard deviation 4.2 years) with no record of neurological or psychiatric illness participated in the fMRI study. All volunteers were naïve with respect to the experimental task as well as to the purpose of the study. The study was approved by the local ethics committee of the University of Bonn, Germany.

General information

Using the software package “Poser 4.0” (Curious Lab®) dynamic video animation sequences were designed which depict virtual

Behavioral data during fMRI scanning

Dependent variables were the percent correctness scores of answers to the first question (“WHO?”) and mean response scores to the second question (“CONTACT?”) averaged over all events for each particular condition. One test subject was removed from the analysis as he had consistently failed to answer the questions correctly. The mean correctness score for all remaining subjects for answering the first question was 97.35% (n = 17, S.D. = 1.95, standard error of the mean = 0.46). Mean response score

Discussion

With this study, we suggest to have taken a step towards making second-person phenomena accessible to neuroscientific investigation by using fully controllable anthropomorphic virtual characters which present dynamic facial expressions in order to socially entrain human observers. The results show that different regions of the medial prefrontal cortex are recruited during the perception of social communication and the feeling of personal involvement. Whereas the ventral medial prefrontal cortex

Conclusions

Taken together our results demonstrate that human observers can be socially entrained by virtual characters to whom a communicative intention can be attributed. Behavioral ratings show that socially relevant behavior exhibited by virtual others is judged differently depending upon whether it is self- or other-directed. Ecologically relevant, socially realistic virtual characters can serve as a well-functioning tool for operationalizing second-person phenomena.

The neural activations observed

Acknowledgements

We are grateful for the support of our colleagues in the MRI and Cognitive Neurology group of the Institute of Medicine at the Research Center Juelich and to Dr. Frank Schwab (Universität des Saarlandes, Saarbrücken) for performing FACS coding of our video sequences.

L.S. would like to thank Rüdiger Ilg, Susanne Neufang and Motoaki Sugiura in particular for stimulating discussions and helpful advice.

G.R.F. was supported by the Deutsche Forschungsgemeinschaft (DFG-KFO 112).

The MRI facility is

References (88)

  • B. Knutson et al.

    A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: Characterization with rapid event-related fMRI

    NeuroImage

    (2003)
  • M.D. Lieberman et al.

    Reflection and reflexion: A social cognitive neuroscience apparoch to attributional inference

    Advances in Experimental Social Psychology

    (2002)
  • Y. Nagai et al.

    Activity in ventromedial prefrontal cortex varies with sympathetic skin conductance levels: Physiological account of a “default mode” of brain function

    NeuroImage

    (2004)
  • K.N. Ochsner

    Current directions in social cognitive neuroscience

    Current Opinion in Neurobiology

    (2004)
  • V. Reddy

    On being the object of attention: Implications for self-other consciousness

    Trends in Cognitive Sciences

    (2003)
  • J.K. Rilling et al.

    A neural basis for social cooperation

    Neuron

    (2002)
  • M.A. Sabbagh

    Understanding orbitofrontal contributions to theory-of-mind reasoning: Implications for autism

    Brain and Cognition

    (2004)
  • R. Saxe et al.

    Making sense of another mind: The role of the right temporo-parietal junction

    Neuropsychologia

    (2005)
  • C.A. Seger et al.

    Cortical activations during judgments about the self and an other person

    Neuropsychologia

    (2004)
  • T. Singer et al.

    Brain responses to the acquired moral status of faces

    Neuron

    (2004)
  • K. Vogeley et al.

    Neural correlates of first-person-perspective

    Trends in Cognitive Science

    (2003)
  • K. Vogeley et al.

    The prefrontal cortex generates the basic constituents of the self

    Consciousness and Cognition

    (1999)
  • K. Vogeley et al.

    Mind reading: Neural mechanisms of theory of mind and self-perspective

    NeuroImage

    (2001)
  • M.J. Westerhaus et al.

    Central representation of the sympathetic nervous system in the cerebral cortex

    Brain Research

    (2001)
  • B. Wild et al.

    Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: Quality, quantity, time course and gender differences

    Psychiatry Research

    (2001)
  • J.H.G. Williams et al.

    An fMRI study of joint attention experience

    NeuroImage

    (2005)
  • R. Adolphs

    Cognitive neuroscience of human social behavior

    Nature Reviews Neuroscience

    (2003)
  • J.N. Bailenson et al.

    Equilibrium revisited: Mutual gaze and personal space in virtual environments

    Presence: Teleoperators and Virtual Environments

    (2001)
  • J.N. Bailenson et al.

    Interpersonal distance in immersive virtual environments

    Personality and Social Psychology Bulletin

    (2003)
  • M.S. Beauchamp et al.

    fMRI responses to video and point-light displays of moving humans and manipulable objects

    Journal of Cognitive Neuroscience

    (2003)
  • A. Bechara et al.

    Characterization of the decision-making effect of patients with ventromedial prefrontal cortex lesions

    Brain

    (2000)
  • G. Bente et al.

    Computer animated movement and person perception: Methodological advances in nonverbal behavior research

    Journal of Nonverbal Behavior

    (2001)
  • F. Biocca et al.

    Toward a more robust theory and measure of social presence

    Presence: Teleoperators and Virtual Environments

    (2003)
  • J. Blascovich

    Social influence within immersive virtual environments

  • E. Bonda et al.

    Evidence for a dorso-medial parietal system involved in mental transformation of the body

    Journal of Neurophysiology

    (1996)
  • H.D. Critchley et al.

    Neural activity relating to generation and representation of galvanic skin conductance responses: An fMRI study

    Journal of Neuroscience

    (2000)
  • A.R. Damasio

    The somatic marker hypothesis and the possible functions of the prefrontal cortex

    Philosophical Transactions of the Royal Society London

    (1996)
  • A.R. Damasio

    The feeling of what happens: Body and emotion in the making of consciousness

    (1999)
  • G. Dawson et al.

    Neurocognitive function and joint attention ability in young children with autism spectrum disorder versus developmental delay

    Child Development

    (2002)
  • Y.A.W. de Kort et al.

    Virtual laboratories: Comparability of real and virtual environments for environmental psychology

    Presence

    (2003)
  • P. Ekman et al.

    Facial action coding system: A technique for the measurement of facial movement

    (1978)
  • P. Fossati et al.

    In search of the emotional self: An fMRI study using positive and negative emotional words

    American Journal of Psychiatry

    (2003)
  • K.J. Friston et al.

    Statistical parametric maps in functional imaging: A general linear approach

    Human Brain Mapping

    (1995)
  • U. Frith et al.

    Development and neurophysiology of mentalizing.

    Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences

    (2003)
  • Cited by (354)

    View all citing articles on Scopus
    View full text