Elsevier

Cognition

Volume 94, Issue 3, January 2005, Pages 193-213
Cognition

Simulationist models of face-based emotion recognition

https://doi.org/10.1016/j.cognition.2004.01.005Get rights and content

Abstract

Recent studies of emotion mindreading reveal that for three emotions, fear, disgust, and anger, deficits in face-based recognition are paired with deficits in the production of the same emotion. What type of mindreading process would explain this pattern of paired deficits? The simulation approach and the theorizing approach are examined to determine their compatibility with the existing evidence. We conclude that the simulation approach offers the best explanation of the data. What computational steps might be used, however, in simulation-style emotion detection? Four alternative models are explored: a generate-and-test model, a reverse simulation model, a variant of the reverse simulation model that employs an “as if” loop, and an unmediated resonance model.

Introduction

Mindreading is the capacity to identify the mental states of others, for example, their beliefs, desires, intentions, goals, experiences, sensations and also emotion states. One approach to mindreading holds that mental-state attributors deploy a naı̈ve psychological theory to infer mental states in others from their behavior, the environment, and/or their other mental states. According to different versions of this “theory-theory” (TT), the naı̈ve psychological theory is either a component of an innate, dedicated module or is acquired by domain-general learning. A second approach holds that people typically execute mindreading by a different sort of process, a simulation process. Roughly, according to simulation theory (ST), an attributor arrives at a mental attribution by simulating, replicating, or reproducing in his own mind the same state as the target's, or by attempting to do so. For example, the attributor would pretend to be in initial states thought to correspond to those of the target, feed these states into parts of his own cognitive equipment (e.g. a decision-making mechanism), which would operate on them to produce an output state that is imputed to the target.

Mindreading has been studied in many disciplines, and both TT and ST have had proponents in each of them. In developmental psychology TT has been endorsed by Gopnik and Meltzoff, 1997, Gopnik and Wellman, 1992, Gopnik and Wellman, 1994, Leslie, 1994, Perner, 1991, Premack and Woodruff, 1978, Wellman, 1990, whereas ST has been defended by Harris, 1991, Harris, 1992. In philosophy ST has been endorsed by Currie and Ravenscroft, 2002, Goldman, 1989, Goldman, 1992a, Goldman, 1992b, Goldman, 2000, Gordon, 1986, Gordon, 1992, Gordon, 1996, Heal, 1986, Heal, 1996, Heal, 1998, whereas TT has been defended as an explicit approach to the execution of mindreading by Fodor, 1992, Nichols et al., 1996, Stich and Nichols, 1992, or as a theory of how the folk conceptualize mental states by Armstrong, 1968, Lewis, 1972, Sellars, 1956, Shoemaker, 1975. Studies of mentalizing in neuroscience (e.g. Fletcher et al., 1995, Frith and Frith, 1999, McCabe et al., 2001) typically ignore the TT–ST controversy but work with ‘theory of mind’ terminology, which is suggestive of TT, and cite the TT-leaning literature. On the other hand, much recent neuroscientific work is quite receptive to simulationist ideas (Blakemore and Decety, 2001, Carr et al., 2003, Chaminade et al., 2001, Gallese, 2001, Gallese, 2003, Gallese and Goldman, 1998, Iacoboni et al., 1999, Jeannerod, 2001), although the majority of this research is addressed less to mindreading per se than to related topics such as simulation of action, imitation, or empathy. In recent years a number of researchers have moved away from pure forms of TT or ST in the direction of some sort of TT/ST hybrid (Adolphs, 2002, Goldman (in preparation), Nichols and Stich, 2003, Perner, 1996), though the exact nature of the hybrid is rather fluid. In light of this continuing controversy, any research that provides substantial evidence for either ST or TT, even in a single subdomain of mindreading, deserves close attention.

In this paper we review a body of neuropsychological research that, we shall argue, supports ST for a certain circumscribed mindreading task. This is the task of attributing emotion states to others based on their facial expressions. This task is different from those usually studied in the mindreading literature, in part because the attributed mental states differ from the usual ones. The vast majority of the literature is devoted to propositional attitudes such as desires and beliefs, almost entirely ignoring emotion states like fear, anger, disgust, or happiness. There is no good reason to exclude these mental states, which are routinely attributed to others in daily life. So it is time to extend research and theory into this subdomain of the mental. At the same time, it cannot be assumed that the style of mindreading in this subdomain is the same as the style that characterizes other subdomains. So we make no attempt to generalize from the type of mental state ascriptions studied here to mindreading tout court.

There are at least two reasons why the properties of face-based emotion recognition (FaBER) might not be shared by methods of mindreading in other subdomains. First, the recognition or classification of propositional attitude contents may introduce a level of complexity that goes beyond the task of classifying emotion types. Second, the reading of emotions, especially basic emotions, may have unique survival value, so it is conceivable that specialized programs have evolved for the recognition of emotions, and these specialized programs may not operate in other mindreading tasks. Because of these differences between FaBER and other types of mindreading, it cannot be assumed that the processes characteristic of FaBER can be extrapolated to other types of mindreading.

We begin by reviewing existing findings, some clinical and some experimental, that display a striking pattern of paired deficits between emotion production and face-based recognition (attribution). These findings have not previously been brought together with the explicit intent of examining them in the context of the TT–ST controversy. Next we argue that this pattern readily lends itself to a simulationist explanation, whereas existing data do not fit with a theory-based explanation. Finally, the core project of the paper is to formulate and evaluate four specific models of how normal mindreaders could use simulation to arrive at emotion classifications.

Section snippets

Paired deficits in emotion production and face-based recognition

In early studies, Ralph Adolphs and colleagues investigated whether damage to the amygdala affects face-based emotion recognition (Adolphs, 1995, Adolphs et al., 1994). These studies were motivated by the well-known role of the amygdala in mediating fear, including its prominent role in fear-conditioning and the storage of fear-related emotion memories (LeDoux, 1993, LeDoux, 2000). One patient studied by Adolphs et al. was SM, a 30-year-old woman with Urbach-Wiethe disease, which resulted in

Emotion mindreading by theory versus simulation

Let us further clarify and expound the two basic theoretical positions towards mindreading, which have loomed large in the literature. There are numerous ways of developing the TT idea, but the main idea is that the mindreader selects a mental state for attribution to a target based on inference from other information about the target. According to one popular version of TT, such an inference is guided by folk psychological generalizations concerning relationships or transitions between

Explaining the emotion recognition data by TT versus ST

The central claim presented in Section 2 was that there is substantial evidence concerning three emotions indicating that deficits in the production (experience) of an emotion and deficits in the face-based recognition of that emotion reliably co-occur. This strongly suggests that the same neural mechanisms subserve both the experience and the recognition of an emotion. In addition, the Wicker et al. (2003) study found that, in normals, the same neural regions were implicated in both the

Possible simulationist models

Although the foregoing case for a simulational approach to face-based emotion recognition strikes us as compelling, it leaves open the question of how the simulational process proceeds. Those skeptical of our case for simulation may remain skeptical as long as no plausible, sufficiently detailed story of the simulation process in these cases is forthcoming. We get little help on this question from the existing literature. Articles describing paired deficits often contain conclusions hinting at

Why simulation?

In the preceding sections, we marshaled evidence that FaBER proceeds by simulation rather than theory-based mechanisms. At any rate, simulation is the fundamental or primitive method of recognizing emotion from faces, although theorizing might also be used, for example, as a compensatory strategy. The evidence is much less clear-cut, however, with regard to distinguishing possible simulationist models, and much further study is warranted. Our hope is that the specific models we have formulated

Acknowledgements

The authors wish to thank Andrew Lawrence, Vittorio Gallese, Giacomo Rizzolatti, Christian Keysers, Ralph Adolphs, Daniel Osherson, and three anonymous reviewers for valuable comments on earlier drafts of this paper.

References (95)

  • J.E. LeDoux

    Emotion memory systems in the brain

    Behavioral Brain Research

    (1993)
  • A.M. Leslie

    Pretending and believing: Issues in the theory of ToMM

    Cognition

    (1994)
  • G. Rizzolatti et al.

    Premotor cortex and the recognition of motor actions

    Cognitive Brain Research

    (1996)
  • D.M. Small et al.

    Dissociation of neural representation of intensity and affective valuation in human gustation

    Neuron

    (2003)
  • B. Wicker et al.

    Both of us disgusted in my insula: The common neural basis of seeing and feeling disgust

    Neuron

    (2003)
  • P. Adelman et al.

    Facial efference and the experience of emotion

    Annual Review of Psychology

    (1989)
  • R. Adolphs

    Fear and the human amygdala

    Journal of Neuroscience

    (1995)
  • R. Adolphs

    Recognizing emotion from facial expressions: Psychological and neurological mechanisms

    Behavioral and Cognitive Neuroscience Reviews

    (2002)
  • R. Adolphs et al.

    A role for the somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping

    Journal of Neuroscience

    (2000)
  • R. Adolphs et al.

    Impaired recognition of emotion in facial expressions following bilateral damage to the amygdala

    Nature

    (1994)
  • D.M. Armstrong

    A materialist theory of the mind

    (1968)
  • A. Bartels et al.

    The architecture of the colour centre in the human visual brain: New results and a review

    European Journal of Neuroscience

    (2000)
  • A. Bechara et al.

    Double dissociation of conditioning and declarative knowledge relative to the amygdala and hippocampus in humans

    Science

    (1995)
  • A.L. Benton et al.

    Contributions to neuropsychological assessment

    (1983)
  • S. Blakemore et al.

    From the perception of action to the understanding of intention

    Nature Reviews Neuroscience

    (2001)
  • A.J. Calder et al.

    Facial expression recognition by people with Mobius syndrome

    Cognitive Neuropsychology

    (2000)
  • A.J. Calder et al.

    Impaired recognition and experience of disgust following brain injury

    Nature Reviews Neuroscience

    (2000)
  • A.J. Calder et al.

    Neuropsychology of fear and loathing

    Nature Reviews Neuroscience

    (2001)
  • L. Carr et al.

    Neural mechanisms of empathy in humans: A relay from neural systems for imitation to limbic areas

    Proceedings of the National Academy of Science USA

    (2003)
  • T. Chaminade et al.

    Is perceptual anticipation a motor simulation? A PET study

    NeuroReport

    (2001)
  • G. Currie et al.

    Recreative minds

    (2002)
  • A. Damasio

    Descartes' error

    (1994)
  • A. Damasio

    The feeling of what happens

    (1999)
  • U. Dimberg et al.

    Rapid facial reactions to emotional facial expressions

    Scandinavian Journal of Psychology

    (1998)
  • P. Ekman

    Are there basic emotions?

    Psychological Review

    (1992)
  • L. Fadiga et al.

    Motor facilitation during action observation: A magnetic stimulation study

    Journal of Neurophysiology

    (1995)
  • C.D. Frith et al.

    Interacting minds—a biological basis

    Science

    (1999)
  • R.K. Fulbright et al.

    MR functional imaging of regional brain responses to pleasant and unpleasant odors

    American Journal of Neuroradiology

    (1998)
  • V. Gallese

    The ‘shared manifold’ hypothesis: From mirror neurons to empathy

    Journal of Consciousness Studies

    (2001)
  • V. Gallese

    The manifold nature of interpersonal relations: The quest for a common mechanism

    Philosophical Transactions of the Royal Society of London, B

    (2003)
  • V. Gallese et al.

    Action recognition in the premotor cortex

    Brain

    (1996)
  • A.I. Goldman

    Interpretation psychologized

    Mind and Language

    (1989)
  • A.I. Goldman

    In defense of the simulation theory

    Mind and Language

    (1992)
  • A.I. Goldman

    Empathy, mind and morals

    Proceedings and Addresses of the American Philosophical Association

    (1992)
  • A.I. Goldman

    The mentalizing folk

  • Goldman, A. I (in preparation). The simulating mind. New York: Oxford University...
  • A. Gopnik et al.

    Words, thoughts, and theories

    (1997)
  • Cited by (0)

    View full text