Skip to main content
Log in

Audiovisual mirror neurons and action recognition

  • Review
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Many object-related actions can be recognized both by their sound and by their vision. Here we describe a population of neurons in the ventral premotor cortex of the monkey that discharge both when the animal performs a specific action and when it hears or sees the same action performed by another individual. These 'audiovisual mirror neurons' therefore represent actions independently of whether these actions are performed, heard or seen. The magnitude of auditory and visual responses did not differ significantly in half the neurons. A neurometric analysis revealed that based on the response of these neurons, two actions could be discriminated with 97% accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Similar content being viewed by others

References

  • Bach M, Bouis D, Fischer B (1983) An accurate and linear infrared oculometer. J Neurosci Methods 9:9–14

    CAS  PubMed  Google Scholar 

  • Bi G, Poo M (2001) Synaptic modification by correlated activity: Hebb's postulate revisited. Annu Rev Neurosci 24:139–166

    Article  CAS  PubMed  Google Scholar 

  • Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13:400–404

    Article  CAS  PubMed  Google Scholar 

  • Cantalupo C, Hopkins WD (2001) Asymmetric Broca's area in great apes. Nature 414:505

    Article  CAS  PubMed  Google Scholar 

  • Decety J, Grezes J (1999) Neural mechanisms subserving the perception of human actions. Trends Cogn Sci 3:172–178

    Article  PubMed  Google Scholar 

  • Fadiga L, Fogassi L, Pavesi G, Rizzolatti G (1995) Motor facilitation during action observation: a magnetic stimulation study. J Neurophysiol 73:2608–2611

    CAS  PubMed  Google Scholar 

  • Fries W (1984) Cortical projections to the superior colliculus in the macaque monkey: a retrograde study using horseradish peroxidase. J Comp Neurol 230:55–76

    Google Scholar 

  • Gallese V, Fadiga L, Fogassi L, Rizzolatti G (1996) Action recognition in the premotor cortex. Brain 119:593–609

    PubMed  Google Scholar 

  • Grafton ST, Arbib MA, Fadiga L, Rizzolatti G (1996) Localization of grasp representations in humans by positron emission tomography. 2. Observation compared with imagination. Exp Brain Res 112:103–111

    CAS  PubMed  Google Scholar 

  • Graziano MS, Reiss LA, Gross CG (1999) A neuronal representation of the location of nearby sounds. Nature 397:428–430

    Google Scholar 

  • Hebb DO (1949) The organization of behavior. Wiley, New York

  • Keysers C, Xiao DK, Foldiak P, Perrett DI (2001) The speed of sight. J Cogn Neurosci 13:90–101

    Article  CAS  PubMed  Google Scholar 

  • Kohler E, Keysers C, Umiltà MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848

    Article  CAS  PubMed  Google Scholar 

  • Newsome WT, Britten KH, Movshon JA (1989) Neuronal correlates of a perceptual decision. Nature 341:52–54

    Google Scholar 

  • Rizzolatti G, Arbib MA (1998) Language within our grasp. Trends Neurosci 21:188–194

    CAS  PubMed  Google Scholar 

  • Rizzolatti G, Fadiga L, Gallese V, Fogassi L (1996) Premotor cortex and the recognition of motor actions. Brain Res Cogn Brain Res 3:131–141

    Article  CAS  PubMed  Google Scholar 

  • Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP (1999) Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci 12:1131–1136

    Article  Google Scholar 

  • Umiltà MA, Kohler E, Gallese V, Fogassi L, Fadiga L, Keysers C, Rizzolatti G (2001) I know what you are doing. A neurophysiological study. Neuron 31:155–165

    PubMed  Google Scholar 

  • Watanabe M (1992) Frontal units of the monkey coding the associative significance of visual and auditory stimuli. Exp Brain Res 89:233–247

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

The research was funded by a MIURST and an ESF Grant. E.K. was supported by a Fonds fuer Medizinische Forschung der Universitaet Zuerich fellowship, C.K. by an EU Marie-Curie Fellowship. We thank G. Rizzolatti for help and advice on all aspects of the work, G. Pavan, M. Manghi and C. Fossati for their invaluable help in computing and acoustics, S. Rozzi, M. Matelli, and G. Luppino for their anatomical advice and F. Orzi and P. Rossi for caring for the monkeys.

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to C. Keysers or V. Gallese.

Additional information

The first two authors contributed equally to the work

Appendix: Box 1: the Receiver Operator Characteristic Analysis

Appendix: Box 1: the Receiver Operator Characteristic Analysis

The fundamental issue behind the Receiver Operator Characteristic (ROC) analysis is simple: the brain contains no photoreceptors, and thus has no direct vision of the outside world—instead, it contains neurons that fire a certain number of times. The brain then has to analyze what happens in the outside world based on the firing of its neurons. The ROC analysis calculates how well an imaginary observer of the firing activity of a given neuron could be at detecting a particular target stimulus. In the case at hand, this ideal observer has to decide on each trial if a particular action (his target action or best action) was performed. He has to take this perceptual decision based on the number of spikes produced by an audiovisual mirror neuron on each individual trial.

The decision rule used by the imaginary observer of the neural activity is simple: he decides on a threshold spike-count value theta, and compares the count x i on a given trial i with this threshold. If x i theta, the observer responds that his target action occurred. If x i <theta, he reports that another action must have been performed. The decision of the observer is then compared with what action was really performed on that trial.

The working of the ROC analysis can be best explained by applying the method to two different neurons: one, which firing has nothing to do with what action the experimenter performed in front of the monkey, and one—an audiovisual mirror neuron—that fires more when its best action was performed.

Figure 2A shows the spike-count distribution for the audiovisual mirror neuron. Note how the histogram when the experimenter performed the best action (top of Fig. 2A) is shifted rightwards compared with the lower histogram when the less effective action was performed (bottom). This shift means that the neuron is more likely to produce large spike-counts when the best action is performed. In the example of Fig. 2A, the observer places his theta at an intermediate value (e.g., 20, dotted vertical line in Fig. 2A). Not knowing what action was really performed by the experimenter, the observer applies his decision rule blindly to both conditions. He reports the best action when the spike count is larger than or equal to theta (i.e., at the right of his threshold) in both cases. In our example, he reports the best action in nine out of the ten best action trials (i.e., when the experimenter really performed the best action), and in two of the ten less effective action trials. The former 9/10 are correct decisions, called 'hits' (black bars in Fig. 2A), and his hit rate is thus 90%. The latter 2/10 are errors, called false alarms (gray bars), and his false alarm rate is thus 20%. The hit and false alarm rate depend not only on the response of the neuron, but also on the theta used by the observer: Placing theta very low (i.e., moving the dotted line leftwards), the observer would always report the best action, and both the hit and false alarm rate would approach 1. Choosing very large theta, he would never report the best action, and both the rates will approach 0. Figure 2B illustrates the relationship between the hit rate and the false alarm rate for this audiovisual mirror neuron as a function of theta. This curve, called the receiver operator characteristic curve, is obtained by testing all the possible theta values, plotting the hit and false alarm rates for each theta, and connecting all the points. For this neuron, this curve is very far away from the dotted diagonal, and the surface under the curve is close to 1 (0.96). What does this large surface mean? Given that the two histograms overlap very little, if the observer starts at a theta=42 (bottom left of Fig. 2B), and reduces theta until 23, the observer only responds with best action for best action trials, i.e., the criterion does not include trials of the lower histogram in Fig. 2A, and thus the hit rate increases without increasing the false alarm rate. Only if the observer reduces theta below 23 will he respond with best action also in trials where the worst action was really performed. The curve thus rises vertically until the 80% hit rate, and then moves almost horizontally towards a false alarm of 100%, covering almost all the surface in the box. This is symptomatic for cases where the neuron accurately discriminates between the two types of actions.

In contrast to the audiovisual mirror neuron, let us now consider an imaginary neuron that does not discriminate between the two actions. Figure 2C illustrates the spike counts for this neuron. Note how much overlap there is between the two histograms. Under such conditions, placing theta at 20 means that the observer will respond with best action 6/10 times when the best action and 5/10 times when the worst action was really performed. The result is that the hit and false alarm rate (60 and 50% respectively) are almost equal. Figure 1D illustrates the ROC curve in this case. The curve remains very close to the diagonal, meaning that a decrease of theta increases the hit rate, but at the cost of equally increasing the false alarm rate. The observer is essentially randomly guessing: the spike count gives him no information about what action was performed. The surface under the curve will be close to 0.5 (here 0.54), which is symptomatic for the cases when the activity of the neuron is unrelated to the action performed by the experimenter.

The surface under the ROC curve is thus an indication of the proportion of correct decisions that the observer makes, taking all the thetas into account. From the two examples, it is intuitive that a surface close to 0.5 represents random performance while a surface close to 1 indicates that the observer is very good at telling what action was performed. This surface then gives us valuable information about what function the neuron might have in the brain. If the imaginary observer is very good at telling what action was performed (large surface under the ROC curve), then the brain too could use this neuron to discriminate between the two actions. The neuron could thus participate in the perception of the actions. If the observer is very poor at telling the difference between the two actions using the spike count of this neuron, so would the brain be, and the neuron therefore is unlikely to be involved in the perception of the actions.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Keysers, C., Kohler, E., Umiltà, M.A. et al. Audiovisual mirror neurons and action recognition. Exp Brain Res 153, 628–636 (2003). https://doi.org/10.1007/s00221-003-1603-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-003-1603-5

Keywords

Navigation