Skip to main content
Top
Gepubliceerd in: Psychological Research 8/2021

24-11-2020 | Original Article

The emotion–facial expression link: evidence from human and automatic expression recognition

Auteurs: Anna Tcherkassof, Damien Dupré

Gepubliceerd in: Psychological Research | Uitgave 8/2021

Log in om toegang te krijgen
share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail

Abstract

While it has been taken for granted in the development of several automatic facial expression recognition tools, the question of the coherence between subjective feelings and facial expressions is still a subject of debate. On one hand, the “Basic Emotion View” conceives emotions as genetically hardwired and, therefore, being genuinely displayed through facial expressions. Consequently, emotion recognition is perceiver independent. On the other hand, the constructivist approach conceives emotions as socially constructed, the emotional meaning of a facial expression being inferred by the perceiver. Hence, emotion recognition is perceiver dependent. In order (1) to evaluate the coherence between the subjective feeling of emotions and their spontaneous facial displays, and (2) to compare the recognition of such displays by human perceivers and by an automatic facial expression classifier, 232 videos of expressers recruited to carry out an emotion elicitation task were annotated by 1383 human perceivers as well as by Affdex, an automatic classifier. Results show a weak consistency between self-reported emotional states by expressers and their facial emotional displays. They also show low accuracy both of human perceivers and of the automatic classifier to infer the subjective feeling from the spontaneous facial expressions displayed by expressers. However, the results are more in favor of a perceiver-dependent view. Based on these results, the hypothesis of genetically hardwired emotion genuinely displayed is difficult to support, whereas the idea of emotion and facial expression as being socially constructed appears to be more likely. Accordingly, automatic emotion recognition tools based on facial expressions should be questioned.
Literatuur
go back to reference Camras, L. A., Malatesta, C., & Izard, C. E. (1991). The development of facial expressions in infancy. In R. S. Feldman & B. Rimé (Eds.), Studies in emotion & social interaction. Fundamentals of nonverbal behavior (pp. 31–54). New York: Cambridge University Press. Camras, L. A., Malatesta, C., & Izard, C. E. (1991). The development of facial expressions in infancy. In R. S. Feldman & B. Rimé (Eds.), Studies in emotion & social interaction. Fundamentals of nonverbal behavior (pp. 31–54). New York: Cambridge University Press.
go back to reference Darwin, C. (1872). The expression of the emotions in man and animals. London: John Murray.CrossRef Darwin, C. (1872). The expression of the emotions in man and animals. London: John Murray.CrossRef
go back to reference de Waal, F. (2019). Mama’s last hug: Animal emotions and what they teach us about ourselves. London: Granta Books. de Waal, F. (2019). Mama’s last hug: Animal emotions and what they teach us about ourselves. London: Granta Books.
go back to reference Den Uyl, M., & Van Kuilenburg, H. (2005). The facereader: Online facial expression recognition. Proceedings of Measuring Behavior, 30, 589–590. Den Uyl, M., & Van Kuilenburg, H. (2005). The facereader: Online facial expression recognition. Proceedings of Measuring Behavior, 30, 589–590.
go back to reference Dente, P., Küster, D., Skora, L., & Krumhuber, E. (2017). Measures and metrics for automatic emotion classification via facet. In: Proceedings of the Conference on the Study of Artificial Intelligence and Simulation of Behaviour, pp 160–163. Dente, P., Küster, D., Skora, L., & Krumhuber, E. (2017). Measures and metrics for automatic emotion classification via facet. In: Proceedings of the Conference on the Study of Artificial Intelligence and Simulation of Behaviour, pp 160–163.
go back to reference Duchenne, G.-B. (1876). Mécanisme de la physionomie humaine: Où, analyse électro-physiologique de l’expression des passions. Paris: J.-B. Baillière. Duchenne, G.-B. (1876). Mécanisme de la physionomie humaine: Où, analyse électro-physiologique de l’expression des passions. Paris: J.-B. Baillière.
go back to reference Durán, J. I., Reisenzein, R., & Fernández-Dols, J.-M. (2017). Coherence between emotions and facial expressions. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 107–129). New York: Oxford University Press. Durán, J. I., Reisenzein, R., & Fernández-Dols, J.-M. (2017). Coherence between emotions and facial expressions. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 107–129). New York: Oxford University Press.
go back to reference Ekman, P. (2007). The directed facial action task. In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 47–53). New York: Oxford University Press. Ekman, P. (2007). The directed facial action task. In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 47–53). New York: Oxford University Press.
go back to reference Ekman, P. (2017). Facial expressions. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 39–56). New York: Oxford University Press.CrossRef Ekman, P. (2017). Facial expressions. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 39–56). New York: Oxford University Press.CrossRef
go back to reference Fernández-Dols, J. M. (2017). Natural facial expression: A view from psychological constructionism and pragmatics. In J. M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 457–478). New York: Oxford University Press.CrossRef Fernández-Dols, J. M. (2017). Natural facial expression: A view from psychological constructionism and pragmatics. In J. M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 457–478). New York: Oxford University Press.CrossRef
go back to reference Fridlund, A. J. (1994). Human facial expression: An evolutionary view. San Diego: Academic Press. Fridlund, A. J. (1994). Human facial expression: An evolutionary view. San Diego: Academic Press.
go back to reference Frijda, N. H. (2012). Recognition of emotion in others. In F. Paglieri (Ed.), Consciousness in interaction. The role of the natural and social context in shaping consciousness (pp. 237–256). Amsterdam: J. Benjamins. Frijda, N. H. (2012). Recognition of emotion in others. In F. Paglieri (Ed.), Consciousness in interaction. The role of the natural and social context in shaping consciousness (pp. 237–256). Amsterdam: J. Benjamins.
go back to reference Frijda, N. H., & Tcherkassof, A. (1997). Facial expressions as modes of action readiness. In J. A. Russell & J.-M. Fernández-Dols (Eds.), The psychology of facial expression (pp. 78–102). New York: Cambridge University Press.CrossRef Frijda, N. H., & Tcherkassof, A. (1997). Facial expressions as modes of action readiness. In J. A. Russell & J.-M. Fernández-Dols (Eds.), The psychology of facial expression (pp. 78–102). New York: Cambridge University Press.CrossRef
go back to reference Gendron, M., & Barrett, L. F. (2017). Facing the past: A history of the face in psychological research on emotion perception. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 15–36). Oxford: Oxford University Press. Gendron, M., & Barrett, L. F. (2017). Facing the past: A history of the face in psychological research on emotion perception. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 15–36). Oxford: Oxford University Press.
go back to reference Hess, U., & Fischer, A. H. (2016). Emotional mimicry in social context. New York: Cambridge University Press.CrossRef Hess, U., & Fischer, A. H. (2016). Emotional mimicry in social context. New York: Cambridge University Press.CrossRef
go back to reference Jed Wing, M. K. C., Weston, S., Williams, A., Keefer, C., Engelhardt, A., Cooper, T., Mayer, Z., Kenkel, B., R Core Team, Benesty, M., Lescarbeau, R., Ziem, A., Scrucca, L., Tang, Y., Candan, C., & Hunt., T. (2019). Caret: Classification and regression training. https://CRAN.R-project.org/package=caret. Accessed 6 Nov 2020. Jed Wing, M. K. C., Weston, S., Williams, A., Keefer, C., Engelhardt, A., Cooper, T., Mayer, Z., Kenkel, B., R Core Team, Benesty, M., Lescarbeau, R., Ziem, A., Scrucca, L., Tang, Y., Candan, C., & Hunt., T. (2019). Caret: Classification and regression training. https://​CRAN.​R-project.​org/​package=​caret. Accessed 6 Nov 2020.
go back to reference Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial expression analysis. In: Proceedings Fourth International Conference on Automatic Face and Gesture Recognition, pp 46–53. Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial expression analysis. In: Proceedings Fourth International Conference on Automatic Face and Gesture Recognition, pp 46–53.
go back to reference Kappas, A. (2003). What facial activity can and cannot tell us about emotions. In M. Katsikitis (Ed.), The human face (pp. 215–234). Boston: Springer. Kappas, A. (2003). What facial activity can and cannot tell us about emotions. In M. Katsikitis (Ed.), The human face (pp. 215–234). Boston: Springer.
go back to reference Krause, R., Steimer-Krause, E., Merten, J., & Ullrich, B. (1998). Dyadic interaction regulation, emotion, and psychopathology. In W. F. J. Flack & J. D. Laird (Eds.), Emotions and psychopathology: Theory and research (pp. 70–80). New York: Oxford University Press. Krause, R., Steimer-Krause, E., Merten, J., & Ullrich, B. (1998). Dyadic interaction regulation, emotion, and psychopathology. In W. F. J. Flack & J. D. Laird (Eds.), Emotions and psychopathology: Theory and research (pp. 70–80). New York: Oxford University Press.
go back to reference Lottridge, D., & Chignell, M. (2009). Emotional bandwidth: Information theory analysis of affective response ratings using a continuous slider. In: IFIP Conference on Human-Computer Interaction, 111–114. Lottridge, D., & Chignell, M. (2009). Emotional bandwidth: Information theory analysis of affective response ratings using a continuous slider. In: IFIP Conference on Human-Computer Interaction, 111–114.
go back to reference Meillon, B., Tcherkassof, A., Mandran, N., Adam, J.-M., Dubois, M., Dupré, D., Benoit, A.-M., Guérin-Dugué, A., & Caplier, A. (2010). DynEmo: A corpus of dynamic and spontaneous emotional facial expressions. In: Proceedings of International Workshop Series on Multimodal Corpora, Tools and Resources. Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, pp 31–36. Meillon, B., Tcherkassof, A., Mandran, N., Adam, J.-M., Dubois, M., Dupré, D., Benoit, A.-M., Guérin-Dugué, A., & Caplier, A. (2010). DynEmo: A corpus of dynamic and spontaneous emotional facial expressions. In: Proceedings of International Workshop Series on Multimodal Corpora, Tools and Resources. Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, pp 31–36.
go back to reference Niedenthal, P. M., Wood, A., Rychlowska, M., & Korb, S. (2017). Embodied simulation in decoding facial expression. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 397–414). New York: Oxford University Press. Niedenthal, P. M., Wood, A., Rychlowska, M., & Korb, S. (2017). Embodied simulation in decoding facial expression. In J.-M. Fernández-Dols & J. A. Russell (Eds.), The science of facial expression (pp. 397–414). New York: Oxford University Press.
go back to reference Nielsen, L., & Kaszniak, A. W. (2007). Conceptual, theoretical, and methodological issues in inferring subjective emotion experience. In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 361–375). New York: Oxford University Press. Nielsen, L., & Kaszniak, A. W. (2007). Conceptual, theoretical, and methodological issues in inferring subjective emotion experience. In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 361–375). New York: Oxford University Press.
go back to reference Pacherie, E. (2003). Modes de structuration des contenus perceptifs visuels. In J. Bouveresse & J.-J. Rosat (Eds.), Philosophies de la perception. Phénoménologie, grammaire et sciences cognitives (pp. 263–289). Paris: Odile Jacob. Pacherie, E. (2003). Modes de structuration des contenus perceptifs visuels. In J. Bouveresse & J.-J. Rosat (Eds.), Philosophies de la perception. Phénoménologie, grammaire et sciences cognitives (pp. 263–289). Paris: Odile Jacob.
go back to reference Sander, D., & Scherer, K. (2014). Oxford companion to emotion and the affective sciences. New York: Oxford University Press. Sander, D., & Scherer, K. (2014). Oxford companion to emotion and the affective sciences. New York: Oxford University Press.
go back to reference Tcherkassof, A., Bollon, T., Dubois, M., Pansu, P., & Adam, J.-M. (2007). Facial expressions of emotions: A methodological contribution to the study of spontaneous and dynamic emotional faces. European Journal of Social Psychology, 37(6), 1325–1345. https://doi.org/10.1002/ejsp.427.CrossRef Tcherkassof, A., Bollon, T., Dubois, M., Pansu, P., & Adam, J.-M. (2007). Facial expressions of emotions: A methodological contribution to the study of spontaneous and dynamic emotional faces. European Journal of Social Psychology, 37(6), 1325–1345. https://​doi.​org/​10.​1002/​ejsp.​427.CrossRef
go back to reference Tcherkassof, A., & Frijda, N. H. (2014). Les émotions: Une conception relationnelle. L’Année Psychologique, 114(3), 501–535.CrossRef Tcherkassof, A., & Frijda, N. H. (2014). Les émotions: Une conception relationnelle. L’Année Psychologique, 114(3), 501–535.CrossRef
go back to reference Wagner, H. L. (1997). Methods for the study of facial behavior. In J. A. Russell & J.-M. Fernández-Dols (Eds.), The psychology of facial expression (pp. 31–54). Cambridge: Cambridge University Press.CrossRef Wagner, H. L. (1997). Methods for the study of facial behavior. In J. A. Russell & J.-M. Fernández-Dols (Eds.), The psychology of facial expression (pp. 31–54). Cambridge: Cambridge University Press.CrossRef
Metagegevens
Titel
The emotion–facial expression link: evidence from human and automatic expression recognition
Auteurs
Anna Tcherkassof
Damien Dupré
Publicatiedatum
24-11-2020
Uitgeverij
Springer Berlin Heidelberg
Gepubliceerd in
Psychological Research / Uitgave 8/2021
Print ISSN: 0340-0727
Elektronisch ISSN: 1430-2772
DOI
https://doi.org/10.1007/s00426-020-01448-4

Andere artikelen Uitgave 8/2021

Psychological Research 8/2021 Naar de uitgave