The effect of dynamics on identifying basic emotions from synthetic and natural faces
Introduction
Faces provide crucial information in social communication. Static facial features are important for identifying identity, gender and age from faces. Transient changes on face, driven by complex facial musculature, convey both verbal (speech) and non-verbal information. Facial expressions regulate turn-taking between speakers, emphasize speech, convey culture-specific signals and, importantly, reflect feelings of the speakers (Pelachaud et al., 1991).
A long research tradition suggests that at least six emotions (anger, disgust, fear, happiness, sadness and surprise) are ‘basic’ because they are identified distinctively from their characteristic facial expressions in all human cultures (Ekman et al., 1982). Although this claim has been debated, sometimes heatedly (e.g., Ortony and Turner, 1990; Ekman, 1994; Izard, 1994; Russell, 1994, Russell, 1995), several studies have supported the consistent identification of basic facial expressions in different cultures. Studies with a forced-choice task requiring participants to match presented facial expressions with a given list of emotion labels have confirmed that basic expressions are most often matched with intended basic emotions (e.g., Ekman, 1973, Ekman, 1984; Ekman et al., 1982). Similar results have been found when participants have been asked to rate the intensity of each basic emotion in stimuli (e.g., Ekman et al., 1987) or to produce emotional labels for them freely (e.g., Rosenberg and Ekman, 1995; Haidt and Keltner, 1999). Such studies have also shown that although basic expressions are most often identified as their intended emotions, certain confusions are common. Most notably, fearful faces tend to be confused with surprise, and angry and disgusted faces with each other, whereas happy and surprised faces are seldomly confused with any other emotion. Happy and surprised faces are typically identified the best and anger, disgust and fear the worst.
Studies of basic emotions have typically utilized pictures of posed facial expressions such as the Ekman–Friesen collection of facial affects (Ekman and Friesen, 1978). The use of posed instead of authentic emotional facial expressions has raised questions on the ecological validity of such studies (e.g., Russell, 1994). Trivially, authentic everyday expressions are more natural than posed expressions. However, the latter ones do exhibit some advantages over the former. When using posed stimuli, actors can be trained in posing certain theoretically derived facial configurations exactly, producing homogeneous and distinctive emotional displays. On the other hand, authentic emotional expressions are more heterogeneous and their emotional content is typically ambiguous (cf. Ekman, 1973). For example, in a recently published collection of authentic basic expressions (O’Toole et al., 2005), several instances of happy facial displays were evoked with few instances of anger and fear (O’Toole, personal communication). Perhaps a more important issue than the use of posed instead of authentic emotional displays is the predominant use of pictures instead of moving emotional stimuli. Because most studies have been conducted with static facial pictures, the role of dynamic information on perceiving facial emotions has received little attention.
Previous studies have shown that dynamics (head movement and facial expression transitions) may have an important role in recognizing identity and age from faces. Identity recognition is to some extent possible from dynamic point-light displays (moving points) extracted from original faces of actors (Bruce and Valentine, 1988). Both identity and sex can be recognized when original facial movements are replicated on a computer-animated head showing none of the original static features (Hill and Johnston, 2001). These studies indicate that movement alone conveys some information about person's identity and sex. Direct comparisons between static and dynamic displays have shown that observing movement enhances identity identification from faces when their presentation has been degraded by inversion, pixelation, blurring, luminance value thresholding or color negation (Knight and Johnston, 1997; Lander, 1999, Lander, 2001).
There is evidence of the importance of dynamics also in identifying emotions from facial expressions. Studies with dynamic point-light displays have indicated that facial emotions can be identified from pure movement information (Bassili, 1978; Bruce and Valentine, 1988). Some neurological patients impaired in identifying emotions from still images of facial expressions do nevertheless recognize them from video sequences (Adolps et al., 2003) and point-light displays (Humphreys et al., 1993). By using schematic animations as stimuli, Wehrle et al. (2000) found better identification of dynamic rather than static displays of emotions. However, it is not clear whether this result was specific to synthetic facial stimuli because their results were not compared to dynamic vs. static natural facial expressions. Importantly, the synthetic static stimuli were identified worse than their static natural counterparts, suggesting that their result should not be generalized to natural faces. Studies using natural facial expressions have provided inconsistent results. Harwood et al. (1999) reported better identification of dynamic over static facial displays of emotions; however, such effect was observed only for anger and sadness. Kamachi et al. (2001) used image morphing to generate a video of a face changing from neutral to emotional, and found no difference between such dynamic and original static displays. Ehrlich et al. (2000) have reported better identification of basic emotions from dynamic rather than static facial expressions. However, because their results were pooled over good-quality facial expressions and their degraded versions, it is possible that the better identification of dynamic expressions was specific to the degraded stimuli. Recently, Ambadar et al. (2005) demonstrated that dynamics improves the identification of subtle, low-intensity, facial emotion expressions. However, full intensity facial expressions were not used as control stimuli. As a conclusion, dynamics appears to improve the identification of facial emotions from synthetic faces whose static presentations are not identified optimally. It is not established whether this effect applies also to full-intensity and non-degraded natural facial expressions.
The aim of the present study was to compare the identification of basic expressions from static and dynamic natural and synthetic faces in the same experiment. Natural stimuli consisted of posed, clearly distinguishable facial expressions of basic emotions. Synthetic stimuli were created with a three-dimensional head animation model (Frydrych et al., 2003). The used model did not capture all realistic facial details (cf. Section 2.2.1). Our hypothesis was that dynamics has no effect on the identification of already well-recognizable natural facial emotions but that it does improve the identification of those synthetic facial animations that are identified poorly from static displays.
Section snippets
Participants
Participants were 54 university students (36 males, 18 females; 20–29 years old) from Helsinki University of Technology (TKK) who participated in the experiment as a part of their studies. All participants were native speakers of Finnish and had either normal or corrected vision. The level of subjects’ alexithymic personality trait (Taylor et al., 1991) was evaluated with a Toronto Alexithymia Scale (TAS-20) self-report questionnaire (Bagby et al., 1994). Alexithymia is defined as involving
Effect of dynamics
A mixed-design ANOVA with factors Display (static, dynamic), Type (natural (CK and TKK), synthetic (TH)), and Expression (six basic expressions) was used with naturalness ratings and identification scores to evaluate the significance of the first factor and its interactions with the other factors. Control stimuli (EF) were excluded from this analysis as they were always static. With naturalness ratings, only the interaction between Display and Expression reached significance (F(5,260)=3.12, p
Discussion
We studied the effect of dynamics on identifying six basic emotions from natural (human actors’) and synthetic (computer-animated) facial expressions. Our results showed no significant differences in the identification of static and dynamic expressions from natural faces. In contrast, dynamics increased the identification of synthetic facial expressions, particularly those of anger and disgust. Although our static synthetic stimuli were identified generally worse than their natural
Acknowledgments
We thank Robotics Institute of Carnegie Mellon University for access to the Cohn–Kanade facial expression database. We thank Michael Frydrych, Martin Dobšik and Vasily Klucharev for insightful discussions and for their concrete contributions to the present study. This study was supported partly by the Academy of Finland (Grant 213938 to M.S.; Finnish Centre of Excellence Programs 2000–2005 and 2006–2011, Grant nos. 202871 and 213470).
References (54)
- et al.
Subtle emotional expressions of synthetic characters
Journal of Human–Computer Studies
(2005) - et al.
Categorizing sex and identity from the biological motion of faces
Current Biology
(2001) - et al.
Expression is computed separately from facial identity, and it is computed separately for moving and static faces: neuropsychological evidence
Neuropsychologia
(1993) - et al.
The many faces of configural processing
Trends in Cognitive Science
(2002) - et al.
Prevalence of alexithymia and its association with sociodemographic variables in the general population of Finland
Journal of Psychosomatic Research
(1999) - et al.
The Alexithymia construct. A potential paradigm for psychosomatic medicine
Psychosomatics
(1991) - et al.
Dissociable neural systems for recognizing emotions
Brain and Cognition
(2003) - et al.
Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions
Psychological Science
(2005) - et al.
The twenty-item Toronto Alexithymia Scale
Journal of Psychosomatic Research
(1994) - et al.
Another advanced test of theory of mind: evidence from very high functioning adults with autism or Asperger Syndrome
Journal of Child Psychology and Psychiatry
(1997)
The “Reading the mind in the eyes” test revised version: a study with normal adults, and adults with Asperger Syndrome or high-functioning autism
Journal of Child Psychology and Psychiatry
How convincing is Mr. Data's smile: affective expressions of machines
User Modeling and User-Adapted Interaction
Facial motion in the perception of faces and of emotional expression
Journal of Experimental Psychology: Human Perception and Performance
Principles and procedures of exploratory data analysis
Psychological Methods
When a nod's as good as a wink: the role of dynamic information in face recognition
Cross-cultural studies of facial expression
Expression and the nature of emotion
Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique
Psychological Bulletin
Unmasking the Face. A Guide to Recognizing Emotions from Facial Expressions
Pictures of Facial Affect
What emotion categories or dimensions can observes judge from facial behavior?
Universals and cultural differences in the judgments of facial expressions of emotion
Journal of Personality and Social Psychology
Facial Action Coding System
Facial Action Coding System: Investigator's Guide
Culture and facial expression: open-ended methods find more expressions and a gradient of recognition
Cognition and Emotion
Cited by (67)
The role of facial skin tone and texture in the perception of age
2023, Vision ResearchPrivacy-preserving facial recognition based on temporal features
2020, Applied Soft Computing JournalCitation Excerpt :Thus, this kind of stimuli is more valid than the other dynamic face stimuli ecologically. Findings from the studies in the past suggested that the advantages due to the dynamic face stimuli are more obvious to subtle facial expressions compared to basic facial expressions [10,23,24]. Note that this is different from our aim.
The power of facial expressions in branding: can emojis versus human faces shape emotional contagion and brand fun?
2024, Journal of Brand ManagementAge-dependent changes in the anger superiority effect: Evidence from a visual search task
2024, Psychonomic Bulletin and ReviewThe role of facial movements in emotion recognition
2023, Nature Reviews Psychology