The effect of dynamics on identifying basic emotions from synthetic and natural faces

Abstract

The identification of basic emotions (anger, disgust, fear, happiness, sadness and surprise) has been studied widely from pictures of facial expressions. Until recently, the role of dynamic information in identifying facial emotions has received little attention. There is evidence that dynamics improves the identification of basic emotions from synthetic (computer-animated) facial expressions [Wehrle, T., Kaiser, S., Schmidt, S., Scherer, K.R., 2000. Studying dynamic models of facial expression of emotion using synthetic animated faces. Journal of Personality and Social Psychology 78 (1), 105–119.]; however, similar result has not been confirmed with natural human faces. We compared the identification of basic emotions from both natural and synthetic dynamic vs. static facial expressions in 54 subjects. We found no significant differences in the identification of static and dynamic expressions from natural faces. In contrast, some synthetic dynamic expressions were identified much more accurately than static ones. This effect was evident only with synthetic facial expressions whose static displays were non-distinctive. Our results show that dynamics does not improve the identification of already distinctive static facial displays. On the other hand, dynamics has an important role for identifying subtle emotional expressions, particularly from computer-animated synthetic characters.

Introduction

Faces provide crucial information in social communication. Static facial features are important for identifying identity, gender and age from faces. Transient changes on face, driven by complex facial musculature, convey both verbal (speech) and non-verbal information. Facial expressions regulate turn-taking between speakers, emphasize speech, convey culture-specific signals and, importantly, reflect feelings of the speakers (Pelachaud et al., 1991).

A long research tradition suggests that at least six emotions (anger, disgust, fear, happiness, sadness and surprise) are ‘basic’ because they are identified distinctively from their characteristic facial expressions in all human cultures (Ekman et al., 1982). Although this claim has been debated, sometimes heatedly (e.g., Ortony and Turner, 1990; Ekman, 1994; Izard, 1994; Russell, 1994, Russell, 1995), several studies have supported the consistent identification of basic facial expressions in different cultures. Studies with a forced-choice task requiring participants to match presented facial expressions with a given list of emotion labels have confirmed that basic expressions are most often matched with intended basic emotions (e.g., Ekman, 1973, Ekman, 1984; Ekman et al., 1982). Similar results have been found when participants have been asked to rate the intensity of each basic emotion in stimuli (e.g., Ekman et al., 1987) or to produce emotional labels for them freely (e.g., Rosenberg and Ekman, 1995; Haidt and Keltner, 1999). Such studies have also shown that although basic expressions are most often identified as their intended emotions, certain confusions are common. Most notably, fearful faces tend to be confused with surprise, and angry and disgusted faces with each other, whereas happy and surprised faces are seldomly confused with any other emotion. Happy and surprised faces are typically identified the best and anger, disgust and fear the worst.

Studies of basic emotions have typically utilized pictures of posed facial expressions such as the Ekman–Friesen collection of facial affects (Ekman and Friesen, 1978). The use of posed instead of authentic emotional facial expressions has raised questions on the ecological validity of such studies (e.g., Russell, 1994). Trivially, authentic everyday expressions are more natural than posed expressions. However, the latter ones do exhibit some advantages over the former. When using posed stimuli, actors can be trained in posing certain theoretically derived facial configurations exactly, producing homogeneous and distinctive emotional displays. On the other hand, authentic emotional expressions are more heterogeneous and their emotional content is typically ambiguous (cf. Ekman, 1973). For example, in a recently published collection of authentic basic expressions (O’Toole et al., 2005), several instances of happy facial displays were evoked with few instances of anger and fear (O’Toole, personal communication). Perhaps a more important issue than the use of posed instead of authentic emotional displays is the predominant use of pictures instead of moving emotional stimuli. Because most studies have been conducted with static facial pictures, the role of dynamic information on perceiving facial emotions has received little attention.

Previous studies have shown that dynamics (head movement and facial expression transitions) may have an important role in recognizing identity and age from faces. Identity recognition is to some extent possible from dynamic point-light displays (moving points) extracted from original faces of actors (Bruce and Valentine, 1988). Both identity and sex can be recognized when original facial movements are replicated on a computer-animated head showing none of the original static features (Hill and Johnston, 2001). These studies indicate that movement alone conveys some information about person's identity and sex. Direct comparisons between static and dynamic displays have shown that observing movement enhances identity identification from faces when their presentation has been degraded by inversion, pixelation, blurring, luminance value thresholding or color negation (Knight and Johnston, 1997; Lander, 1999, Lander, 2001).

There is evidence of the importance of dynamics also in identifying emotions from facial expressions. Studies with dynamic point-light displays have indicated that facial emotions can be identified from pure movement information (Bassili, 1978; Bruce and Valentine, 1988). Some neurological patients impaired in identifying emotions from still images of facial expressions do nevertheless recognize them from video sequences (Adolps et al., 2003) and point-light displays (Humphreys et al., 1993). By using schematic animations as stimuli, Wehrle et al. (2000) found better identification of dynamic rather than static displays of emotions. However, it is not clear whether this result was specific to synthetic facial stimuli because their results were not compared to dynamic vs. static natural facial expressions. Importantly, the synthetic static stimuli were identified worse than their static natural counterparts, suggesting that their result should not be generalized to natural faces. Studies using natural facial expressions have provided inconsistent results. Harwood et al. (1999) reported better identification of dynamic over static facial displays of emotions; however, such effect was observed only for anger and sadness. Kamachi et al. (2001) used image morphing to generate a video of a face changing from neutral to emotional, and found no difference between such dynamic and original static displays. Ehrlich et al. (2000) have reported better identification of basic emotions from dynamic rather than static facial expressions. However, because their results were pooled over good-quality facial expressions and their degraded versions, it is possible that the better identification of dynamic expressions was specific to the degraded stimuli. Recently, Ambadar et al. (2005) demonstrated that dynamics improves the identification of subtle, low-intensity, facial emotion expressions. However, full intensity facial expressions were not used as control stimuli. As a conclusion, dynamics appears to improve the identification of facial emotions from synthetic faces whose static presentations are not identified optimally. It is not established whether this effect applies also to full-intensity and non-degraded natural facial expressions.

The aim of the present study was to compare the identification of basic expressions from static and dynamic natural and synthetic faces in the same experiment. Natural stimuli consisted of posed, clearly distinguishable facial expressions of basic emotions. Synthetic stimuli were created with a three-dimensional head animation model (Frydrych et al., 2003). The used model did not capture all realistic facial details (cf. Section 2.2.1). Our hypothesis was that dynamics has no effect on the identification of already well-recognizable natural facial emotions but that it does improve the identification of those synthetic facial animations that are identified poorly from static displays.