Effects of affective and emotional congruency on facial expression processing under different task demands

Highlights

• Facial expressions of emotion are processed differently depending on the context.

• Context effects depend on task demands (valence, emotion or congruency tasks).

• Different context effects are obtained with positive and negative facial expressions.

Abstract

Contextual influences on responses to facial expressions of emotion were studied using a context-target paradigm that allowed distinguishing the effects of affective congruency (context and target of same/different valence: positive or negative) and emotional congruency (context and target representing the same/different emotion: anger, fear, happiness). Sentences describing anger, fear or happiness-inducing events and faces expressing each of these emotions were used as contexts and targets, respectively. While between-valence comparisons (context and target of similar/different valence) revealed affective congruency effects, within-valence comparisons (context and target of similar valence and same/different emotion) revealed emotional congruency effects. In Experiment 1 no evidence of emotional congruency and limited evidence of affective congruency were found with an evaluative task. In Experiment 2 effects of both affective and emotional congruency were observed with an emotion recognition task. In this case, angry and fearful faces were recognized faster in emotionally congruent contexts. In Experiment 3 the participants were asked explicitly to judge the emotional congruency of the target faces. Emotional congruency effects were again found, with faster judgments of angry and fearful faces in the corresponding emotional contexts. Moreover, judgments of angry expressions were faster and more accurate in happy than in anger contexts. Thus, participants found easier to decide that angry faces did not match a happy context than to judge that they did match an anger context. These results suggest that there are differences in the way that facial expressions of positive and negative emotions are discriminated and integrated with their contexts. Specifically, compared to positive expressions, contextual integration of negative expressions seems to require a double check of the valence and the specific emotion category of the expression and the context.

Introduction

Recent research has shown that processing of facial expressions of emotion is modulated by the context in which they are perceived (see Wieser & Brosch, 2012, for a review). Modulation occurs when recognition of the expression is facilitated or impaired depending on its congruency with the context (e.g., Diéguez-Risco, Aguado, Albert, & Hinojosa, 2013; Righart & de Gelder, 2008a) or when contextual information disambiguates an ambiguous expression (e.g., Kim et al., 2004). We focus here on the situational context in which facial expressions are perceived, that is, the physical or social environment in which the face showing a given expression is embedded. Operationally, manipulation of situational contexts has involved presenting a face along with a pictorial or verbal description of the related situation. Emotion recognition is more efficient when the expression and the context are congruent (e.g., fearful face/threat context) than when they are incongruent (e.g., fearful face/happy context). For example, in a study by Righart and de Gelder (2008a) happy, fearful or disgusted faces were presented superimposed on pictures representing different scenes. Faster recognition of target expressions was observed in those trials in which the expression was congruent with the context. Similar results have been reported in other studies with either pictorial or verbal contexts (Diéguez-Risco, Aguado, Albert, & Hinojosa, 2015; Diéguez-Risco et al., 2013; Hietanen & Astikainen, 2013; Righart & de Gelder, 2008b).

When speaking of contextual congruency it is important to be precise about its meaning. A specific context-target combination, for example a sad face presented on the background of an anger scene, can be said to be affectively congruent because both the expression and the context have negative valence. This would be an example of affective congruency. On the other hand, this same combination can be said to be emotionally incongruent in the sense that the expression and the context represent different specific emotions, sadness and anger, respectively. Thus, the expression of sadness is at the same time affectively congruent and emotionally incongruent with the context. Discriminating the effects of affective and emotional congruency requires a design including stimuli related to one positive emotion (happiness) and at least two negative emotions (e.g., fear and anger). The comparison between affectively congruent and incongruent trials (between-valence effects) would reveal the effects of affective congruency. On the other hand, the comparison between trials that are affectively congruent but emotionally incongruent (within-valence effects) should reveal the effects of emotional congruency. Although a similar design has been used in some studies (Dozolme, Brunet-Gouet, Passerieux, & Amorim, 2015; Righart & de Gelder, 2008a) the relative influence of affective and emotional congruency was not explicitly considered. For example, Dozolme et al. used target faces expressing sadness, joy, fear or anger, that were presented after the participant had read a sentence describing joy-, fear- or anger-inducing situations. However, affective and emotional congruencies were confounded in the analysis given that the data were averaged based only on affective congruency.

The distinction between affective and emotional congruency is closely related to the contrasting dimensional and categorical approaches to the study of emotion. Affective congruency fits well with the dimensional approach that considers emotions as varying along two general dimensions of valence and arousal (e.g., Bradley, 2000). On the other hand, emotional congruency fits better with a characterization of affect in terms of specific emotional categories as proposed by the discrete or basic emotion theories (e.g., Ekman, 1992; Izard, 1993). Based on these theories, affective stimuli would be also processed and differentiated in terms of their specific thematic content (e.g., threat-related vs loss-related). In fact, empirical evidence suggest that the dimensional and categorical approaches to emotion are complementary as they capture fundamental features and components of emotional phenomena at the behavioral, psychophysiological, experiential and brain systems levels. For example, although some psychophysiological responses show differential sensitivity to variations in the dimensions of valence and arousal (e.g., Bradley, 2000), changes in expressive behavior including body posture, facial expression and voice intonation are instead associated to different specific emotional states (e.g. Dael, Mortillaro, & Scherer, 2012; Matsumoto, Keltner, Shiota, O'Sullivan, & Frank, 2008). Taking facial expression as a main example of expressive behavior our main question here refers to the extent to which affective and emotional congruency influence contextual integration of facial expressions of emotion.

While there is much experimental evidence on the influence of affective congruency on the processing of affective stimuli, the role of emotional congruency has received less attention. Some studies have considered the effects of the context provided by expressive body postures on the categorization of facial expressions (e.g., Aviezer et al., 2008; Aviezer, Trope, & Todorov, 2012) showing that both valence attribution and emotion categorization are highly malleable and sensitive to contextual influences. There is also evidence that the social context in which facial expressions are perceived modulates emotion recognition. This was the conclusion of a study by Mumenthaler and Sander (2015) in which recognition of fearful faces was improved and recognition of blended fear/surprise expressions biased by concurrent, subliminally presented faces showing angry expressions or gazing directly at the observer. The results of these studies suggest that both the valence and the specific emotional meaning of the context modulate perception of facial expressions of emotion. In a different vein, Carroll and Young (2005) have shown categorical priming effects using different combinations of stimuli as primes and targets (e.g., words and faces). In Experiments 2 and 3 it was found that facial expressions of different basic emotions were recognized faster in related trials (same emotion for prime and target) than in unrelated ones (different emotion for prime and target). However, the interpretation of this result is complicated by the fact that the analyses were based on the averages of all trials of the related or the unrelated conditions, leading to a potential confound of valence and emotion category effects. Acknowledging this possibility, the authors reanalyzed their results removing all trials with positive targets. Although priming of facial expressions by auditory primes (Experiment 3) survived this analysis, priming by affective pictures (Experiment 2) was significantly reduced. A further possibility not considered by the authors but that might also complicate interpretations based on data averaged over trials with positive and negative targets is that congruency based on emotion categories of different valence might not have uniform effects. One of the main goals of the present study was precisely to investigate the possibility that the mechanisms involved in the computation of the contextual congruency of facial expressions differ between emotions of positive and negative valence. In the following section we delineate the reasons why this possibility should be explicitly considered when studying the effects of affective and emotional congruency.

Although qualitative differences between different positive emotions in terms of experience and appraisal have been described (e.g., Cohn & Fredrickson, 2009; Cohn, Fredrickson, Brown, Mikels, & Conway, 2009; Morrone-Strupinsky & Lane, 2007), a smiling face is the common expressive hallmark of all of them. Because of this, affective and emotional congruencies are coincident in the case of smiling target faces (a happy face “goes” with any positive context but does not go with any negative context). In contrast to this, the fact that there are different configurations of facial movements corresponding to different negative emotions may make the computation of congruency more difficult in the case of negative targets. Affective and emotional congruency will be not coincident on trials in which the context and the target are both negative but represent different specific emotions (e.g., a sad face presented on an anger context). In this case, computing the congruency between expression and context will require a double check of affective and emotional meaning (see Aguado, Dieguez-Risco, Méndez-Bértolo, Pozo, & Hinojosa, 2013, for an application of this interpretation in the context of an affective priming procedure). It can thus be predicted that congruency effects that go beyond affective valence should be found in the case of negative expressions. More specifically, within-valence congruency effects would be expected when the context and the target represent different negative emotions. Moreover, between-valence congruency effects on trials in which the target and the context have different valence might go in the opposite direction to that usually seen in affective priming studies. For example, responses to negative targets on affectively incongruent trials should be faster than on affectively congruent trials. This would be because responses to the target in affectively incongruent trials only require a valence check. In contrast, a double check of valence and emotion would be needed in trials with negative contexts and targets.

A second goal of our study was to assess the role of task demands in modulating the relative influence of affective and emotional congruency. Context and priming studies with facial expression targets have assigned the participants tasks that do not require explicit attention to the relationship between the target and its context. Most usually, the participants have to identify the emotion shown by the target face or evaluate it as positive or negative, although in other cases they may be simply asked to passively view the faces (Kim et al., 2004) or perform an unrelated task (Righart & de Gelder, 2006). Although alternative interpretations have been proposed (e.g. Klauer & Musch, 2003; Klinger, Burton, & Pitts, 2000) congruency effects obtained under these conditions suggest that the affective valence of contexts and prime stimuli is automatically activated and influences target processing by means of implicit mechanisms. However, there is also evidence that congruency effects are more likely when the participant is assigned an evaluative task. For example, using positive and negative words as primes and targets Klauer and Musch (2003) did not obtain evidence of affective priming in a series of studies in which the participant was simply asked to read aloud the target word (see Herring et al., 2013, for a review). This suggests that different task orientations increase the saliency of different features of the stimuli, thus determining the way in which primes and contexts influence target processing. A main distinction would be between tasks that orient the participant to general affective properties of the stimulus (evaluative tasks) and those that require attention to more specific affective properties (e.g., emotion recognition). These alternative orientations would foster categorization of the target at different levels of generality, as an affectively positive or negative face or as fearful, angry or happy face, respectively. Moreover, different task demands might also determine the extent to which the context or prime stimulus is task-relevant. For example, the results of a study by Feldman-Barrett and Kensinger (2010) showed that attention to the context in which an expressive face was presented was promoted by the instruction to identify the emotion shown by the face, compared to the instruction to make a general affective evaluation such as approach or avoid. More specifically, the participants showed better memory of the contexts after an emotion identification task. This result indicates that even when the instructions do not make any reference to the context, its relevance increases automatically due to the demands of the task.

In the present study we report the results of three experiments using a context-target procedure in which happy, angry or fearful face targets were presented immediately after the participant had read a short sentence describing happiness, anger or fear-eliciting events. The main manipulations were the emotional congruency between the sentence and the face, that was varied within-experiment and the specific task assigned to the participants that varied between experiments. The use of happy, angry and fearful faces allowed us to test for affective congruency effects in trials in which the context and the target face differed in valence (e.g., angry face/happy context) and categorical emotion effects in trials in which the context and the target had similar valence but different emotion content (e.g. angry face/fear context). Moreover, the use of different tasks allowed us to evaluate the extent to which congruency effects vary with tasks demands that put different emphasis on the processing of valence and emotion category.

The task assigned in Experiment 1 was to evaluate the face as pleasant or unpleasant (evaluative judgment). In Experiment 2, the participant had to identify the expression shown by the target face (emotion recognition). Finally, in Experiment 3 the participants were asked to judge whether the expression shown by the face was congruent or incongruent with the context (congruency judgment). This last condition is similar to that used in previous studies by Diéguez-Risco et al. (2015) and Dozolme et al. (2015). The three task conditions differ in terms of the degree to which the emotional content of the context is explicitly relevant for the task at hand. Task-independent congruency effects, possibly based on automatic processes, should result in similar effects under all task conditions. However, if congruency effects would be under the influence of strategic, task-dependent processes, different results should be observed under different task conditions. Based on the results of the aforementioned study by Barret and Kensinger (2010) we assumed that the emotional content of the context would be made more relevant in the recognition and congruency tasks and that this would increase the probability to find effects of emotional congruency. Even in the absence of explicit instructions to attend to the context, the emotion recognition task would increase the relative salience of emotion-specific information, thus potentiating the effects of emotional congruency. We further hypothesized that explicit attention to the emotional content of the context in the congruency task would result in a larger effect of emotional congruency. Finally the design allowed us to test a prediction that is privative of the double-check hypothesis, namely that responses to negative faces (angry or fearful) should be slower and/or less accurate in negative than in positive contexts because in the first case double check of valence and emotion category is needed.