Processing of facial expressions in peripheral vision: Neurophysiological evidence
Graphical abstract
Time course and ERP components and processes differentiating happy from non-happy expressions in peripheral vision.
Introduction
Facial expressions reflect a person's emotional state, current motives, and intentions. It is therefore important for adaptive purposes that the cognitive system can rapidly extract accurate information from the observed expressions. Furthermore, for a viewer, faces (and other stimuli) often appear initially in the visual periphery (e.g., among other objects or within a group of people), prior to attracting overt attention. It would thus be highly useful if expressions could be processed in extrafoveal vision, before or in the absence of eye fixations. This would speed up selective orienting to the most significant faces and people in a given situation for further analysis, thus bypassing the slower sequential selection with foveal vision. A major question driving the current study was whether emotional facial expressions can be recognized in peripheral vision and, if so, what is encoded of them and when, and why some expressions might have a recognition advantage in such conditions.
A bulk of studies has investigated the processing of facial expressions appearing in central or foveal vision. Electrophysiological research has revealed that expression modulates a wide range of ERP (event-related-potential) components such as P1 and N1 (100–150-ms peak latency from stimulus onset; e.g., Luo, Feng, He, Wang, & Luo, 2010), N170 (150–200 ms; e.g., Williams, Palmer, Liddell, Song, & Gordon, 2006), vertex positive potential (VPP; 150–200 ms; e.g., Smith, Weinberg, Moran, & Hajcak, 2013), P2 (150–275 ms; e.g., Calvo, Marrero, & Beltrán, 2013), early posterior negativity (EPN; 200–350 ms; e.g., Rellecke, Palazova, Sommer, & Schacht, 2011), P3 and late positive potential (LPP; 350–700 ms; e.g., Frühholz, Fehr, & Herrmann, 2009). Relatedly, behavioral research has shown discrimination among the six basic emotions (happy, sad, angry, fearful, disgusted, and surprised) in categorization or explicit recognition tasks (e.g., Calvo and Lundqvist, 2008, Nelson and Russell, 2013, Palermo and Coltheart, 2004, Tottenham et al., 2009).
However, research on facial expression processing in extrafoveal vision has been sparse. With ERP measures, two studies found early modulations by fearful relative to neutral expressions in parafoveal (2.4°, stimulus eccentricity; Wijers & Banis, 2012) or peripheral (15° or 30°; Rigoulot et al., 2011) vision. Wijers and Banis (2012) observed an enhanced positivity in response to fearful faces from 160 ms post-stimulus onset at lateral–occipital sites, and from 220 ms at midline sites. Rigoulot et al. (2011) reported an increased negativity at temporo-occipital sites between 140 and 240 ms for fearful faces. In two additional studies, happy faces were included. Rigoulot, D’Hondt, Honoré, and Sequeira (2012) noted that peripherally presented (15° or 30°) fearful and happy faces increased the activity peaking at 100, 120 and 150 ms (fearful) and at 95 and 150 ms (happy) after the stimulus onset at temporal–occipital sites. Nevertheless, fearful and happy faces were not directly compared. Stefanics, Csukly, Komlósi, Czobor, and Czigler (2012) compared fearful and happy faces (∼6°). Both expressions increased negativity over temporo-occipital sites in the 150–220 ms range and also in the 250–360 ms interval, although the effect started earlier for fearful faces (70–120 ms).
With behavioral measures, Goren and Wilson (2006) observed that recognition accuracy was impaired, relative to central vision, for sad, angry, and fearful faces, but not for happy faces, presented peripherally (5.5°; 110-ms display). Calvo, Nummenmaa, and Avero (2010) prevented overt attention to parafoveal stimuli (2.5°; 150 ms) by means of gaze-contingent foveal masking. Recognition sensitivity (A′) was above chance level for all six basic expressions, and happy faces were recognized faster than the others. Calvo and Nummenmaa, 2009, Calvo and Nummenmaa, 2011 presented pairs of faces parafoveally (2.5°; 30 ms) each with a different expression. As assessed by saccadic response latencies in a discrimination task, the identification of expressions started between 180 and 280 ms post-stimulus, with an advantage (180 ms) for happy faces (Calvo and Nummenmaa, 2009, Calvo and Nummenmaa, 2011). Assuming a delay of about 25 ms for saccade programming (Schiller & Kendall, 2004), these data imply that expressions can be recognized between 155 and 255 ms.
Accordingly, there is some evidence of early facial expression encoding in peripheral vision. Nevertheless, a limited number of studies have addressed this issue, and only a few expression categories have been included in prior ERP research (fearful vs. neutral: Rigoulot et al., 2011; and Wijers & Banis, 2012; fearful vs. neutral, or happy vs. neutral: Rigoulot et al., 2012; or fearful vs. happy: Stefanics et al., 2012). Importantly, a larger range of comparisons would be required to examine the extent to which there is discrimination among different emotions and when this occurs. To deal with this issue, we used happy, angry, fearful, sad, and neutral faces in the current study, within the same experimental design. The comparison of ERP modulations across multiple emotional expressions is expected to reveal the degree of discrimination subtlety. If there is expression discrimination, not only will emotional faces enhance ERP amplitudes relative to neutral faces, but ERP modulations should differ in their time course for positively vs. negatively valenced expressions, and among the various negative expressions. Depending on whether perceptual, affective, or categorical content is encoded, the specific ERPs involved will vary (see Sections 1.2 Is affective significance encoded in peripheral vision, or only perceptually salient facial features?, 1.3 Is there any neural time course primacy in the recognition of some expressions in peripheral vision, and why?).
Researchers have often assumed that ERP differences between emotional and neutral faces reflect affective encoding. Some ERP components are particularly sensitive to the emotional content of facial expressions, such as the VPP (Willis, Palermo, Burke, Atkinson, & McArthur, 2010), N2 (Williams et al., 2006), and EPN (Schupp et al., 2004), or, more controversially, the N170 (see Rellecke, Sommer, & Schacht, 2013), for faces presented to central vision. Relatedly, behavioral research using affective priming paradigms has provided evidence that affect is indeed differentially extracted from expressions when face stimuli appear at fixation (e.g., Calvo et al., 2012, Carroll and Young, 2005, Lipp et al., 2009, McLellan et al., 2010). The question is whether affective significance is extracted also in peripheral vision. Is encoding driven by an affective evaluation of the expression or, rather, by the mere perceptual analysis of visually salient physical features in a face? In the latter case, expression recognition and categorization could be performed on the basis of perceptual processing, devoid of affective processing. The hypothesis of perceptual rather than affective encoding is particularly relevant to peripheral vision. The reason is that visual acuity degradation typically occurs for eccentric stimuli in the visual field. Accordingly, the visual saliency of certain morphological features (e.g., a smile, a frown, open eyes, etc.) could make some expressions (e.g., happy, angry, fearful, etc.) particularly resistant to acuity degradation, thus remaining visible to some extent.
Visual saliency is an index of the perceptual prominence of an image region in relation to its surroundings, and involves a combination of physical properties such as luminance, contrast, spatial orientation, and color intensity (Itti & Koch, 2000). Saliency is proposed to guide initial shifts of covert and overt attention, with saliency weights being updated after each attentional shift (see Borji & Itti, 2013). Supporting evidence has shown that attentional orienting is affected by saliency weights in a visual scene (Underwood & Foulsham, 2006) and a face (Calvo & Nummenmaa, 2008). Facial expressions differ in their morphological features, some of which are especially salient and thus more accessible to peripheral vision. The smiling mouth of happy faces is more salient than any other region of all the basic expressions (Calvo & Nummenmaa, 2008). It is therefore possible that – due to saliency – happy faces remain perceptually accessible in peripheral vision, regardless and in the absence of affective evaluation.
To examine the hypothesis that facial expression processing in peripheral vision is driven by perceptual rather than affective factors, we used computational modeling of the visual saliency of expressive face regions (i.e., the eyes and the mouth). In addition, we foregrounded some ERP components involved in attentional capture (N1) and selective attention (N2pc), relative to those assumed to be sensitive to affective content (VPP, N170, N2, and EPN). The visual N1 is distributed over the entire scalp, but peaks between 100 and 150 ms after stimulus onset over fronto-central regions (Mangun & Hillyard, 1991). The N1 amplitude indexes sensory gain control or a gating mechanism that enhances stimulus perception (Luck, Woodman, & Vogel, 2000), and is tied to initial attentional capture (Foti, Hajcak, & Dien, 2009). It is thus well-suited to detect visual saliency of expressive facial features early. The N2pc is an enhanced negativity over posterior scalp sites (lateral extrastriate cortex and infero-temporal visual areas; Hopf et al., 2000), between 180 and 300 ms post-stimulus. It reflects spatial attention selection of a stimulus competing with others (Eimer & Kiss, 2010), and the subsequent direction of processing resources (Theeuwes, 2010). Importantly for the current paradigm, the N2pc appears contralateral to the spatial location of an attended stimulus. These characteristics make the N2pc a valuable tool for measuring the allocation of attention to lateralized stimuli. If ERP modulation and expression recognition are driven by perceptual factors, the faces with more salient features (i.e., happy faces), will enhance N1 and N2pc in correspondence with the time course of visual saliency, and those expressions will also show a recognition advantage.
Prior behavioral research using expression categorization tasks has consistently demonstrated an advantage in the explicit recognition of happy faces: Happy expressions are identified more accurately and faster than all the other basic emotions, both in central (see Nelson and Russell, 2013, Palermo and Coltheart, 2004) and peripheral (Calvo et al., 2014, Goren and Wilson, 2006) vision. What ERP components could be especially sensitive to happy faces and account for their explicit recognition or categorization advantage? Thus far, we have argued that the primacy effect for happy faces in peripheral vision could start at early processing stages (between 100 and 300 ms), through the perceptual influence of the visually salient smiling mouth. Within this timescale, we expect to observe an influence on the N1 and N2pc components as they reflect attention capture and spatial attention. Because the smile is a highly diagnostic feature – from which facial happiness can be inferred – (Calvo and Marrero, 2009, Calvo et al., 2014), the fact the smiling mouth is made accessible to attention by visual saliency would endow happy faces with such an early processing advantage. Nevertheless, explicit recognition or expression categorization typically take longer (∼1 s, in behavioral measures; e.g., Calvo and Lundqvist, 2008, Palermo and Coltheart, 2004), and involve semantic elaboration and response selection. Accordingly, later and additional neural components must also be involved beyond 300 ms post-stimulus onset.
Prior ERP research with faces in peripheral vision has provided little evidence regarding the neural processes leading to explicit recognition, or those underlying a potential recognition advantage of some (e.g., happy) expressions. In fact, generally, task-relevant instructions involving expression categorization and the corresponding behavioral measures have not been used. Only in Rigoulot et al. (2011) was expression categorization task-relevant (gender categorization was task-relevant in Rigoulot et al., 2012, and Wijers & Banis, 2012; detection of changes in the size of a central fixation cross was task-relevant in Stefanics et al., 2012). To make a contribution, we combined an explicit expression recognition task with continuous EEG recording in the same paradigm. A face was displayed peripherally for 150 ms, followed by a 650-ms blank interval, at the end of which participants categorized the expression (as happy, angry, etc.). This way we could assess three ERP components that are particularly relevant to expression categorization: The P3 and LPP (300–700 ms post-stimulus) are assumed to reflect semantic elaborative processes (Balconi and Mazza, 2009, Leppänen et al., 2007), and the slow positive waves (SPW) following the LPP peak are related to response selection and decision (Calvo and Beltrán, 2013, Debruille et al., 2011).
The SPW component is especially useful in paradigms requiring explicit categorization of expressions among multiple alternatives (Calvo and Beltrán, 2013, Debruille et al., 2011, García-Larrea and Cézanne-Bert, 1998), where response selection and decision are thus particularly demanding, as was the case in the current study. The SPW extends the P3/LPP complex, following the peak around 600–700 ms, at frontal, central, and parietal scalp sites (Folstein & Van Petten, 2011). SPWs are linked to categorization processes (Dien, Spencer, & Donchin, 2004), and reveal decisional difficulty and response selection (García-Larrea & Cézanne-Bert, 1998). In fact, Debruille et al. (2011) found that SPWs (700–1200-ms time window) were larger for the facial expressions that took more time to categorize, due to ambiguity. If there is an explicit recognition advantage of happy faces, they will elicit smaller SPW activity (i.e., reduced SPWs), immediately preceding the categorization response.
To sum up, in the current study, we recorded ERPs at successive stages during and following the presentation of happy, angry, sad, fearful, and neutral facial expressions in the visual periphery, in an explicit recognition task involving expression categorization. Computational modeling of the dynamics of visual saliency of expressive facial features was aimed at addressing the hypothesis that expression discrimination in peripheral vision is driven perceptually. The N1 and N2pc ERPs were assessed to obtain further support for this perceptual hypothesis. In an alternative, affective processing hypothesis, VPP, N170, N2, and EPN, served to examine whether emotional significance is also extracted. Finally, P3b/LPP and SPWs were used to explore the relationship of elaboration and response selection with expression categorization. We predicted that happy faces, due to their smiling mouth visual saliency, would enhance the N1 and N2pc activity, reflecting increased attentional capture and selection. Furthermore, as such a salient feature is highly diagnostic of facial happiness, it was expected to facilitate expression recognition because of reduced elaboration demands, as reflected in decreased P3b/LPP amplitudes; and, especially, it was expected to facilitate selection of a categorization response, as reflected in decreased SPWs. This approach thus combines modeling, neurophysiological, and behavioral measures. With it, we aim to uncover the neurocognitive mechanisms involved in the encoding of facial expressions in peripheral vision, and the recognition advantage of happy faces.
Section snippets
Participants
Twenty-four psychology undergraduates (16 female; all between 18 and 25 years of age) from the University of La Laguna gave informed consent and received either course credit or were paid (7 € per hour) for their participation. All were right-handed and reported normal or corrected-to-normal vision, and no neurological or neuropsychological disorder. Three additional subjects were excluded because of excessive eye-movements, and another one because of low accuracy in the expression
Behavioral data: categorization performance
For this and the following analyses, a Greenhouse–Geisser corrected ANOVA with expression (angry, sad, fearful, happy, and neutral) as a within-subject factor was performed on the dependent variables, followed by Bonferroni corrections for multiple comparisons, unless otherwise indicated. An expression effect emerged for response accuracy, F(4, 92) = 13.93, p < .0001, , and reaction times, F(4, 92) = 21.12, p < .0001, , with happy faces being recognized more accurately and faster than all
Discussion
We aimed to investigate whether emotional facial expressions can be recognized in peripheral vision, what is encoded of them and when, and why some expressions have a recognition advantage. Our findings revealed, first, that there was above-chance categorization accuracy for all the expressions (happy, sad, angry, fearful, and neutral faces), thus showing discrimination, although happy expressions were categorized more accurately and faster than the others. Second, the ERP components modulated
Acknowledgements
This research was supported by Grant PSI2009-07245 from the Spanish Ministerio de Ciencia e Innovación, and the Agencia Canaria de Investigación, Innovación y Sociedad de la Información (NEUROCOG project), and the European Regional Development Funds, and by CEI CANARIAS: Campus Atlántico Tricontinental (project supported by Spanish Ministerio de Educación).
References (65)
- et al.
Brain lateralization of holistic versus analytic processing of emotional facial expressions
Neuroimage
(2014) - et al.
Perceptual, categorical, and affective processing of ambiguous smiling facial expressions
Cognition
(2012) - et al.
When does the brain distinguish between genuine and ambiguous smiles? An ERP study
Brain and Cognition
(2013) - et al.
Time course of discrimination between emotional facial expressions: The role of visual saliency
Vision Research
(2011) - et al.
The top-down control of visual selection and how it is linked to the N2pc component
Acta Psychologica
(2010) - et al.
Early and late temporo-spatial effects of contextual interference during perception of facial affect
International Journal of Psychophysiology
(2009) - et al.
P3, positive slow wave and working memory load: A study on the functional correlates of slow wave activity
Electroencephalography and Clinical Neurophysiology
(1998) - et al.
Quantifying facial expression recognition across viewing conditions
Vision Research
(2006) - et al.
N170 response to facial expressions is modulated by the affective congruency between the emotional expression and preceding affective picture
Biological Psychology
(2013) - et al.
A saliency-based search mechanism for overt and covert shifts of visual attention
Vision Research
(2000)
Differences in facial expressions of four universal emotions
Psychiatry Research
Differential electrocortical responses to increasing intensities of fearful and happy emotional expressions
Brain Research
Event-related potential studies of attention
Trends in Cognitive Sciences
The many faces of configural processing
Trends in Cognitive Sciences
Modelling the role of salience in the allocation of overt visual attention
Vision Research
On the automaticity of emotion processing in words and faces: Event related brain potentials from a superficial task
Brain and Cognition
Implicit emotional processing in peripheral vision: Behavioral and neural evidence
Neuropsychologia
Emotions in word and face processing: Early and late cortical responses
Brain and Cognition
Electrocortical responses to NIMSTIM facial expressions of emotion
International Journal of Psychophysiology
Processing of unattended facial emotions: A visual mismatch negativity study
Neuroimage
Top-down and bottom-up control of visual selection
Acta Psychologica
The NimStim set of facial expressions: Judgments from untrained research participants
Psychiatry Research
Modelling attention to salient protoobjects
Neural Networks
Foveal and parafoveal spatial attention and its impact on the processing of facial expression: An ERP study
Clinical Neurophysiology
The when and where of perceiving signals of threat versus non-threat
Neuroimage
Switching associations between facial identity and emotional expression: A behavioural and ERP study
Neuroimage
Recognizing emotion from facial expressions: Psychological and neurological mechanisms
Behavioral and Cognitive Neuroscience Reviews
Consciousness and emotion: ERP modulation and attentive vs. pre-attentive elaboration of emotional facial expressions by backward masking
Motivation and Emotion
State-of-the-art in visual attention modeling
IEEE Transactions on Pattern Analysis and Machine Intelligence
Recognition advantage of happy faces: Tracing the neurocognitive processes
Neuropsychologia
Recognition of facial expressions in peripheral vs. central vision: Role of the eyes and the mouth
Psychological Research
Facial expressions of emotion (KDEF): Identification under different display-duration conditions
Behavior Research Methods
Cited by (35)
Impact of visual eccentricity on emotional reactivity: implications for anxious and depressive symptomatology
2022, Journal of Affective Disorders ReportsThe early processing of fearful and happy facial expressions is independent of task demands – Support from mass univariate analyses
2021, Brain ResearchCitation Excerpt :The null results seen in this re-analysis and the original study may be due to the nose fixation enforcement, leaving the smile in participants’ visual periphery (Itier & Neath-Tavares, 2017). On the contrary, however, when happy expressions are presented parafoveally, they can be accurately recognized (Calvo et al., 2010) and modulate ERPs (Calvo et al., 2014), suggesting direct fixation on the face or smile is not required for the processing of this emotion. Based on previous research and the current findings, it is unclear whether happy and neutral expressions reliably differ from each other, and how gaze fixation relates to these results.
The interactive effects of reward expectation and emotional interference on cognitive conflict control: An ERP study
2021, Physiology and BehaviorCitation Excerpt :However, this idea is in contrast with the typical assumption that negative (especially fearful) expressions have processing priority and consume more attentional resources than do positive expressions [43]. Advantages in the processing of happy faces have been reported in categorization tasks where a facial expression must be consciously and explicitly identified [59,69], likely because categorization is easier for happy expressions since “happy” is the only positive emotion among six basic emotions [59]. The current results, suggesting that the processing advantage for happy faces may exist in early processing stages, may indicate an advantage for expediting conflict resolution in this task context.