Uncovering the neural magnitude and spatio-temporal dynamics of natural image categorization in a fast visual stream

Highlights

• Magnitude of a face-selective response is about 4 µV from frequency-domain analysis.

• Onset of the face-selective response is about 100 ms post-stimulus presentation.

• Duration of this high-level face-selective response is about 420 ms.

• Spatial and temporal response progression across four successive time-windows.

• Empirical evidence for the summation of multi-harmonic frequency responses.

Abstract

Perceptual categorization occurs rapidly under natural viewing conditions. Yet, the neural spatio-temporal dynamics of category-selective processes to single-glanced, natural (i.e., unsegmented) images in a rapidly changing presentation stream remain unknown. We presented human observers with natural images of objects at a fast periodic rate of 12.5 Hz, i.e., every 80 ms. Images of faces were inserted every 3, 5, 7, 9, or 11 stimuli, defining stimulus-onset-asynchronies (SOAs) between 240–880 ms, i.e., presentation frequencies (Fs) between 4.17–1.14 Hz. Robust face-selective responses were objectively identified and quantified at F and its harmonics (2F, 3F, etc.) for every condition in the electroencephalogram (EEG). The summed-harmonic face-selective response was significantly reduced by 25% at the lowest face SOA, i.e. 240 ms between two faces, but remained stable from 400 ms SOA onward. This high-level, right lateralized face-selective response emerged at about 100 ms post-stimulus onset and progressed spatially throughout four successive time-windows (i.e., P1-face, N1-face, P2-face, P3-face) from posterior to anterior occipito-temporal electrode sites. The total duration of a category-selective response to a briefly presented face stimulus in a rapid sequence of objects was estimated to be 420 ms. Uncovering the neural spatio-temporal dynamics of category-selectivity in a rapid stream of natural images goes well beyond previous evidence obtained from spatially and temporally isolated stimuli, opening an avenue for understanding human vision and its relationship to categorization behavior.

Introduction

Perceptual categorization, the process by which sensory events are differentiated and classified in subgroups, is critical in enabling human interaction with the world. Human faces, which carry ecologically important social information, constitute the most salient class of visual images for understanding perceptual categorization. Indeed, faces can be differentiated from other objects with astounding accuracy and speed (Hershler and Hochstein, 2005, Crouzet et al., 2010, Hershler et al., 2010, Crouzet and Thorpe, 2011, Scheirer et al., 2014). Furthermore, the perception of segmented images of faces is known to elicit a large, widely distributed and partly specific neural response in the human ventral occipito-temporal (VOT) cortex, with a right hemisphere advantage (Sergent et al., 1992, Allison et al., 1994, Allison et al., 1999, Puce et al., 1995, Kanwisher et al., 1997, Weiner and Grill-Spector, 2010, Rossion et al., 2012a, Zhen et al., 2015).

Scalp electroencephalography (EEG), or more rarely magnetoencephalography (MEG), defines the speed and temporal dynamics of face-selective responses in the millisecond range at a system-level of organization. Most significantly, an early response peaking at about 170 ms following stimulus onset (i.e., the N170/VPP complex) differs in amplitude in response to faces compared to other object categories (Jeffreys, 1989, Jeffreys and Tukmachi, 1992, Bötzel et al., 1995, Bentin et al., 1996, Eimer, 2000, Halgren et al., 2000, Rossion et al., 2000, Itier and Taylor, 2004, Rousselet et al., 2008, Ganis et al., 2012; for reviews, Rossion and Jacques, 2011, Rossion, 2014a). Contrary to earlier, potentially spurious differences between faces and objects, this N170 selectivity to faces is not accounted for by Fourier amplitude information, which carries global low-level statistical properties of images (Rossion and Caharel, 2011; see also Tanskanen et al., 2005, Rousselet et al., 2008).

However, crucially, despite natural viewing conditions providing us with continually changing streams of information in complex scenes, categorization of faces, and perceptual categorization in general, have almost exclusively been investigated at the behavioral and neural level with images presented in spatial and temporal isolation.

Spatial isolation refers to face and non-face object stimuli being segmented from their natural backgrounds. In the rare use of natural images (Itier and Taylor, 2004, Rousselet et al., 2004, Rousselet et al., 2007, Hershler and Hochstein, 2005, Hershler et al., 2010, Crouzet et al., 2010, Cauchoix et al., 2014), controlling for low-level statistical properties differing between faces and objects (e.g., Torralba and Oliva, 2003, VanRullen, 2006, Keil, 2008) is particularly challenging, and their contribution to behavioral and neural face-selective responses is difficult to exclude (Itier and Taylor, 2004, VanRullen, 2006, Rousselet et al., 2007, Cerf et al., 2008, Honey et al., 2008, Crouzet and Thorpe, 2011, Cauchoix et al., 2014; but see Hershler and Hochstein, 2006).

Temporal isolation of the stimuli of interest is the norm in behavioral and neural studies of perceptual categorization and refers to the stimuli being presented as unique events separated by long and often variable stimulus onset asynchronies (SOAs). Alternatively, a train of stimuli with brief SOAs is sometimes used in neuroimaging (i.e., a block design), but the responses to the individual stimuli are lumped into a global brain response. Moreover, in EEG/MEG studies, unmasked faces and nonface objects are typically presented for a long stimulus duration (e.g., Bötzel et al., 1995: 3–4 s; Rossion et al., 2000: 500 ms; Crouzet et al., 2010: 400 ms; Ganis et al., 2012: 800 ms; Carlson et al., 2013: 533 ms; Cauchoix et al., 2014: 300–600 ms; Cichy et al., 2014: 500 ms) and SOAs of 1–2 s at least. Thus, object/face categorization may appear to be a prolonged process (Cichy et al., 2014, Mur and Kriegeskorte, 2014) merely because of this long and uninterrupted stimulus duration: in reality, while a single glance suffices for categorization of faces (Crouzet et al., 2010), the duration of category-selective processes from this brief encounter with the stimulus, in context, remains completely unknown.

An alternative stimulus presentation mode has been offered by rapid serial visual presentation (RSVP), which has been used with natural stimuli presented in such rapid succession that they are backward- and forward-masked and may be visible for only a single glance; this technique has been employed to investigate the contributions of memory and attention processes to behavioral image recognition over time (Potter and Levy, 1969, Potter, 2012, Potter et al., 2014). However, to derive behavioral performance (i.e., detection) from RSVP, a limited number of stimuli are presented in each sequence (i.e., fewer than 20 in the previously cited studies), and the rapidity of within-category stimulus presentation (e.g., as fast as 13 ms per stimulus in Potter et al. (2014)) limits the availability of temporal information in response to a stimulus category at a neural system-level.

Here, we provide the first comprehensive report of the magnitude, onset, and duration, or more generally the temporal dynamics, of the differential neural response between natural images of faces and other object categories viewed at a single glance within a rapid visual presentation stream. The approach that we use is termed Fast Periodic Visual Stimulation (FPVS; Rossion, 2014b), in which stimuli are presented in a fast periodic stimulation stream (here, at 12.5 Hz, i.e., one stimulus every 80 ms) while EEG is recorded. Similarly to RSVP, natural and highly variable images are forward- and backward-masked and are visible only long enough to be seen in a single fixation. However, since a neural response to a selected image category (i.e., faces) is investigated here rather than an explicit behavioral response, we are able to present long stimulation sequences (2 min sequences, each containing about 1500 images, i.e., 200 s/12.5 Hz) and to periodically embed face images within the sequence at a lower rate, for instance every five items (i.e., 400 ms). Thus, we build on a recently introduced FPVS-EEG paradigm to measure high-level perceptual categorization in the human adult (Rossion et al., 2015, Jacques et al., 2016b; Jonas et al., 2016) and infant (de Heering and Rossion, 2015) brain.

Since face stimuli are presented periodically in this paradigm as a proportion of rapidly presented images from various non-face object categories, two distinct response types emerge in the EEG recording: 1) general visual responses synchronized with the base presentation rate of object stimuli (here 12.5 Hz) and 2) face-selective responses, representing differential responses to faces in contrast to non-face objects, present at the slower face stimulation rate (Fig. 1). In these conditions, note that a response to faces would not emerge were it identical to the response to non-face objects: thus, the response at the face stimulation rate inherently represents the differential response to faces, eliminating the need for post-hoc subtraction across conditions (Rossion et al., 2015, Jacques et al., 2016b; Jonas et al., 2016; see also Liu-Shuang et al. (2014)). Moreover, the response to faces in this paradigm reflects not only discrimination (since faces are contrasted to numerous object categories, about 250 variable object stimuli in total), but also generalization (i.e. invariance) across face exemplars (about 50 different face stimuli are used, varying in background, identity, expression, size, viewpoint and lighting conditions), and thus truly reflects face categorization (see Fig. 1).

Additionally, the embedded periodic presentation of a natural image category within the periodic base stimulation stream allows the contribution of low-level image features to the face-selective response to be restricted with minimal artificial stimulus standardization: putative amplitude spectrum differences that may vary across face and non-face images on average, but which do not vary consistently within the face stimulus set, are not present periodically and so are not captured at the face presentation rate. The variance within the face stimulus set is put in competition with the variability of a large number of natural stimuli in the non-face stimulus set: changes of local contrast, luminance and spatial frequency that occur at every stimulation cycle project to the 12.5 Hz base stimulus presentation rate. Finally, low-level visual cues which might vary systematically (i.e., periodically) at the slower face stimulation rate are reduced by the variability within the natural face stimulus set. Thus, electrophysiological activity at the face-stimulation rate reflects high-level face-selective responses that are absent when the amplitude spectrum is preserved, i.e., for periodically presented phase-scrambled face stimuli vs. phase-scrambled non-face object stimuli (Rossion et al., 2015, de Heering and Rossion, 2015).

An important advantage of a FPVS-EEG categorization paradigm is that it enables the review of the EEG data in both the frequency and time domains, each providing its unique advantages. The periodicity of the stimulus presentation can be exploited in the EEG frequency domain, which captures periodic responses exactly at the frequency (or frequencies) of stimulation.¹ Such periodic responses, typically referred to as “Steady-State Visual Evoked Potentials” (SSVEPs, Regan, 1966, 1989; Norcia et al., 2015), are known for their objective localization and extremely high signal-to-noise ratio (SNR) in the frequency domain, and will be utilized here for face-selective response quantification. Moreover, the spatio-temporal dynamics of the face-selective response may be observed in the time domain: given the relatively low stimulation frequencies of face stimuli afforded by their spaced placement within the relatively fast presentation stream, FPVS-EEG is able to provide a rich description of information flow in response to faces in the time domain (e.g., Dzhelyova and Rossion, 2014, Rossion et al., 2015, Jacques et al., 2016b).

In summary, the specific goals of the present study were to exploit the FPVS-EEG paradigm to determine for the first time the magnitude of comprehensive face-categorization responses in a rapid visual stream of non-face objects, as well as to define their exact onset, duration and spatio-temporal pattern. These goals were achieved by 1) modifying the base stimulus presentation rate (i.e., 12.5 Hz here vs. 5.88 Hz previously) to severely constrain stimulus duration and segregate in time and space (i.e., scalp topography) face-selective responses from this fast base rate response (Alonso-Prieto et al., 2013); 2) quantifying multi-harmonic face-selective responses at a group and individual level across five manipulations of temporal distance between face stimuli, i.e., face SOAs, in the rapid visual stimulation stream.; and 3) comparing a typically used sinusoidal contrast modulation stimulation mode to an abrupt (i.e., squarewave) stimulation (Experiment 2) in order to determine the exact onset, propagation and temporal dynamics of face-selective responses in a rapid stimulation stream.