Cueing effects on semantic and perceptual categorization: ERPs reveal differential effects of validity as a function of processing stage
Introduction
Cueing tasks have often been used to study how expectancy and the violation of expectancy affect a variety of behavioral and neural measures (Downing, 1988; Hawkins, Shafto, & Richardson, 1988; Henderson, 1991; Miniussi, Marzi, & Nobre, 2005; Pestilli & Carrasco, 2005; Posner, 1980; Posner, Synder, & Davidson, 1990). For example, preparatory top-down orienting of attention to either the spatial location or physical properties (i.e., color, shape, or motion) of a visual stimulus has been shown to accelerate detection and categorization of the stimulus when it is presented (Carrasco & McElress, 2001; Corbetta, Miezin, Dormeyer, Shulman, & Peterson, 1990; Mangun & Luck, 1998; Posner, 1980). These differences have been associated with changes in the magnitude of neural activity in brain regions that process information related to the selected location or attribute. Most event-related potential (ERPs) studies have found that activity in early perceptual processing regions (100–200 ms) is enhanced when the stimulus is expected compared to when it is unexpected (e.g., Mangun & Hillyard, 1991; Luck et al., 1994; Soldan, Mangels, & Cooper, 2006 for a similar effect in repetition priming). Yet, numerous studies find that greater congruency between a concept and its preceding semantic context decreases the amplitude of the N400 component, findings taken to indicate that semantic expectancy reduces neural demands on retrieval of stored conceptual knowledge (e.g., Kutas & Federmeier, 2000). Taken together, these results suggest that expectancy can have differential effects on neural activity depending on the stage of stimulus processing biased by the preparatory information and the type of stimulus categorization required by the task. Thus far, however, few EEG/ERP studies have used cueing paradigms to examine the effects of top-down preparatory attention on multiple stages of visual information processing, from early perceptual identification through to later semantic classification (Miniussi et al., 2005; Tallon-Baudry, Bertrand, Henaff, Isnard, & Fischer, 2005).
In a study addressing this issue, Tallon-Baudry et al. (2005) demonstrated that preparatory attention differentially modulates activity along successive stages of the ventral visual processing stream. Intracranial EEG was recorded while patients viewed two types of visual stimuli, one that that they were prompted to attend to and encode for a delayed match-to-sample task, and another that was irrelevant to the memory task, and hence, did not need to be attended. Attention had differential effects on gamma band activity (GBA) recorded from lateral occipital cortex (LO) and higher-level fusiform gyrus. During the anticipatory period there was an increase of GBA in LO for the attention relative to the no-attention task, but this was followed by a decrease in this area upon presentation of the stimulus. In contrast, although the fusiform gyrus exhibited no GBA effects of preparatory attention during the pre-stimulus period, attended stimuli elicited an increase in fusiform GBA upon stimulus presentation. The authors posit that preparatory attention leads to an increase of activity in early perceptual processing stages before the stimulus is presented, which results in more efficient processing through these earlier stages upon stimulus presentation (i.e., reduced activity) and increased activity at downstream processing stages. Because responses were not made to the unattended stimuli, however, their paradigm was not ideal for directly studying the relationship between the behavioral and neural effects of preparatory attention on stimulus identification and categorization (see also Kilner, Bott, & Posada, 2005).
In the present study, we used ERPs to investigate the effects of valid and invalid preparatory attention on target processing in two cued-categorization tasks, one requiring categorization primarily on the basis of perceptual information and the other requiring categorization primarily on the basis of semantic information. Specifically, ERPs were recorded while subjects made perceptual (right/left) judgments on centrally presented arrows (‘> > > >’), or semantic (living/non-living) judgments on concrete nouns (‘bunny’). Although both tasks required the initial detection of a visual stimulus, we hypothesized that subsequent categorization in the arrow task would rely primarily on processing within regions associated with object recognition, whereas, categorization in the word task would necessarily engage semantic regions downstream of both object and lexical recognition. Thus, to the extent that categorization processes associated with these tasks would be expected to engage correspondingly distinct object and semantic processing regions in the brain, it may be possible to explore the effects of expectancy on target-specific processing activity even with the lower spatial resolution of ERP. We note that in contrast to Tallon-Baudry et al. (2005), all stimuli in the present task required a response, regardless of cue validity. In addition to providing behavioral data, we can also be confident that any systematic differences of neural modulation between conditions result from cueing specific processes rather than from general differences in attentional allocation.
Expectancy was manipulated using a standard cueing procedure in which we varied the validity of the cues. For the majority of trials (60 percent), arrow or word judgments were preceded by valid cues (‘< >’ or ‘W’) that accurately predicted the category (arrow or word) of the subsequent target and associated task set. On 10 percent of trials, however, stimuli were preceded by an invalid cue. Invalid cues result in preparatory attention being directed toward the inappropriate task-set (e.g., cue: ‘W’; target: arrows). Finally, to provide a baseline against which to directly compare the relative costs and benefits of valid and invalid cueing, 30 percent of trials were preceded by a neutral cue that provided no information about the upcoming target. We expect that valid cues would increase the speed with which the target stimulus is processed compared to those following neutral cues because they provide for optimal preparation of target-specific processing prior to stimulus onset. Inversely, invalid cues should result in longer RTs than either valid or neutral cues because they draw attention away from the appropriate target-specific processes. In addition, to redirect attention to the appropriate processes when the unexpected target arrives, subjects must disengage attention from the incorrect task-set and reorient attention to the appropriate task-set. Thus, we predict that the behavioral costs of invalid cues should be reflected not only in regions associated with task-specific processing of perceptual or semantic targets, but also in regions involved in processing conflict and task-set switching. Unlike the stimulus-specific modulation of target identification and categorization processes, however, task-set switching may involve more domain-general top-down attentional modulation (i.e., evidenced equally by both tasks).
We emphasize that unlike traditional cued spatial selective attention tasks, cues in the present task provide no explicit information about the correct response, only the physical nature of the upcoming stimulus, which itself signals only the type of task to be performed (i.e., word = semantic task; arrow = perceptual task). Subjects learn that a cue is “invalid” only after they perceptually identify the (unexpected) stimulus. They must then use this information to reorient attention to a new task set. Thus, this design represents a novel hybrid between the spatial selective attention designs commonly used to assess the effects of cued attention, and task-switching designs, in which there are generally no “invalid” conditions (for review see Monsell, 2003). To our knowledge the only study that bears some similarity to this design is that of Miniussi et al. (2005), who used ERPs to compare the effects of expectancy on a perceptual angle-orientation task and a lexical decision task. As we will detail below, results of that study provide a basis for predictions about what we might find in the present study. However, Miniussi et al. (2005) did not include neutral cues for comparison, making it more difficult to link neural modulation to behavioral costs and benefits. Addition of the neutral cue, as well as other design differences between our study and that of Miniussi et al. (2005), may also allow us to better isolate the effects of validity on target-specific activity from more domain-general task-switching processes. Unlike that study, however, we will not examine the effects of task-set preparation on cue period activity. Rather, by focusing our analyses to the target period activity, we will be able to explore not only the benefits of top-down expectation on processing of valid targets, but also the costs of task-switching initiated by identification of an unexpected target.
Event related potentials (ERPs) have been widely used in studying the time-course of the effects of spatial cueing on sites of early visual sensory and perceptual processes. Visual ERP components over posterior electrodes are typically characterized by an early positivity (P1) and subsequent negativity (N1). Studies of preparatory attention using spatial cueing paradigms generally find an enhancement in P1 amplitude for validly cued stimuli relative to invalidly cued stimuli in stimulus detection tasks, with the additional enhancement of the N1 when a perceptual choice is required (Mangun & Hillyard, 1991; Vogel & Luck, 2000). Although the stimuli in the present study were centrally presented, these components have also demonstrated sensitivity to object-based selective attention (Valdes-Sosa, Bobes, Rodriguez, & Pinilla, 1998), as well as manipulation of central attentional resources to foveal stimuli (Mangels, Picton, & Craik, 2001). Thus, they may also be sensitive to the manipulation of expectancy in the present study. Nonetheless, Miniussi et al. (2005) did not observe cueing effects in the P1 or N1. Instead, they found effects of cueing only at a later posterior component (240–280 ms) that was larger in the invalid compared to the valid condition of the angle-orientation task, the functional correlates of which were unclear. Thus, we will re-examine the effects of cueing at the early perceptual P1 and N1, along with any later posterior components that appear to be modulated by the validity manipulation.
We predict that the word categorization task will also engage multiple processing stages, beginning with early perceptual processing of the word form and continuing onward to lexical and semantic processes necessary to make the living/nonliving judgment. Thus, we will examine the effects of cueing on language-related ERP components previously associated with lexical and semantic processing. These include a left anterior-temporal negativity peaking at about 320 ms (N3/N320/early ITN: Butterfield & Mangels, 2003; Mangels et al., 2001; Nobre & McCarthy, 1994), and a later N400 often observed over central scalp electrode sites (Kutas & Hillyard, 1980; Nobre, Allison, & Mccarthy, 1994; Nobre & McCarthy, 1994; see Van Petten & Luka, 2006 for review). Indeed, Miniussi et al. (2005) examined the effects of cue validity on a lexical decision task and found that valid cues enhanced the N3 and N400, which were both maximal over midline sites. They interpreted these modulations as evidence that valid cueing was successful in enhancing attention to language-related processing.
The finding that both the N3 and N400 were similarly modulated by expectation is surprising in light of other research suggesting that these components may index functionally distinct processes that could be differentially affected by cue validity. Specifically, Nobre and McCarthy (1994) found that semantic priming enhanced an early anterior-temporal negative peak at around 316 ms (N3), but attenuated the N400, consistent with studies indicating that the N400 indexes effort necessary to integrate the lexical-semantic content of an item with the surrounding context (Kutas & Hillyard, 1980; Nobre et al., 1994; Rossell, Price, & Nobre, 2003). Priming can be viewed as a type of lexical-semantic preparation, however, the cues in both the present study and in Miniussi et al. (2005) only specify that one should prepare the language system to process semantic information, hence facilitating retrieval of a wide range of semantic information. While semantically primed words result in attenuated N400s, greater N400s are typically found for words that have a less constrained set of associations, such as words shown in isolation, low frequency words, concrete (vs. abstract) words, and words that can easily be formed into other words by changing a letter (Van Petten & Luka, 2006). Such a difference between semantic priming and non-specific expectation of semantic information may explain why Miniussi et al. found that words elicited a larger N400 when they were expected (valid cues) than when they were unexpected (invalid cues).
In an attempt to replicate and extend the findings of Miniussi et al. (2005), we will examine the effects of cueing semantic categorization judgments on components at latencies corresponding to the N3 and N400. However, in addition to pursuing these effects at traditional midline sites, we will also analyze any observed modulations of temporal electrode sites, which are more proximal to the ventral stream regions putatively involved in the successive stages of word identification and semantic processing (Nobre et al., 1994; see also Van Petten & Luka, 2006 for review). When an average reference is used, as in the present study, modulations associated with various aspects of conceptual processing are often observed at these sites (Butterfield & Mangels, 2003; Mangels et al., 2001; Nessler, Mecklinger, & Penney, 2005; Stern & Mangels, 2006). Although historically, use of an average reference in studying these effects has been less common that use of a mastoid reference, studies specifically comparing the N400 recorded with an average versus a mastoid reference have found considerable advantages to the average reference (Curran, Tucker, Kutas, & Posner, 1993; Johnson & Hamm, 2000). With an average reference, the distribution of the N400 shifts from a negativity over centro-parietal sites to a pronounced positivity over temporal electrodes (Curran et al., 1993; Johnson & Hamm, 2000). A temporal distribution is more consistent with intracranial recordings (McCarthy, Nobre, Bentin, & Spencer, 1995) and fMRI studies (Rossell et al., 2003; Van Petten & Luka, 2006) that locate one of the principle N400 generators to this lobe. Thus, the use of an average-reference may provide for a more accurate representation of the anatomical topography of semantic scalp-recorded ERPs.
Finally, relevant to our second aim to locate domain-general components of top-down control, we will focus our analyses on the P3a component of the P300. This scalp potential has been associated with novelty of a variety of stimulus types, including words, pictures, tones, and tactile stimuli (Friedman, Cycowics, & Gaeta, 2001). However, within the context of cueing tasks, a recent study found that it was also enhanced when subjects were signaled to switch to a new task set, even when the signal cue was presented well before the target (Barcelo, Perianez, & Knight, 2002; Kopp, Tabeling, Moschner, & Wessel, 2006; Rushworth, Passingham, & Nobre, 2002). This suggests that the P3a may be involved in attentional orienting processes associated with the initiation of task-set switching, and hence, should be greater for invalidly cued stimuli. Miniussi et al. (2005) observed such a pattern on a P300 component extending across both frontal and parietal electrodes, but only during their angle-judgment task. It is unclear whether the absence of such an effect in the lexical decision task reflects a fundamental difference in cueing effects, or whether the large fronto-central N400 in this task may have obscured similar effects of validity on the overlapping P3a. Indeed, a desirable “side-effect” of the average reference's redistribution of the N400 to temporal sites is that there is less chance of overlap between the language-related modulations and midline P300 components. Arguably, this should enhance our ability to directly compare midline P300 activity across the semantic and non-semantic tasks in the search for domain-general processes.
Section snippets
Participants
Nineteen healthy right-handed, native English speakers (10 females; mean age = 23.9, SD = 3.8) were recruited from the Columbia University campus. All subjects gave informed consent and the study was approved by the Columbia University Institutional Review Board. Subjects were paid $10/hr for their participation.
Materials
Word stimuli were medium-frequency concrete nouns (mean SFI1
Costs and benefits of cueing on behavior
Percentage of errors was computed for each condition (valid, invalid, and neutral) and for each type of stimulus (word and arrow). Subjects made more errors overall on the word task than the arrow task (word = 12.2 percent, arrow = 1.2 percent; F(1, 17) = 29.9, p < .001), but there were no differences in error rate as a function of cue validity.
Mean reaction time (RT) for correct responses was computed for each condition (valid, invalid, neutral) and type of stimulus3
Discussion
The present study integrated aspects of cued attention and task-switching paradigms in order to evaluate the effects of cueing on processes including and extending beyond stimulus detection and identification. Using tasks that required subjects to make higher-level categorical decisions regarding either object form (arrow task) or word meaning (word task), we found strong similarities in the overall pattern of cueing effects on ERP waveforms across tasks, even though the specific components
Acknowledgements
This research was supported by an NIH grant R21MH066129 to J.A. Mangels. We are grateful to Matt Greene who provided assistance with programming, and to Anna Nobre, Bernadette Sibuma, and two anonymous reviewers for comments on previous drafts of this manuscript. Aspects of this data were presented at the 2005 meeting of the Cognitive Neuroscience Society (CNS), New York, NY.
References (56)
- et al.
Neural correlates of error detection and correction in a semantic retrieval task
Cognitive Brain Research
(2003) - et al.
Topography of the N400: Brain electrical activity reflecting semantic expectancy
Electroencephalography and clinical Neurophysiology
(1993) - et al.
Visual perceptual learning in human object recognition areas: a repitition priming study using high-density electrical mapping
Neuroimage
(2001) - et al.
The novelty P3: An event-related brain potential (ERP) sign of the brain's evaluation of novelty
Neuroscience and Biobehavioral Reviews
(2001) The neural basis of object perception
Current Opinion in Neurobiology
(2003)- et al.
The time course of visual word recognition as revealed by linear regression analysis of ERP data
NeuroImage
(2006) - et al.
Luminance and spatial attention effects on early visual processing
Cognitive Brain Research
(1995) - et al.
High-density mapping in an N400 paradigm: Evidence for bilateral temporal lobe generators
Clinical Neurophysiology
(2000) - et al.
Electrophysiology reveals semantic memory use in language comprehension
Trends in Cognitive Sciences
(2000) - et al.
Attention and successful episodic encoding: An event-related potential study
Cognitive Brain Research
(2001)
Sustained visual-spatial attention produces costs and benefits in response time and evoked neural activity
Neuropsychologia
Linking semantic priming effect in functional MRI and event-related potentials
NeuroImage
Task switching
TRENDS in Cognitive Science
Perceptual fluency, semantic familiarity and recognition-related familiarity: An electrophysiological exploration
Cognitive Brain Research
Attention enhances contrast sensitivity at cued and impairs it at uncued locations
Vision Research
The anatomy and time course of semantic priming investigated by fMRI and ERPs
Neuropsychologia
Spatiotemporal dynamics of human object recognition processing: An integrated high-density electrical mapping and functional imaging study of “closure” processes
NeuroImage
Neural localization of semantic context effects in electromagnetic and hemodynamic studies
Brain and Language
Think differently: A brain orienting response to task novelty
NeuroReport
Who comes first? The role of the prefrontal and parietal cortex in cognitive control
Journal of Cognitive Neuroscience
Covert attention accelerates the rate of visual information processing
PNAS
Attentional modulation of neural processing of shape, color, and velocity in humans
Science
Neural evidence for dissociable components of task-switching
Cerebral Cortex
Expectancy and visual-spatial attention: Effects on perceptual quality
Journal of Experimental Psychology: Human perception and performance
Neuroimaging studies of word reading
PNAS
Top-down enhancement and suppression of the magnitude and speed of neural activity
Journal of Cognitive Neuroscience
A sequence of object-processing stages revealed by fMRI in the human occipital lobe
Human Brain Mapping
Effects of target luminance and cue validity on the latency of visual detection
Perception and Psychophysics
Cited by (19)
Neural bases of unconscious orienting of attention in hemianopic patients: Hemispheric differences
2020, CortexCitation Excerpt :This might explain the perceptual deficits found in the intact visual field of hemianopic patients described as “sightblind” by Bola, Gall, and Sabel (2013; see also (Chokron, Perez, & Peyrin, 2016; Chokron, Peyrin, & Perez, 2018). As to the unusual effect on N1 (greater for the invalid condition), it could reflect an enhanced attention to a salient visual input appearing in an unexpected location as has been reported in some studies (Fu, Caggiano, Greenwood, & Parasuraman, 2005, 2001; Lai & Mangels, 2007). Blind field: a paradoxical effect was found in P1 with a larger amplitude for the invalid condition while N1 showed the canonical pattern.
Distinct brain activation in response to negative feedback at different stages in a variant of the Wisconsin Card Sorting Test
2020, Biological PsychologyCitation Excerpt :The purpose of the present study is to investigate the electrophysiological distinction, particularly the difference of P3 amplitude, between switch-NF and learn-NF in a modified WCST, in which participants were instructed to match sorting cards based on the shared shapes, colors, or patterns (Du et al., 2018; Lange, Seer, Mueller et al., 2015). Other previous relevant studies showed that, when compared with the repeat cue (or feedback), switch-NF evoked an enhanced P3 amplitude, an enhancement interpreted as the neural signature of a cognitive control mechanism required for task-set reconfiguration during task switching (Kieffaber & Hetrick, 2005; Kopp, Tabeling, Moschner, & Wessel, 2006; Lai & Mangels, 2007; Nicholson, Karayanidis, Poboka, Heathcote, & Michie, 2005; Slagter, Kok, Mol, Talsma, & Kenemans, 2005). Therefore, we expect to replicate the results obtained in previous relevant studies (Barceló et al., 2002; Barceló, 2003; Cunillera et al., 2012; Furumoto, 1991; Stuss & Picton, 1978; Watson et al., 2006), i.e., the switch-NF is expected to evoke a larger P3-like component than the repeat-positive feedback (repeat-PF).
Neurophysiological correlates of cognitive flexibility and feedback processing in violent juvenile offenders
2015, Brain ResearchCitation Excerpt :This ERP comprises of an increased positivity that peaks at approximately 300–500 ms after the cue signaling to switch to another rule, and was not observed after cues signaling subjects to repeat the same sorting rule. The switch-related P3 component has been interpreted as the neural signature of a cognitive control mechanism requested for task-set reconfiguration during task switching (Rushworth et al., 2002; Brass et al., 2005; Kieffaber and Hetrick, 2005; Nicholson et al., 2005; Slagter et al., 2005; Kopp et al., 2006; Swainson et al., 2006; Lai and Mangels, 2007). On the other hand, a similar P3 component appears, but in that case, after the presentation of the first positive feedback in the trial series (Cunillera et al., 2012).
Signal enhancement and suppression during visual-spatial selective attention
2010, Brain ResearchCitation Excerpt :When the location of the target was correctly pre-cued, the P1 contralateral to the attended location was relatively enhanced. Effects of selective attention have also been seen at longer latencies for attention to other physical stimulus features, such as color (e.g., Aine and Harter, 1986; Lange et al., 1998; Müller and Hillyard, 2000) and has been shown to impact later processing of higher order features (e.g., Lai and Mangels, 2007; Spencer et al., 2001). It has been hypothesized that signal enhancement works via a gain control mechanism, increasing sensitivity of neurons to properties of the attended stimulus (Mangun and Hillyard, 1987; Handy and Khoe, 2005; Hillyard et al., 1998; Navalpakkam and Itti, 2005).
Brain activities immediately after finding rare targets
2008, Neuroscience Letters