Selective attention to stimulus representations in perception and memory: commonalities and differences

Attentional selection has been investigated in perception (selective attention; e.g., Driver, 2001; Moore & Zirnsak, 2017) and memory (selective retrieval; e.g., Buckner & Wheeler, 2001; Eichenbaum, 2017; Mecklinger, 2010). Moreover, there is evidence that one influences the other, such as disengaging the gaze to withdraw attention from the external world to facilitate its focus on internal memory representations (Glenberg, Schroeder, & Robertson, 1998), or that the novelty or familiarity of an item affects how long it captures attention (Parks & Hopfinger, 2008). However, it is unclear whether each domain has its own selection process or whether selection in both domains shares a common neurocognitive basis, only differing in the stimuli it acts upon (external sensory environment vs. internal memory space). It has been repeatedly observed that the deployment of attention to sensory input and to representations in long-term memory (LTM) is governed by similar neurocognitive processes (Cabeza, 2008; Cabeza et al., 2003). Specifically, lesions in the posterior parietal cortex, especially the inferior parietal lobe, are associated with attentional selection as well as LTM retrieval deficits (Berryhill, 2012; Finke, Myers, Bublak, & Sorg, 2013; Hower, Wixted, Berryhill, & Olson, 2014). The attention-to-memory (AtoM) theory attempts to explain this commonality by proposing the same brain regions to be involved in attentional selection in perception and in LTM, i.e., in external versus. internal representational space (Cabeza, Ciaramelli, Olson, & Moscovitch, 2008; Ciaramelli, Grady, & Moscovitch, 2008).

While numerous studies support the AtoM theory, there are also findings that challenge it. For example, in their review of the contribution of the posterior parietal cortex (PPC) to episodic memory, Sestieri, Shulman, and Corbetta (2017) argue against a complete anatomical and functional overlap between attention to memory and perception. Furthermore, research on the fronto-parietal network of attentional control suggests that different subcomponents of the network are involved in different attentional demands (Cole et al., 2013; Corbetta & Shulman, 2002; Wang et al., 2010; Zanto & Gazzaley, 2013). For example, Dixon et al. (2018) propose, based on an extensive meta-analysis, that at least two functionally and anatomically distinct sub-networks of the fronto-parietal network can be differentiated, one associated with more internally oriented attention, such as mentalizing and emotional processing, and the other one with more externally oriented attention, such as reading and sensory-motor tasks.

Note that virtually all of the attempts to compare selective attention in memory versus perception did this across different studies with different stimulus materials and tasks (Hutchinson, Uncapher, & Wagner 2009, 2014; Sestieri Shulman, & Corbetta, 2010). In fact, we know of only one study that tried to compare attention to perception and memory using the same stimulus material and a comparable task. Sestieri et al. (2010) employed short video clips for both a perceptual as well as a memory-search paradigm in functional magnetic resonance imaging (fMRI) study. Their results showed adjacent, but distinct regions to be activated by search processes in memory and perception. However, one needs to take into consideration that at least part of the discrepancies in activation patterns might have resulted from systematic differences in the applied experimental tasks, such as stimulus material and the demand on cognitive-control processes, rather than from differences in the underlying brain mechanisms.

In the present study, we developed two variants of a behavioral paradigm to study the attentional control processes involved in selective stimulus processing in perception versus memory within subjects. The respective processes were probed by introducing different levels of interference across trials. Typical paradigms to assess this kind of interference to trigger attentional control employ trials during which several stimuli are presented simultaneously. Usually, one stimulus out of this stimulus set is task-relevant (target) while the others are distractors that have to be ignored (e.g., Egner & Hirsch, 2005; Stadler & Hogan, 1996; Tipper & Driver, 1988). To evaluate what kind of control processes were applied to the target representation and the distractors, the following trial-to-trial effects are compared: (1) repetition of the target (and/or distractor/s), (2) former distractor becoming the target (and/or former target becoming a distractor), and (3) all stimuli change. The third case is thought to be a neutral control condition, in which neither beneficial nor detrimental effects from the preceding trial should be observed. Typical findings are a response time (RT) benefit for repetitions of the target (positive priming) and prolonged RTs for switches between targets and distractors (negative priming; e.g., Tipper & Driver, 1988), especially when a previous distractor becomes a target in the immediately following trial. The most common interpretation of these effects is to assume attentional selection of targets and inhibition of distractors. This two-process account is typically referred to as the inhibition account (Tipper, 2001).

In memory, cognitive control during selective retrieval is often studied by having participants learn several associations with the same cue (usually category-exemplar associations), and then having them selectively retrieve some (retrieval practice: Rp⁺) but not others (Rp⁻). Normally, there is also a neutral control condition in which no association has to be selectively retrieved after initial learning (N). The manipulation is list-wise, not trial-wise as in selective perception. This paradigm is called retrieval-practice paradigm and was established by Anderson, Bjork, and Bjork (1994). When comparing the three conditions at the end of the procedure (1. learning phase of all cues and their associated items, 2. retrieval practice of Rp⁺ items via cued recall, 3. final test of all items via cued recall), the typical finding¹ is the following order of retrieval success: Rp⁺  > N > Rp⁻. This pattern has long been interpreted as evidence for inhibition in memory and has been termed retrieval-induced forgetting (RIF) (Anderson et al. 1994; Ciranni & Shimamura, 1999). Evidence from paradigms testing memory for the retrieval competitors with alternative cues suggests that the representation of retrieval competitors itself had been weakened, which is taken as additional support for the inhibition account (Anderson & Spellman, 1995; Johnson & Anderson, 2004).

Based on these considerations of beneficial and detrimental effects across trials, we present a novel attentional-selection paradigm for the investigation of attention to percepts and memories based on positive and negative priming (e.g., Schrobsdorff et al., 2007; Stadler & Hogan, 1996) that carefully matched the demands on cognitive-control processes while keeping stimulus material and task requirements as comparable as possible. To this end, we conducted two experiments in which selection in perception and memory were matched to fulfill various criteria. In both experiments, we used line drawings of objects which could be grouped into categories on which selection had to be carried out. In the first experiment, the main focus was on parallelizing the timing of the processing steps as closely as possible, while sticking closely to classical experimental positive/negative priming and selective long-term memory retrieval designs. One problematic aspect was that while representations of the stimuli are processed online for perception, they first have to be activated in memory to perform any additional process on them. Hence, to ensure that only the selection was carried out during the critical display, in experiment 1, retrieval of all potentially relevant stimuli (targets and distractors) was carried out at the beginning of each trial, prior to the cue indicating the target (Fig. 1). In experiment 2, we focused more strongly on parallelizing the exact number and appearance of the stimuli for the perception and memory tasks (Fig. 5). Inter-trial effects were examined in both domains by systematically repeating or changing the underlying set of potentially response-relevant internal/external representations, as well as repetitions of the target and distractors switching their roles.

Based on the reported findings on positive and negative priming in visual attention tasks (Stadler & Hogan, 1996), we expected shorter RTs and/or reduced error rates (compared to a complete change of the target and distractor stimuli) when both the target and distractors were repeated as well as when the target was repeated while the distractor changed. In contrast, a repetition of the set of stimuli but with a swap of their roles (former target becomes distractor and vice versa) should be associated with increased RTs and/or error rates. In case of a strong correspondence between attentional selection in perception and memory, we would predict similar effects for the selective LTM retrieval task. In accordance with positive priming, we should find facilitative effects on RTs in the LTM task when the retrieval target was being repeated. In contrast to negative priming, which in the selective LTM retrieval task would correspond to detrimental effects of switches to a previously irrelevant association with the same cue as in trial i-1, we expected no negative consequences or even beneficial effects due to spreading activation. This prediction was based on our previous findings (Kizilirmak, Rösler, & Khader, 2014) and would be in line with the found differences between attention-related brain regions for perception versus LTM retrieval (e.g., Hutchinson et al., 2009).

Figure 1 illustrates the perceptual and memory selection tasks. As can be seen by comparing both panels, the task procedures were parallelized as far as possible: Both, the selective perception and selective LTM retrieval task (1) used the same line-drawings as stimuli, (2) always activated a set of potentially task-relevant representations (external or internal, respectively) first and then (3) cued the target, prompting selective attention (see Fig. 1). Step 2 was implemented to ensure that differences could not be attributed to the one additional step required in memory retrieval: the activation of the memory representations, that is, the uploading into a working-memory buffer for online processing.

The different conditions were based on different demands on selective attention from trial i−1 to i. For both tasks, we included a control condition, which should be free from effects of any control processes (enhancement of the target and/or inhibition of distractors) applied in the previous trial. To this end, the complete set of stimuli changed from trial i−1 to trial i. The other conditions varied systematically with respect to sets and/or target stimuli being repeated or changed, which are typical conditions under which positive and negative priming can be observed.

To keep the results comparable, we performed 2 × 2 ANOVAs for the perception and memory tasks, including factors Set (change, repetition) and Target (change, repetition). An overview of the results of experiment 1 for both perception and memory task performance is depicted in Fig. 4.

In the first experiment, our focus was on matching the experimental stimuli and selection demands as closely as possible, while preserving the inherent structure of the classical positive/negative priming and selective long-term memory retrieval experimental designs. One problematic aspect we tried to address was that while representations of the stimuli are processed online for perception, they first have to be activated in memory to perform any additional (selection) processes on them. Hence, to ensure that only the selection was carried out during the critical display, retrieval (or perception) of all potentially relevant stimuli (targets and distractors) was carried out at the beginning of each trial, prior to the cue indicating the target.

The result patterns found for the perceptual task, where the repetition of the target was the driving factor, are in sharp contrast to the memory task where we found a highly significant main effect of set repetition, but only a marginal effect of target repetition. In the memory task, error rates were lower when the set of potentially relevant representations was repeated, even when the former target became a retrieval competitor and vice versa, however, this was not the case for the perceptual task. In other words, for memory, it is the repetition of the set of representations while for perception, it is the repetition of the target that proved to be more beneficial. To conclude, switching the target within a pre-activated set is detrimental only for perceptual selection, but not for selection in memory, where the shifting of the set is more detrimental.

However, processing differences may also have occurred due to the differing number (one versus two) and nature of distractors (different uncolored objects vs. different colors of the same object). We had chosen only one distractor for the perception task, because it is the standard for positive and negative priming tasks, such as the one by Tipper and Driver (1988), and because more than one distractor may have made the display too crowded. We also had refrained from using different colors in the perception task, because it is well-established that color is a more prominent feature than form for selective attention in visual perception, even enabling parallel search (Alexander, Nahvi, & Zelinsky, 2019; Kopp, Tabeling, Moschner, & Wessel, 2007; Turatto & Galfano, 2000). Hence, selecting according to color seemed too easy and therefore not comparable to the memory task. Nonetheless, to rule out that these differences between tasks could have been the leading factors for the diverging result patterns between perception and memory, we conducted a second experiment.

In an attempt to parallelize the selective perception and memory tasks even further, we used exactly the same stimuli for targets and distractors, as well as the exact same task (see Fig. 5). The task in both domains now was to indicate the location of a gap in the indicated target stimulus, which either had to be focused on during perception or selectively retrieved from memory (Fig. 5). There were two distractors in both tasks and all distractors were of the same object as the target, but in different colors.

The second experiment focused on fully parallelizing the exact number and appearance of the stimuli for the perception and memory tasks, as well as the response, while systematically repeating or changing the underlying set of potentially response-relevant internal/external representations. We generally replicated the result pattern from Exp. 1 for selection in memory, which again showed that target repetition only had an effect when it occurred in conjunction with set repetition. In other words, there was no positive priming of a single feature (color) alone, only of the integrated object representation. Set repetitions decreased RTs and reduced error rates, but to a much higher degree when stimulus set and target color were repeated. Thus, selection in memory benefits from repetitions of the previously activated set of memory representations, and even more so from repetitions of set and retrieval target.

This finding is in contrast to the perception data. Here, targetcolor and stimulus set repetitions affected error rates without interaction. Surprisingly, repetitions of target color had a more beneficial effect of reduced error rates when the stimulus set changed (i.e., different line drawings; S_ch/T_rep), then when the set was repeated (i.e., same line drawings; S_rep/T_rep). This was unexpected, because the S_rep/T_rep condition can be considered full positive priming, because distractors and targets both were repeated and kept their roles, whereas S_ch/T_rep meant that only the target color was repeated, whereas the stimuli themselves were changed (partial positive priming). However, this finding can be reconciled with the literature on positive priming, when considering that the actual response to the target, i.e., the specific press on one of the 8 response buttons, was only repeated in 1/8 of the trials,⁴ because the location of the gap that needed to be found was always chosen randomly from 8 different options. Hence, even when all stimuli including the identities of the targets and distractors were repeated, the response still changed in most cases. According to the stimulus–response retrieval account of priming (Frings, Schneider, & Fox, 2015), which proposes that advantages of target repetitions (and distractor) are mainly due to repetitions of stimulus–response associations, our full positive priming condition S_rep/T_rep should not lead to pronounced priming at all, because the exact stimulus–response mapping is repeated in only 1/8 of the trials. In fact, it could rather lead to substantial retrieval interference with the previous stimulus–response mapping of the same stimulus. Now in the S_ch/T_rep condition, all stimuli are changed, not cuing any previous stimulus–response associations, so any detrimental effect due to interfering stimulus–response mappings should be absent, whereas, at the same time, there might still be benefit from focusing on the same color. However, since response repetitions have not been systematically varied here, possible effects of differences in the number of response repetitions have to be investigated in a follow-up study.

In the current study, we compared attentional selection in memory versus perception within participants, using the same stimulus material and a closely corresponding trial structure in both domains. Specifically, in two experiments, the set of stimuli on which attentional selection had to be performed was either presented visually or had to be activated in memory. Out of this set, the target was indicated, that is, selective attention had to be applied, and a decision had to be made. We were interested in the effects of applying cognitive-control processes (selective attention towards the target and possibly the inhibition of distractors) that can be measured on a trial-by-trial basis.

Across both experiments, a relatively coherent behavioral pattern emerged for perception and memory. For selection in perception, we found a consistent benefit for target repetition (for error rates in both experiments, for RTs in experiment 1), not just for a repetition of the whole target object, but also for a repetition of merely the target’s color, even with different objects. Importantly, this benefit cannot be attributed to response repetition, because the likelihood of a response repetition for any condition, even when target and distractors were repeated, was 50% in Exp. 1 and only 12.5% in Exp. 2. Hence, the benefit is most likely due to a benefit from refocusing on the same object and/or color. Although we did find a small disadvantage for the classical negative priming condition in terms of slightly increased error rates for set repetition when target and distractor swapped their roles (S_rep/T_ch) in Exp. 1, there was no indication of negative priming in Exp. 2. Hence, we found no clear evidence for distractor inhibition (Mayr & Buchner, 2007; Tipper, 2001).

In contrast, for selection in memory, set repetition seemed to be the driving force behind all beneficial effects, while there was no condition which showed weaker performance as compared with the control condition. In both experiments, we found behavioral benefits of repeating the set of relevant memory representations. In Exp. 1, error rates for both set repetitions were reduced, that is, even the condition S_rep/T_ch, in which a former distractor became the target (and vice versa), showed a reduction in error rates compared to the control condition, although not as much as the condition in which set and target were repeated (S_rep/T_rep). This finding is a replication of our previous study, in which we also varied fast attentional switching from trial to trial in a selective LTM retrieval task (Kizilirmak et al., 2014) and suggests that the benefit of having already retrieved and activated the LTM contents outweighed any potential detrimental effects of selectively focusing on a previous retrieval competitor. This benefit further seems to be independent from the instruction to retrieve only one or all associations with a cue in trial i−1, which we could show in another study in which we also manipulated the number of retrieval targets from trial i−1 to i (Kizilirmak, Rösler, Bien, & Khader 2015). Such a finding is highly surprising in terms of an inhibitory account of negative priming (for a discussion, see Frings et al., 2015; Schrobsdorff, Ihrke, Behrendt, Herrmann, & Hasselhorn, 2012; Tipper, 2001), because one would expect interference to be stronger the more recent the activation of distractors is (or retrieval competitors in this case). In RTs, we again found no detrimental effects of shifting selective attention to a previous distractor. Mean RT for the only condition that significantly differed from the others was the one in which both set and target were repeated (S_rep/T_rep). Because we took care to that the whole set of memory representations was pre-activated intentionally in Exp. 1, this result is probably due to the fact that only this condition had the additional advantage of selecting exactly the same representation again. In line with this interpretation, in Exp. 2, where the set was not pre-activated before the critical selection display, RTs showed a significant reduction for both set repetition conditions. Thus, the processing advantages of prior activations of associated representations and repeated selections of one and the same target representation could be dissociated. To summarize, we again only found beneficial effects of repetitions of the set and an additional advantage on top when set and target were repeated.

Coming back to the main question at hand, that is, whether similar control mechanisms are involved in selection in memory and perception, the diverging result patterns suggest that there is an overlap in terms of benefitting from re-attending to a target representation, but whereas in perception, just refocusing on the same target or target color is already associated with a general behavioral advantage, target repetitions are only of note for memory if it is the very same memory representation (not just a feature like color) which needs to be re-attended. In memory, any advantages seem to mainly depend on repetitions of the set of representations, even when it is the previous distractor that becomes the target.

We propose that there are two main factors playing an important role in selection in memory and perception which differently affect selection in the two domains. The main factor seems to be an additional process that comes on top of the selection, but only in memory: The initial activation or retrieval of the memory representations. Whereas stimulus representations in perception are processed online, while selection takes place in representations of stimuli that are being currently perceived, in memory, those representations are processed offline, and have to be first retrieved and second actively held in working memory (Exp. 1) while selection takes place or target and distractors are being activated (intentionally and automatically via spreading activation) during the process of selective retrieval. In the LTM task, attentional resources are already deployed to a substantial degree to search for and reactivate a set of potentially task-relevant LTM representations, and to keep them active in working memory (WM). There is evidence that the same attentional resources are involved in holding WM representations active (i.e., activated LTM contents; internal stimuli) and attending towards external stimuli (Chun, 2011; Kiyonaga, Dowd, & Egner, 2017; Kiyonaga & Egner, 2014). The resource-demanding initial search for the respective LTM representations requires cognitive resources that are not necessary, at least not as much, in the visual perception task, leading to the observed general benefit of repeating sets of representations independent of stimulus–response associations. This leads to a general advantage of repeating the same set of stimulus representations in the memory task, which became only evident in the error rates for Exp. 1, in which we attempted to separate the step of activating the set of potentially relevant representations before the critical selection interval, and also in the RTs in Exp. 2, where this process had to be carried out during the selection interval.

The second factor which is able to explain a main overlap is facilitation for re-focusing on the previously attended target, in the selective attention (in perception) literature referred to as positive priming (e.g., Stadler & Hogan, 1996). The condition in which the stimulus set and target, meaning both target and distractors, are repeated shows a general advantage in error rates as well as in RTs for both domains, memory and perception. However, whereas in memory, only identity priming was observed (same target representation, exp. 1 + 2) in the perception task, even feature priming (of color) in the absence of identity priming (different line drawing, same color) could be observed (perception task, exp. 2). One explanation would be that color is a more dominant feature for the direction of selective attention in perception than in memory, while in memory grouping, features according to semantic meaning (here: objects) are more relevant. The visual attention literature (Lamy & Egeth, 2003; Maunsell & Treue, 2006; Turatto & Galfano, 2000) as well as literature on associative memory networks seems to support this idea (Anderson, 1976; Collins & Loftus, 1975; Ghosh & Gilboa, 2014; Saxe, McClelland, & Ganguli, 2019).

While we took great care to match the stimuli and how to respond to them (i.e., the stimulus material, task, target-distractor space, and even the response options) across selection domains, recent work suggests that whatever is perceived and whichever action is being performed becomes bound into an episodic event file, which might be the basis for subsequent priming (theory of event-coding; Frings et al., 2020; Hommel, Müsseler, Aschersleben, & Prinz, 2001; Singh, Moeller, & Frings 2016). Accordingly, not only the stimuli and responses, but also the specific stimulus–response bindings should be matched across domains to make the selection task as comparable as possible. In the present study, the associated response with a specific stimulus (e.g. the red shell), that is, the set-target combination, was always the same for the memory task, while the response changed in 7/8 of all trials in the perception task. This should be held constant in future studies that compare priming in memory and perception.

All in all, we propose, based on two experiments, that any attempt of modeling attention to external versus internal stimulus representations in exactly the same way has to cope with issues of comparability that are inherent to the different cognitive processes underlying attentional selection in external versus internal sensory space. As outlined above, we see a substantial difference with respect to the process of activating a set of stimulus representations for attentional selection in the first place. Future studies could try to render this process easier in the memory task or more difficult in the perceptual task, e.g., by presenting stimuli within a certain amount of visual noise, making them harder to perceive and to discriminate, thus creating similar attentional demands for the initial representation as in the memory domain. Another factor is the larger benefit from re-attending the same target color in perception compared with memory. To match selective perception and memory tasks even further, one would need to carefully figure out target features of similar relevance for search in the sensory environment and in memory. An important aspect which needs to be additionally taken into account, and which seems often to be neglected in cognitive research, are response repetitions and changes potentially confounded with the conditions which manipulate the shifting of the focus of internal or external attention.

Keeping those challenges in mind, the present study delineated principal similarities and differences of selection processes in both domains: While positive priming from stimulus repetition was found in both selection domains, we found no consistent effects of negative priming when shifting the focus of attention to a previously to-be-ignored stimulus. However, priming in the perception task was mainly due to repetitions of the target feature (here: color), whereas for the memory task, repetitions of the same set of stimulus representations was most important. We propose that the differences can be attributed to a reduced cognitive effort when the now relevant memory representation had already been pre-activated (even as a distractor) in the previous trial. Additionally, our experiments both underscore the importance of taking stimulus–response associations into account, which may be a hidden factor behind differences between domains.