Attentional prioritisation and facilitation for similar stimuli in visual working memory

Visual working memory (VWM) is a capacity-limited system that allows for the short-term retention of visual information. VWM is limited in that it can store only three to four visual objects (Cowan, 2001), which can explain our frequent failure to detect changes in a complex visual scene (Rensink et al., 1997). Interestingly, the similarity between visual objects can either facilitate or hinder memory performance. Memory performance is better when objects belong to different categories (e.g., faces and houses) than when they all belong to the same category (e.g., all faces; Cohen et al., 2014; Yang & Mo, 2017), perhaps because items from same semantic categories compete more strongly with each other or are more confusable (Cohen et al., 2014; Franconeri et al., 2013; Wheeler & Treisman, 2002). However, when the memory items all belong to the same category, they can often be recalled better when they are more similar to each other (Jiang et al., 2016; Lin & Luck, 2009; Yang & Mo, 2017). In a change detection task, Lin and Luck (2009) found that performance was higher when memory displays contained 3 similar colours (e.g., different hues of green) than when the memory displays contained 3 dissimilar colours (e.g., green, blue, red). This similarity effect was also observed in other visual feature domains, including line orientation and length (Sims et al., 2012), unfamiliar characters (Mate & Baqués, 2009), complex faces (Jiang et al., 2016; Yang & Mo, 2017) and landscapes (Yang & Mo, 2017).

It seems plausible that similar items may enjoy an encoding advantage, as similarity could allow encoding of chunks, or relative colours and difference values (rather than only encoding the exact colour values or positions in feature space; Martin & Becker, 2021). Such an encoding advantage would facilitate detecting a change among similar colours (e.g., Jiang et al., 2016; Sims et al., 2012). However, Lin and Luck (2009) argued that the similarity effect is not just due to differences at the encoding stage, as the similarity benefit also emerged for the first presented item in a sequential presentation i.e., before it could be recognised as being part of a ‘similar’ item set (Jiang et al., 2016; Lin & Luck, 2009; Yang & Mo, 2017). Lin and Luck (2009) argued that the similarity effect must therefore reside at the maintenance stage, as an encoding of the first item in a sequence should remain unaffected by the similarity to subsequent items.

Another important limitation of previous studies is that the similarity effect was only studied in a very limited range of conditions. Studies have used either simultaneous or sequential presentations of all-similar or all-dissimilar items (e.g., Jiang et al., 2016). Lin and Luck (2009) used a display with a mixture of similar and dissimilar items. However, these were only presented sequentially and thus, without direct competition. Moreover, in the experiment, the probed memory was more likely to be in a similar set than the dissimilar item, which may have provided an incentive to observers to attend more to similar items. Johnson et al. (2009) combined similar and dissimilar items into a mixed display and found that there was a benefit at test for similar items over dissimilar (see also; Peterson et al., 2015; Peterson & Berryhill, 2013). However, these trials were not compared against all-similar or all-dissimilar displays. While similar items were still prioritised in some fashion, it is still unknown if the dissimilar item had some influence on the retention of similar items.

According to current bottom-up attentional selection theories, mixed displays should not show a similarity effect, but instead an advantage for dissimilar items (i.e., a reversed similarity effect or dissimilarity effect). This holds because the dissimilar item would be the most visually salient item in the display, and salient items (i.e., items with a high feature contrast) generate an automatic ‘attend-to-me’ signal (Sawaki & Luck, 2010) and thus are prioritized for attention (e.g., Theeuwes, 1992; Wolfe, 1994). As attention is linked to VWM (e.g., Kiyonaga & Egner, 2013; Olivers et al., 2006), prioritized attention to the salient dissimilar item should lead to a prioritisation and hence, interfere with the encoding of the similar items, attenuating the strength of the similarity effect. However, if no attenuation is observed than this would suggest that a dissimilar item did not receive attentional prioritisation.

Trials with responses longer than 5000 ms were excluded from analysis (1.0% of trials). Raw errors for each condition were transformed into d’ values to examine change detection sensitivity (Abdi, 2007). Raw accuracy data for trials are reported in the Appendix.

A 2 (Change Item: Similar, Dissimilar) × 2 (Display Type: Separate, Mixed) repeated measures ANOVA was conducted on the d′ values for the change detection task. Results revealed a main effect of change item, F(1, 22) = 1642.95, p < 0.001, \(\eta_{{\text{p}}}^{2}\) = 0.99, such that change detection was more sensitive for similar items than dissimilar items (see Fig. 2). The effect of display type was also significant F(1, 22) = 348.10, p < 0.001, \(\eta_{{\text{p}}}^{2}\) = 0.94 (reflecting higher sensitivity in the separate over the mixed displays), but importantly this was qualified by a significant interaction, F(1, 22) = 376.10, p < 0.001, \(\eta_{{\text{p}}}^{2}\) = 0.95. Contrary to expectations, the advantage for similar items (d′_similar − d′_dissimilar) was stronger in the mixed displays than the separate displays (i.e., compared to the difference between the all-similar and all-dissimilar displays), t(22) = 19.39, p < 0.001, BF₁₀ = 2.13 × 10¹². As shown in Fig. 2, the interaction was due to change detection performance decreasing in the mixed displays compared to the all-dissimilar displays, t(22) = 30.55, p < 0.001, BF₁₀ = 2.05 × 10¹⁶.

The results of Experiment 1 reproduced the previously observed visual similarity effect (e.g., Lin & Luck, 2009). Change detection sensitivity was higher for the all-similar display compared to the all-dissimilar displays. Importantly, this similarity effect also occurred in the mixed display. Even though the dissimilar item was both more salient and proportionately more likely to be tested than an individual similar item, change detection sensitivity was notably lower. Contrary to our expectations, the similarity benefit was not attenuated in the mixed displays, but instead substantially stronger when the similar and dissimilar items were presented simultaneously. This effect was mainly due to a decrease in memory sensitivity for when the dissimilar item changed.

Contrary to the bottom-up saliency view (e.g., Theeuwes, 1992; Wolfe, 1994), that the salient dissimilar item would be granted prioritised access to VWM, instead the similar items appeared to receive a beneficial allocation of VWM resources. This would suggest a top-down attentional prioritisation or a strategy for encoding similar items that is overriding bottom-up saliency signals to ensure an enhanced representation of similar items in VWM (e.g., Dube et al., 2017; Emrich et al., 2017).

However, it should be noted that this explanation is still speculative, as it is possible that the advantage for similar items arose at a later stage of the change detection task, as for instance, during the maintenance or recall phase of the task (e.g., Lin & Luck, 2009). As Experiment 1 did not use measures that can distinguish between different phases of the task (encoding, maintenance, recall, response execution), it is unknown whether the similarity effect is indeed due to preferential encoding of similar items, only that the supposed attentional prioritisation of salient items did not translate to an advantage for remembering dissimilar items. Instead, the opposite result was observed; a significant detriment for remembering dissimilar items, especially when they were competing with other similar items. Experiment 2 was designed to test the attentional prioritisation explanation, by examining eye movements during the encoding phase of the change detection task.

The behavioural memory accuracy results replicated those from Experiment 1. Change detection sensitivity was greater for similar items than dissimilar in both the separate and mixed displays, and the additional performance cost for dissimilar items in the mixed display was replicated. First eye movements were more likely to land on one of the similar items than the dissimilar ones in the mixed displays, and similar items were likely to receive more fixations overall. Thus, in line with our attentional prioritisation explanation and contrary to bottom-up saliency views, similar items attracted attention and the gaze more strongly than the dissimilar item and this prioritisation lasted for the entire encoding phase, not just the initial attentional allocation.

Lin and Luck (2009) speculated that the similarity effect may be due to beneficial processes for similar items in the memory maintenance phase. Here we show that the advantage for similar items can also be initiated pre-attentively, as participants seem to have a top-down bias to preferentially select and encode similar items (even with the first eye movement). This attentional bias towards similar items overpowered the bottom-up tendency to preferentially select the most salient item in the display and led to prioritised encoding of similar items in VWM. Conversely, this early attentional benefit in the encoding displays may have been influenced by a stimulus-driven mechanism that overruled salience, perceptual grouping. Similar stimuli can be automatically grouped and encoded, even contradicting top-down goals (Jiang et al., 2004; Shomstein et al., 2010). Potentially the similar items used were grouped and prioritised and then emitted a higher priority signal than the salient dissimilar item. Whether the similarity-encoding benefit is due to a stimulus or goal-driven mechanism is an open question.

We can currently only speculate about the exact source of this difficulty. However, it is possible that it is easier to encode items in relation to each other rather than by their exact feature value or position in the feature space (Martin & Becker, 2021). For instance, it is imaginable that each encoded colour in the feature space acts like an ‘anchor’ against which the other colours are compared during encoding and maintenance, whereby this comparison process is easier or only feasible for similar items. In addition, the test displays containing all items rather than, for example, a single probe, may make it easier to notice a change in a similar item, as this changes the relations and anchor distances between the items. The latter ‘probe’ effect is, however, unlikely to account for the entire similarity effect, as we found strong and reliable similarity effects already in the encoding stage, and similarity effects have also been reported with single-item probes at the memory test stage (Lin & Luck, 2009).

Both Jiang et al. (2016) and Lin and Luck (2009) speculated that similarity benefits may have been due to a focus on a single similarity dimension. By focussing on this single dimension (e.g., various hues of orange), memory resources can be concentrated in a more efficient way, increasing memory precision. The current results are in line with this proposal, and additionally show an attentional bias for similar colours. In the mixed displays, the similar items were prioritised, perhaps because they formed part of a single similarity dimension. This focus on a specific region of colour space (and/or the corresponding items) could lead to detriments for other regions. This in turn would lead to less precise memory representations for dissimilar items, accounting for the decrease in dissimilar memory accuracy in the mixed displays, compared to the all-dissimilar displays. Furthermore, less efficient processing of dissimilar items could account for the observed increase in dissimilar item dwell times, as it could require more time to encode stimuli from a different colour region. While this explanation is still tenuous, the results of our study show that the similarity effect is a robust and large effect that does not only survive presenting similar and dissimilar stimuli together in the same display but even becomes stronger in these mixed displays. Contrary to previous explanations of the effect (e.g., Lin & Luck, 2009), our results show a clear bias towards similar stimuli originating in the memory encoding phase. Yet, as described by Jiang et al. (2016), the similarity benefit is likely comprised of effects at multiple stages of processing, and further research is required to map out the effect across the different stages of processing and identify the processes and mechanisms responsible for the similarity effect.