Visual search in scenes involves selective and nonselective pathways

doi:10.1016/j.tics.2010.12.001

Trends in Cognitive Sciences

Volume 15, Issue 2, February 2011, Pages 77-84

https://doi.org/10.1016/j.tics.2010.12.001 Get rights and content

How does one find objects in scenes? For decades, visual search models have been built on experiments in which observers search for targets, presented among distractor items, isolated and randomly arranged on blank backgrounds. Are these models relevant to search in continuous scenes? This article argues that the mechanisms that govern artificial, laboratory search tasks do play a role in visual search in scenes. However, scene-based information is used to guide search in ways that had no place in earlier models. Search in scenes might be best explained by a dual-path model: a ‘selective’ path in which candidate objects must be individually selected for recognition and a ‘nonselective’ path in which information can be extracted from global and/or statistical information.

Section snippets

Searching and experiencing a scene

It is an interesting aspect of visual experience that you can look for an object that is, literally, right in front of your eyes, yet not find it for an appreciable period of time. It is clear that you are seeing something at the location of the object before you find it. What is that something and how do you go about finding that desired object? These questions have occupied visual search researchers for decades. Whereas visual search papers have conventionally described search as an important

Classic guided search

One approach to search, developed from studies of simple stimuli randomly placed on blank backgrounds, can be called ‘classic guided search’ [1]. It has roots in Treisman's Feature Integration Theory [2]. As we briefly review below, it holds that search is necessary because object recognition processes are limited to one or, perhaps, a few objects at one time. The selection of candidate objects for subsequent recognition is guided by preattentively acquired information about a limited set of

The failure of classic guided search

To this point, we have described what could be called ‘classic guided search’ 1, 25. Now, suppose that we wanted to apply this classic guided search theory to the real world. Find the bread in Figure 3a. Guided search, and similar models, would say that the one to two dozen guiding attributes define a high-dimensional space in which objects would be quite sparsely represented. That is, ‘bread’ would be defined by some set of features [21]. If attention were guided to objects lying in the

A nonselective pathway to gist processing

Fortunately, there is another route to semantic scene information. Humans are able to categorize a scene as a forest without selecting individual trees for recognition [54]. A single, brief fixation on the kitchen of Figure 3a would be enough to get the ‘gist’ of that scene. ‘Gist’ is an imperfectly defined term but, in this context, it includes the basic-level category of the scene, an estimate of the distributions of basic attributes, such as color and texture [55], and the spatial layout 54,

Concluding remarks

What is next in the study of search in scenes? It is still not understood how scenes are divided up into searchable objects or proto-objects [76]. There is much work to be done to describe fully the capabilities of nonselective processing and even more to document its impact on selective processes. Finally, we would like to know if there is a neurophysiological reality to the two pathways proposed here. Suppose one ‘lesioned’ the hypothetical selective pathway. The result might be an agnosic

Acknowledgments

This work was supported by NIH EY017001 and ONR MURI N000141010278 to J.M.W. K.K.E. was supported by NIH/NEI 1F32EY019819-01, M.R.G. by NIH/NEI F32EY019815-01and M.L-H.V. by DFG 1683/1-1.

References (90)

A.M. Treisman et al.
A feature-integration theory of attention
Cognit. Psychol.
(1980)
A. Treisman
The binding problem
Curr. Opin. Neurobiol.
(1996)
J.M. Wolfe et al.
Preattentive object files: shapeless bundles of basic features
Vision Res.
(1997)
J.M. Wolfe
Moving towards solutions to some enduring controversies in visual search
Trends Cogn. Sci.
(2003)
P. Verghese
Visual search and attention: a signal detection theory approach
Neuron
(2001)
H. Kirchner et al.
Ultra-rapid object detection with saccadic eye movements: visual processing speed revisited
Vision Res.
(2006)
A. Fazl
View-invariant object category learning, recognition, and search: how spatial and object attention are coordinated using surface-based attentional shrouds
Cognit. Psychol.
(2009)
E. Kowler
The role of attention in the programming of saccades
Vision Res.
(1995)
D. Parkhurst
Modeling the role of salience in the allocation of overt visual attention
Vision Res.
(2002)
J.J. DiCarlo et al.
Untangling invariant object recognition
Trends Cogn. Sci.
(2007)

I. Biederman

Scene perception: detecting and judging objects undergoing relational violations

Cognit. Psychol.

(1982)

M.B. Neider et al.

Scene context guides eye movements during visual search

Vision Res.

(2006)

M.R. Greene et al.

Recognition of natural scenes from global properties: seeing the forest without representing the trees

Cognit. Psychol.

(2009)

T. Sanocki

Representation and perception of scenic layout

Cognit. Psychol.

(2003)

O.R. Joubert

Processing scene context: fast categorization and object interference

Vision Res.

(2007)

S.C. Chong et al.

Representation of statistical properties

Vision Res.

(2003)

C. Chubb

The three dimensions of human visual sensitivity to first-order contrast statistics

Vision Res.

(2007)

D.W. Williams et al.

Coherent global motion percepts from stochastic local motions

Vision Res.

(1984)

N. Demeyere

Automatic statistical processing of visual properties in simultanagnosia

Neuropsychologia

(2008)

D. Melcher et al.

Shapes, surfaces and saccades

Vision Res.

(1999)

J. Haberman et al.

Rapid extraction of mean emotion and gender from sets of faces

Curr. Biol.

(2007)

T.J. Buschman et al.

Serial, covert shifts of attention during visual search are reflected by the frontal eye fields and correlated with population oscillations

Neuron

(2009)

J.M. Wolfe

Guided search 2.0. A revised model of visual search

Psychon. Bull. Rev.

(1994)

G. Müller-Plath et al.

Space-based and object-based capacity limitations in visual search

Vis. Cogn.

(2007)

B.A. Dosher

Information-limited parallel processing in difficult heterogeneous covert visual search

J. Exp. Psychol. Hum. Percept. Perform.

(2010)

D.G. Pelli et al.

The uncrowded window of object recognition

Nat. Neurosci.

(2008)

B. Balas

A summary-statistic representation in peripheral vision explains visual crowding

J. Vis.

(2009)

M.P. Eckstein

The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing

Psychol. Sci.

(1998)

J.T. Townsend et al.

The serial–parallel dilemma: a case study in a linkage of theory and method

Psychon. Bull. Rev.

(2004)

T.S. Horowitz

Revisiting the variable memory model of visual search

Vis. Cogn.

(2006)

J. Theeuwes

A new estimation of the duration of attentional dwell time

Psychon. Bull. Rev.

(2004)

C.M. Moore et al.

Getting beyond the serial/parallel debate in visual search: a hybrid approach

T.L. Thornton et al.

Parallel and serial processes in visual search

Psychol. Rev.

(2007)

L.W. Renninger

Where to look next? Eye movements reduce local uncertainty

J. Vis.

(2007)

W.S. Geisler

Visual search: the role of peripheral information measured using gaze-contingent displays

J. Vis.

(2006)

G.J. Zelinsky

A theory of eye movements during target acquisition

Psychol. Rev.

(2008)

G.J. Zelinsky et al.

Eye movements during parallel-serial visual search

J. Exp. Psychol. Hum. Percept. Perform.

(1997)

L. Huang

What is the unit of visual attention? Object for selection, but Boolean map for access

J. Exp. Psychol. Gen.

(2010)

J.M. Wolfe

Guided Search 4.0: current Progress with a model of visual search

J.M. Wolfe et al.

What attributes guide the deployment of visual attention and how do they do it?

Nat. Rev. Neurosci.

(2004)

J.M. Wolfe et al.

Do intersections serve as basic features in visual search?

Perception

(2003)

J.M. Wolfe

Limitations on the parallel guidance of visual search: color × color and orientation × orientation conjunctions

J. Exp. Psychol. Hum. Percept. Perform.

(1990)

A. Treisman et al.

Feature analysis in early vision: evidence from search asymmetries

Psychol. Rev.

(1988)

L. Itti

A model of saliency-based visual attention for rapid scene analysis

IEEE Trans. Pattern Anal. Machine Intell.

(2002)

D.T. Lindsey

Color channels, not color appearance or color categories, guide visual search for desaturated color targets

Psychol. Sci.

(2010)

Cited by (369)

Individual differences in internal models explain idiosyncrasies in scene perception
2024, Cognition
According to predictive processing theories, vision is facilitated by predictions derived from our internal models of what the world should look like. However, the contents of these models and how they vary across people remains unclear. Here, we use drawing as a behavioral readout of the contents of the internal models in individual participants. Participants were first asked to draw typical versions of scene categories, as descriptors of their internal models. These drawings were converted into standardized 3d renders, which we used as stimuli in subsequent scene categorization experiments. Across two experiments, participants' scene categorization was more accurate for renders tailored to their own drawings compared to renders based on others' drawings or copies of scene photographs, suggesting that scene perception is determined by a match with idiosyncratic internal models. Using a deep neural network to computationally evaluate similarities between scene renders, we further demonstrate that graded similarity to the render based on participants' own typical drawings (and thus to their internal model) predicts categorization performance across a range of candidate scenes. Together, our results showcase the potential of a new method for understanding individual differences – starting from participants' personal expectations about the structure of real-world scenes.
Learned associations serve as target proxies during difficult but not easy visual search
2024, Cognition
The target template contains information in memory that is used to guide attention during visual search and is typically thought of as containing features of the actual target object. However, when targets are hard to find, it is advantageous to use other information in the visual environment that is predictive of the target's location to help guide attention. The purpose of these studies was to test if newly learned associations between face and scene category images lead observers to use scene information as a proxy for the face target. Our results showed that scene information was used as a proxy for the target to guide attention but only when the target face was difficult to discriminate from the distractor face; when the faces were easy to distinguish, attention was no longer guided by the scene unless the scene was presented earlier. The results suggest that attention is flexibly guided by both target features as well as features of objects that are predictive of the target location. The degree to which each contributes to guiding attention depends on the efficiency with which that information can be used to decode the location of the target in the current moment. The results contribute to the view that attentional guidance is highly flexible in its use of information to rapidly locate the target.
The influence of natural image statistics on upright orientation judgements
2024, Cognition
Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.
How information within the perceptual span guides visual search and aids perception
2023, Displays
We explore the guiding effect of task-related information appearing within the perceptual span on visual search based on elements such as the range of effective information attention within the perceptual span and the orientation of task-related objects. In the first experiment, we recorded reaction time and eye movement data of 48 subjects performing a visual search task for target information in the navigation interface and divided them into four experimental groups according to the perceptual span (within or outside the range) and orientation (left or right) of the task-related information. The experimental results showed that: (a) task-related information appearing within the perceptual span had a guiding effect on target search; and (b) the right side closer to the target in task search have better performance in guiding. In the second experiment, eye-movement experiments of the navigation interface based on the eye-movement processing theoretical model of linguistics showed that: (a) within the perceptual span, the shifting of attention causing eye movements (saccade) and brings gaze into the area of interest, and (b) the perceptual span moves with attention shifts. This study provides evidence for cognitive processing of information responses within the perceptual span and dynamic adjustment of perceptual span location and provides a reliable method for effective information presentation and visual layout in navigation interfaces, with a view to providing reference and learning for related studies.
Independent spatiotemporal effects of spatial attention and background clutter on human object location representations
2023, NeuroImage
Spatial attention helps us to efficiently localize objects in cluttered environments. However, the processing stage at which spatial attention modulates object location representations remains unclear. Here we investigated this question identifying processing stages in time and space in an EEG and fMRI experiment respectively. As both object location representations and attentional effects have been shown to depend on the background on which objects appear, we included object background as an experimental factor. During the experiments, human participants viewed images of objects appearing in different locations on blank or cluttered backgrounds while either performing a task on fixation or on the periphery to direct their covert spatial attention away or towards the objects. We used multivariate classification to assess object location information. Consistent across the EEG and fMRI experiment, we show that spatial attention modulated location representations during late processing stages (>150 ms, in middle and high ventral visual stream areas) independent of background condition. Our results clarify the processing stage at which attention modulates object location representations in the ventral visual stream and show that attentional modulation is a cognitive process separate from recurrent processes related to the processing of objects on cluttered backgrounds.
Good-enough attentional guidance
2023, Trends in Cognitive Sciences
Theories of attention posit that attentional guidance operates on information held in a target template within memory. The template is often thought to contain veridical target features, akin to a photograph, and to guide attention to objects that match the exact target features. However, recent evidence suggests that attentional guidance is highly flexible and often guided by non-veridical features, a subset of features, or only associated features. We integrate these findings and propose that attentional guidance maximizes search efficiency based on a 'good-enough' principle to rapidly localize candidate target objects. Candidates are then serially interrogated to make target-match decisions using more precise information. We suggest that good-enough guidance optimizes the speed–accuracy–effort trade-offs inherent in each stage of visual search.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

ReviewVisual search in scenes involves selective and nonselective pathways

Section snippets

Searching and experiencing a scene

Classic guided search

The failure of classic guided search

A nonselective pathway to gist processing

Concluding remarks

Acknowledgments

Cognit. Psychol.

Curr. Opin. Neurobiol.

Vision Res.

Trends Cogn. Sci.

Neuron

Vision Res.

Cognit. Psychol.

Vision Res.

Vision Res.

Trends Cogn. Sci.

Cognit. Psychol.

Vision Res.

Cognit. Psychol.

Cognit. Psychol.

Vision Res.

Vision Res.

Vision Res.

Vision Res.

Neuropsychologia

Vision Res.

Curr. Biol.

Neuron

Guided search 2.0. A revised model of visual search

Psychon. Bull. Rev.

Space-based and object-based capacity limitations in visual search

Vis. Cogn.

Information-limited parallel processing in difficult heterogeneous covert visual search

J. Exp. Psychol. Hum. Percept. Perform.

The uncrowded window of object recognition

Nat. Neurosci.

A summary-statistic representation in peripheral vision explains visual crowding

J. Vis.

The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing

Psychol. Sci.

The serial–parallel dilemma: a case study in a linkage of theory and method

Psychon. Bull. Rev.

Revisiting the variable memory model of visual search

Vis. Cogn.

A new estimation of the duration of attentional dwell time

Psychon. Bull. Rev.

Getting beyond the serial/parallel debate in visual search: a hybrid approach

Parallel and serial processes in visual search

Psychol. Rev.

Where to look next? Eye movements reduce local uncertainty

J. Vis.

Visual search: the role of peripheral information measured using gaze-contingent displays

J. Vis.

A theory of eye movements during target acquisition

Psychol. Rev.

Eye movements during parallel-serial visual search

J. Exp. Psychol. Hum. Percept. Perform.

What is the unit of visual attention? Object for selection, but Boolean map for access

J. Exp. Psychol. Gen.

Guided Search 4.0: current Progress with a model of visual search

What attributes guide the deployment of visual attention and how do they do it?

Nat. Rev. Neurosci.

Do intersections serve as basic features in visual search?

Perception

Limitations on the parallel guidance of visual search: color × color and orientation × orientation conjunctions

J. Exp. Psychol. Hum. Percept. Perform.

Feature analysis in early vision: evidence from search asymmetries

Psychol. Rev.

A model of saliency-based visual attention for rapid scene analysis

IEEE Trans. Pattern Anal. Machine Intell.

Color channels, not color appearance or color categories, guide visual search for desaturated color targets

Psychol. Sci.

Review
Visual search in scenes involves selective and nonselective pathways