A picture says more than a thousand words: Behavioural and ERP evidence for attentional enhancements due to action affordances

Abstract

Previous research has demonstrated that, in addition to ventral stream processing of object form, manipulable objects are represented functionally in the dorsal stream. Here, we demonstrate how the two streams interact via attentional selection and consolidation such that objects whose form fits the function of a previously seen object, such as a tool, benefit from attentional enhancements due to the action affordance of the tool. Using the attentional blink (AB) paradigm and event-related potentials (ERPs), we tested whether providing an action relationship between two objects appearing closely together in time counteracts the typical decrement observed for processing of the second item. We used images (experiments 1a and 2) and names (experiment 1b) of common tools, objects that can be acted upon by those tools, and unrelated objects. We found that pictorial presentation of a tool and its action counterpart results in a diminished attentional blink as well as enhanced attentional selection seen as a larger P3, relative to tools and unrelated objects, and that this attentional enhancement is not driven by semantic associations. This means that the action affordance instantiated by the perception of a tool will reduce the functional blindness normally observed when two targets are presented too closely in time, specifically when the physical properties of the tool that elicit an action affordance are perceived.

Introduction

Action is one of the critical functions of the brain; after all, we need to act in order to survive. Goodale and Milner (1992), Milner and Goodale (1995) suggest that the dorsal stream of the visual pathway has developed to assist in the visual guidance of our actions in the world while the ventral stream supports object recognition. The neuroanatomical and functional separation between the dorsal action stream and the ventral object processing stream has been well-documented; however, it is evident that the two streams need to interact. Objects that afford action appear to be particularly useful to probe this interaction. We argue that tools command special processing because they provide an interface for actively engaging in our environment, which entails perceiving and manipulating other objects in a manner compatible with available tools at any given moment.

According to Gibson (1979), all perceivable surfaces and objects afford certain properties or uses to the observer, and tools constitute a special class because they afford action on other objects. For example, Tucker and Ellis, 1998, Tucker and Ellis, 2004 and Ellis and Tucker (2000) found that visual objects automatically prime the actions they afford, even when exposure is short or masked. Such associations between tools and action can facilitate the allocation of attention. For example, the potential of an object to readily engage in an action leads to enhanced attentional orientation (Handy et al., 2005; Handy, Grafton, Shroff, Ketay, & Gazzaniga, 2003), even in the absence of an explicit intention to perform any action (Ellis & Tucker, 2000; Tucker & Ellis, 1998). These action affordances persist over time (Tucker & Ellis, 2004) and operate across visual space (Humphreys & Riddoch, 2001). Taken together, such behavioural research has demonstrated that tools do indeed command special processing.

Interestingly, the intimate relation between tools, action and attention is also reflected in neuronal processing. For example, a functional MRI study revealed that activity in left dorsal premotor, inferior frontal, and parietal cortices was modulated by how well a prescribed motor movement matched the action affordance of a given object (Grèzes, Tucker, Armony, Ellis, & Passingham, 2003). Similarly, the findings of Creem-Regehr and Lee (2005) suggest that the extent of activity in the activated frontoparietal-temporal network and extrastriate areas that were responsive to graspable objects was modulated by the functional utility of those objects (i.e., by their action affordances). That is, objects with a known use elicited the greatest levels of activity in the network representing functional objects, or tools. Thus, the accumulation of research regarding the brain's responses to perception of graspable objects indicates that functional representations are automatically activated in posterior parietal dorsal areas while object form is identified in ventral, extrastriate areas.

Furthermore, Rushworth, Krams, and Passingham (2001) have localized a left hemisphere network involved in motor attention. Using PET, they found greater levels of activity in left precentral sulcus and in posterior parietal cortices for attentional allocation to an impending hand movement versus an impending vocal response. Interestingly, this network lies parallel to areas that are selectively responsive to tools; parts of the left premotor cortex were identified both in the Rushworth et al. (2001) study and in the findings of Chao and Martin (2000), and neighbouring areas in the left parietal cortex were active in studies of motor attention and of perception of tools (Chao & Martin, 2000; Creem-Regehr & Lee, 2005; Rushworth et al., 2001). In other words, there is evidence for two frontoparietal networks, one for tool perception and one for motor attention, that appear to operate in parallel and may have some functional overlap. Because such frontoparietal attentional networks are known to play a role in selection and consolidation of visual information, action affordances may provide an essential association between dorsal stream processing for function across time and space and ventral stream processing of subsequently presented visual input. Given that the posterior parietal cortex serves as the conduit from early visual processing in occipital cortices to motor output generation in the frontal lobe (Caminiti, Ferraina, & Johnson, 1996; Johnson, Ferraina, Bianchi, & Caminiti, 1996), this terminus of the dorsal stream is ideally situated not only to support action programming but also to provide modulatory feedback onto ventral stream areas in order to optimize processing of impending visual information.

Since tool processing and motor attention seem to share some neural substrates, we asked here whether action relationships between images of objects can result in enhanced attention for those objects. More specifically, we examined whether perception of a tool can lead to attentional enhancement towards subsequently presented objects that are consistent with the action afforded by the tool. We used a well-established manipulation of attention, known as the attentional blink (AB; Raymond, Shapiro, & Arnell, 1992), in which responding to one target typically leads to a transient yet robust lack of awareness for a second target presented within roughly 150–500 ms. An accumulation of behavioural (Giesbrecht, Bischof, & Kingstone, 2003; Giesbrecht & DiLollo, 1998; Raymond et al., 1992), electrophysiological (Kranczioch, Debener, & Engel, 2003; Vogel, Luck, & Shapiro, 1998) and functional imaging (Marois, Chun, & Gore, 2000; Marois, Yi, & Chun, 2004) data suggests that the AB is a result of a limited stage in attentional selection and consolidation processing typically subserved by a frontoparietal network, potentially overlapping with brain areas that support tool identification and action representation. Increased activity in such a network due to processing of a tool may provide a neural basis for the enhancement of attention for subsequently presented stimuli that are afforded by the activated representation of that tool. This attentional enhancement can be observed not only behaviourally, as a diminished attentional blink, but also using electrophysiological measures, such as event-related potentials (ERP).

A well-documented ERP component, the P3, has been associated with attentional allocation and working memory consolidation of perceptual stimuli, and is typically maximal over parietal areas for visual stimuli. Vogel et al. (1998) showed that the P3 component was suppressed for T2s that appeared during the AB window when targets were masked, showing a decrement in the frontoparietal processes that typically contribute toward target consolidation. The early sensory components of the ERP waveform were not affected by subsequent report or failure to report T2, demonstrating that the attentional blink is a postperceptual process related to later stages of attentional processing. Vogel and Luck (2002) further demonstrated that the onset latency of the P3 was affected by the timing between the first and second targets, such that targets appearing within the typical blink period without masks showed a delayed P3 even when they were successfully reported, thus supporting the notion that consolidation of T2 identity is delayed during the AB. In a more recent study, Kranczioch et al. (2003) separately examined the ERPs for T2s that were either detected (hits) or undetected (misses) both during and after the AB. In this case, all hits elicited a P3 regardless of blink timing, while responses to misses did not include a P3. Taking these findings together, we can predict that any detected target should elicit a P3, but that the latency of the P3 should be affected by the timing between the first and second targets particularly in conditions where target consolidation is enhanced.

To summarize, the goal of our current study was to show that although the dorsal and ventral streams process different aspects of visual input, they interact via attentional selection in order to optimize active behaviour, particularly when presented with tools whose functions change the affordances of other object forms. We tested the hypothesis that perception of a tool results in enhanced attention and consolidation specifically for objects that are compatible with the action afforded by the tool using the attentional blink in a behavioural study (experiment 1a) and a companion ERP study (experiment 2). We furthermore provide evidence that this effect is dependent upon pictorial presentation and not determined by purely semantic associations (experiment 1b). We presented images (experiments 1a and 2) or names (experiment 1b) of tools (T1) and their action complements (T2) embedded within a rapid serial visual presentation (RSVP) stream containing other common objects as distractors and comparison T2s, with varying lag intervals between T1 and T2. Enhanced selection and consolidation for T2 should be observed both as a reduction in the magnitude of the AB and as enhancements in the P3 component of the resulting ERP.