Auditory capture of vision: examining temporal ventriloquism

Abstract

Four experiments investigated whether irrelevant sounds can influence the perception of lights in a visual temporal order judgment task, where participants judged which of two lights appeared first. In Experiment 1, presenting a sound before the first light and after the second light improved performance relative to baseline (sounds appearing simultaneously with the lights), as if the sounds pulled the perception of lights further apart in time. Experiment 2 ruled out an alerting explanation for this effect and indicated that the performance improvement resulted from the second sound trailing the second light. Experiment 3 excluded the possibility that leading or simultaneous sounds were interfering with performance and revealed that only the second sound had an effect within the temporal window known to support multisensory integration. Experiment 4 demonstrated that sounds intervening between the two lights led to a decline in performance, as if the sounds pulled the lights closer together. The results suggest a ‘temporal ventriloquism’ phenomenon analogous to spatial ventriloquism.

Introduction

Perception of the world is inherently a multisensory experience. The brain integrates information from the different sensory modalities to better interact with the environment. Multisensory integration is frequently studied using intermodal conflict, as in the ventriloquist effect whereby the perceived location of a sound shifts towards a visual stimulus presented at a different position [1], [10]. Much of the multisensory literature has focused on spatial interactions of this type and on identity interactions such as the McGurk effect [15]. In the McGurk effect, what is being heard is influenced by what is being seen (for example, when hearing /ba/ but seeing the speaker say /ga/ the final perception may be /da/). However, it is also the case that the senses can interact in time, that is, not where or what is being perceived but when it is being perceived. In the present study we investigated how the perception of time between two visual events is affected by presenting two sounds at conflicting times relative to the lights.

The temporal relationships between inputs from the different senses play an important role in multisensory integration. Indeed, a window of synchrony between auditory and visual events is crucial to spatial ventriloquism, as the effect disappears when the audio-visual asynchrony exceeds approximately 300 ms [28]. This is also the case in the McGurk effect, which fails to occur when the audio-visual asynchrony exceeds 200–300 ms [13], [18].

In the spatial domain, vision can bias the perceived location of sounds whereas sounds rarely influence visual localization. One key reason for this asymmetry seems to be that vision provides more accurate location information than sound [32], [33]. Thus, when auditory and visual stimuli co-occur from different positions, the perceived location of the sound is shifted towards the visual event as the best perceptual solution. Outside the spatial domain however, audition can bias vision [24], [25]. For example, multiple beeps accompanying a single flash can induce the perception of multiple flashes [25].

In the temporal domain evidence suggests that audition provides more accurate temporal information than vision [32], [9], [26], [30]. In keeping with the notion of greater auditory temporal acuity, audition has been found to capture visual temporal perception in a phenomenon called auditory ‘driving’ [26], [7], [19]. Here, changes in the rate of a repetitive clicking sound induce corresponding changes in the perceived rate of a repetitive flashing light [7], [31]. This phenomenon has been questioned, however, because it relies on subjective estimation and may reflect a response bias rather than a perceptual effect [6]. In a recent investigation, Fendrich and Corballis [6] attempted to overcome the limitations of auditory driving studies by asking participants to report the location of a rotating visual marker on a clock-face at the time of a flash. When irrelevant repetitive clicks preceded the flash the position reported was earlier, and when the clicks followed the flash the position reported was later. Unfortunately, this finding could reflect post-perceptual processes in the form of a response or spatial bias rather than a temporal perception bias, as the time estimation was based on the perceived location of a moving flash on a clock-face (see Ref. [29] for similar concerns). Consequently, the shortcomings inherent to the previous auditory driving research have not yet been fully addressed. Moreover, it is unclear whether audition may influence vision when non-repetitive stimuli are used, as repetitive stimuli carry supplementary rhythmic information.

Taken together however, the results are suggestive and compatible with the hypothesis tested in the current study: that auditory events can alter the perceived timing of target lights. This would demonstrate a temporal analogue of the spatial ventriloquist effect, where visual events can alter the perceived location of target sounds.