Multimodal Integration of Time
Visual and Auditory Contributions to Perceived Duration and Sensitivity
Abstract
Recent studies suggest that the accuracy of duration discrimination for visually presented intervals is strongly impaired by concurrently presented auditory intervals of different duration, but not vice versa. Because these studies rely mostly on accuracy measures, it remains unclear whether this impairment results from changes in perceived duration or rather from a decrease in perceptual sensitivity. We therefore assessed complete psychometric functions in a duration discrimination task to disentangle effects on perceived duration and sensitivity. Specifically, participants compared two empty intervals marked by either visual or auditory pulses. These pulses were either presented unimodally, or accompanied by task-irrelevant pulses in the respective other modality, which defined conflicting intervals of identical, shorter, or longer duration. Participants were instructed to base their temporal judgments solely on the task-relevant modality. Despite this instruction, perceived duration was clearly biased toward the duration of the intervals marked in the task-irrelevant modality. This was not only found for the discrimination of visual intervals, but also, to a lesser extent, for the discrimination of auditory intervals. Discrimination sensitivity, however, was similar between all multimodal conditions, and only improved compared to the presentation of unimodal visual intervals. In a second experiment, evidence for multisensory integration was even found when the task-irrelevant modality did not contain any duration information, thus excluding noncompliant attention allocation as a basis of our results. Our results thus suggest that audiovisual integration of temporally discrepant signals does not impair discrimination sensitivity but rather alters perceived duration, presumably by means of a temporal ventriloquism effect.
References
(2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262. doi: 10.1016/s0960-9822(04)00043-0
(1977). The time-order error in judgments of duration. Canadian Journal of Psychology/Revue Canadienne de Psychologie, 31, 24.
(2003). Temporal ventriloquism: Crossmodal interaction on the time dimension: 2. Evidence from sensorimotor synchronization. International Journal of Psychophysiology, 50, 157–163. doi: 10.1016/s0167-8760(03)00131-4
(2010). Temporal preparation influences the dynamics of information processing: Evidence for early onset of information accumulation. Vision Research, 50, 1025–1034. doi: 10.1016/j.visres.2010.03.011
(1981). Cross-modal bias and perceptual fusion with auditory – visual spatial discordance. Perception & Psychophysics, 29, 578–584.
(1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.
(2012). Perceptual learning in temporal discrimination: Asymmetric cross-modal transfer from audition to vision. Experimental Brain Research, 221, 205–210.
(2008). Time and attention: Review of the literature. In , Psychology of Time (pp. 111–138). Bingley, UK: Emerald Group.
(2009). Auditory dominance over vision in the perception of interval duration. Experimental Brain Research, 198, 49–57. doi: 10.1007/s00221-009-1933-z
(1963). Estimation and evaluation. In , Handbook of mathematical psychology, Vol. 1, (pp. 429–469). New York, NY: Wiley.
(2009). Asymmetric cross-modal effects in time perception. Acta Psychologica, 130, 225–234. doi: 10.1016/j.actpsy.2008.12.008
(2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433.
(2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162–169. doi: 10.1016/j.tics.2004.02.002
(2001). The temporal cross-capture of audition and vision. Perception & Psychophysics, 63, 718–725.
(2010). Sensory-specific clock components and memory mechanisms: Investigation with parallel timing. European Journal of Neuroscience, 31, 1908–1914. doi: 10.1111/j.1460-9568.2010.07197.x
(1972). Auditory-visual differences in human temporal judgment. Perceptual and Motor Skills, 34, 623–633.
(2003). Variable foreperiods and temporal discrimination. Quarterly Journal of Experimental Psychology, 56A, 731–765. doi: 10.1080/02724980244000611
(2005). Hearing what the eyes see: Auditory encoding of visual temporal sequences. Psychological Science, 16, 228–235. doi: 10.1111/j.0956-7976.2005.00808.x
(1998). Automatic alerting does not speed late motoric processes in a reaction-time task. Nature, 391, 786–788.
(1985). The time-order error and its relatives: Mirrors of cognitive processes in comparing. Psychological Bulletin, 97, 35–61.
(2004). Sensory uncertainty governs the extent of audio-visual interaction. Vision Research, 44, 2875–2884. doi: 10.1016/j.visres.2004.07.001
(2011). Modality-independent role of the primary auditory cortex in time estimation. Experimental Brain Research, 209, 465–471. doi: 10.1007/s00221-011-2577-3
(2011). Crossmodal duration perception involves perceptual grouping, temporal ventriloquism, and variable internal clock rates. Attention, Perception, & Psychophysics, 73, 219–236. doi: 10.3758/s13414-010-0010-9
(1977). Foreperiod effects on time estimation and simple reaction time. Acta Psychologica, 41, 47–59.
(2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154–163. doi: 10.1016/s0926-6410(03)00089-2
(1965). A simplex method for function minimization. Computer Journal, 7, 308–313.
(1981). Foreperiod and simple reaction time. Psychological Bulletin, 89, 133–162.
(1997). The Video Toolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 10, 437–442.
(2011). Auditory stimulus timing influences perceived duration of co-occurring visual stimuli. Frontiers in Psychology, 2, 215. doi: 10.3389/fpsyg.2011.00215
(2012). Audiovisual interactions depend on context of congruency. Attention, Perception, & Psychophysics, 74, 563–574.
(2010). Temporal preparation decreases perceptual latency: Evidence from a clock paradigm. The Quarterly Journal of Experimental Psychology, 63, 2432–2451. doi: 10.1080/17470218.2010.485354
(1964). Auditory flutter-driving of visual flicker. Science, 145, 1328–1330. doi: 10.1126/science.145.3638.1328
(2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport: For Rapid Communication of Neuroscience Research, 12, 7–10. doi: 10.1097/00001756-200101220-00009
(2006). Crossmodal temporal discrimination: Assessing the predictions of a general pacemaker-counter model. Perception & Psychophysics, 68, 1140–1152.
(2004). Temporal ventriloquism: Sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception and Performance, 30, 513–518. doi: 10.1037/0096-1523.30.3.513
(1981). Auditory-visual conflicts in the perceived duration of lights, tones, and gaps. Journal of Experimental Psychology: Human Perception and Performance, 7, 1327–1339. doi: 10.1037/0096-1523.7.6.1327
(1998). Why “sounds are judged longer than lights”: Application of a model of the internal clock in humans. The Quarterly Journal of Experimental Psychology, 51B, 97–120.
(2001). Elementary signal detection theory. Oxford, UK: Oxford University Press.
(1997). Temporal cognition. Current Directions in Psychological Sciene, 6, 12–16.
