Abstract
The importance of multisensory integration for perception and action has long been recognised. Integrating information from individual senses increases the chance of survival by reducing the variability in the incoming signals, thus allowing us to respond more rapidly. Reaction times (RTs) are fastest when the components of the multisensory signals are simultaneous. This response facilitation is traditionally attributed to multisensory integration. However, it is unclear if facilitation of RTs occurs when stimuli are perceived as synchronous or are actually physically synchronous. Repeated exposure to audiovisual asynchrony can change the delay at which multisensory stimuli are perceived as simultaneous, thus changing the delay at which the stimuli are integrated—perceptually. Here we set out to determine how such changes in multisensory integration for perception affect our ability to respond to multisensory events. If stimuli perceived as simultaneous were reacted to most rapidly, it would suggest a common system for multisensory integration for perception and action. If not, it would suggest separate systems. We measured RTs to auditory, visual, and audiovisual stimuli following exposure to audiovisual asynchrony. Exposure affected the variability of the unisensory RT distributions; in particular, the slowest RTs were either speed up or slowed down (in the direction predicted from shifts in perceived simultaneity). Additionally, the multisensory facilitation of RTs (beyond statistical summation) only occurred when audiovisual onsets were physically synchronous, rather than when they appeared simultaneous. We conclude that the perception of synchrony is therefore independent of multisensory integration and suggest a division between multisensory processes that are fast (automatic and unaffected by temporal adaptation) and those that are slow (perceptually driven and adaptable).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The GEE is similar to the one described in “Data analysis” with a few minor differences. The within-subjects variables were SOA and “week” since in order to have enough data at a given SOA, multisensory RT data needed to be combined across sessions.
Using an uncorrected alpha-rate, there were several additional SOAs that produced RTs that were significantly slower than the race model demonstrating a trend towards response inhibition: SOA = −80 ms, mean response inhibition (x̄) = 2.6 ms ± (SE)1.7 ms, t 29 = 2.09, p = .045; SOA = −60 ms, x̄= 3.9 ± 1.6 ms, t 29 = 2.52, p = .018; SOA = −40 ms, x̄= 3.5 ± 1.6 ms, t 29 = 2.14, p = .041; SOA = 60 ms, x̄ = 4.0 ± 1.6 ms, t 29 = 2.59, p = .015; SOA = 100 ms, x̄ = 4.1 ± 1.9 ms, t 29 = 2.18, p = .038.
References
Balota DA, Spieler DH (1999) Word frequency, repetition, and lexicality effects in word recognition tasks: beyond measures of central tendency. J Exp Psychol Gen 128(1):32
Berman AL (1961) Interaction of cortical responses to somatic and auditory stimuli in anterior ectosylvian gyrus of cat. J Neurophysiol 24(6):608–620
Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009. doi:10.1162/0898929041502733
Colonius H, Diederich A (2006) The race model inequality: interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154. doi:10.1037/0033-295X.113.1.148
Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vis 9(12):7.1–716. doi:10.1167/9.12.7
Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404
Diederich A, Colonius H (2015) The time window of multisensory integration: relating reaction times and judgments of temporal order. Psychol Rev 122(2):232–241. doi:10.1037/a0038696
Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721
Donders FC (1868) /1969). Die Schnelligkeit psychischer Processe [On the speed of mental processes]. Acta Psychol 30:412–431
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433. doi:10.1038/415429a
Farmer ME, Klein RM (1995) The evidence for a temporal processing deficit linked to dyslexia: a review. Psychon Bull Rev 2(4):460–493. doi:10.3758/Bf03210983
Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389. doi:10.1007/s00221-010-2240-4
Foucher JR, Lacambre M, Pham BT, Giersch A, Elliott MA (2007) Low time resolution in schizophrenia Lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophr Res 97(1–3):118–127. doi:10.1016/j.schres.2007.08.013
Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778. doi:10.1038/nn1268
Gibbon J, Rutschmann R (1969) Temporal order judgement and reaction time. Science 165(3891):413–415
Goodale MA, Milner AD (1992) Separate visual pathways for perception and action. Trends Neurosci 15(1):20–25
Hanley JA, Negassa A, Forrester JE (2003) Statistical analysis of correlated data using generalized estimating equations: an orientation. Am J Epidemiol 157(4):364–375
Hanson JV, Heron J, Whitaker D (2008) Recalibration of perceived time across sensory modalities. Exp Brain Res 185(2):347–352. doi:10.1007/s00221-008-1282-3
Harrar V, Harris LR (2005) Simultaneity constancy: detecting events with touch and vision. Exp Brain Res 166(3–4):465–473. doi:10.1007/s00221-005-2386-7
Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524. doi:10.1007/s00221-007-1253-0
Harrar V, Tammam J, Perez-Bellido A, Pitt A, Stein J, Spence C (2014) Multisensory integration and attention in developmental dyslexia. Curr Biol 24(5):531–535. doi:10.1016/j.cub.2014.01.029
Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293
Jaekl P, Soto-Faraco S, Harris LR (2012) Perceived size change induced by audiovisual temporal delays. Exp Brain Res 216(3):457–462. doi:10.1007/s00221-011-2948-9
Jaekl P, Seidlitz J, Harris LR, Tadin D (2015) Audiovisual delay as a novel cue to visual distance. PLoS ONE 10(10):e0141125. doi:10.1371/journal.pone.0141125
Jaskowski P (1992) Temporal-order judgment and reaction time for short and long stimuli. Psychol Res 54(3):141–145
Jiang H, Lepore F, Ptito M, Guillemot J-P (1994) Sensory interactions in the anterior ectosylvian cortex of cats. Exp Brain Res 101(3):385–396
Keetels M, Vroomen J (2012) Exposure to delayed visual feedback of the hand changes motor-sensory synchrony perception. Exp Brain Res 219(4):431–440. doi:10.1007/s00221-012-3081-0
Kopinska A, Harris LR (2004) Simultaneity constancy. Perception 33(9):1049–1060
Lacouture Y, Cousineau D (2008) How to use MATLAB to fit the ex-Gaussian and other probability functions to a distribution of response times. Tutor Quant Methods Psychol 4(1):35–45
Leone LM, McCourt ME (2012) The question of simultaneity in multisensory integration. Human Vis Electron Imaging Xvii. doi:10.1117/12.912183
Leone LM, McCourt ME (2013) The roles of physical and physiological simultaneity in audiovisual multisensory facilitation. i-Perception 4(4):213–228. doi:10.1068/i0532
Leone LM, McCourt ME (2015) Dissociation of perception and action in audiovisual multisensory integration. Eur J Neurosci 42(11):2915–2922
Leth-Steensen C, Elbaz ZK, Douglas VI (2000) Mean response times, variability, and skew in the responding of ADHD children: a response time distributional approach. Acta Psychol (Amst) 104(2):167–190
Luce RD (1986) Response times. Oxford University Press, New York
Maier JX, Di Luca M, Noppeney U (2011) Audiovisual asynchrony detection in human speech. J Exp Psychol Hum Percept Perform 37(1):245
Matzke D, Wagenmakers EJ (2009) Psychological interpretation of the ex-Gaussian and shifted Wald parameters: a diffusion model analysis. Psychon Bull Rev 16(5):798–817. doi:10.3758/PBR.16.5.798
Megevand P, Molholm S, Nayak A, Foxe JJ (2013) Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS ONE 8(8):e71608. doi:10.1371/journal.pone.0071608
Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temp Factors. J Neurosci 7(10):3215–3229
Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14(2):247–279
Miller J (1986) Timecourse of coactivation in bimodal divided attention. Percept Psychophys 40(5):331–343
Miller RL, Pluta SR, Stein BE, Rowland BA (2015) Relative unisensory strength and timing predict their multisensory product. J Neurosci 35(13):5213–5220. doi:10.1523/JNEUROSCI.4771-14.2015
Miyazaki M, Yamamoto S, Uchida S, Kitazawa S (2006) Bayesian calibration of simultaneity in tactile temporal order judgment. Nat Neurosci 9(7):875–877. doi:10.1038/nn1712
Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14(1):115–128
Navarra J, Hartcher-O’Brien J, Piazza E, Spence C (2009) Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proc Natl Acad Sci U S A 106(23):9169–9173. doi:10.1073/pnas.0810486106
Otto TU, Mamassian P (2012) Noise and correlations in parallel perceptual decision making. Curr Biol 22(15):1391–1396. doi:10.1016/j.cub.2012.05.031
Otto TU, Dassy B, Mamassian P (2013) Principles of multisensory behavior. J Neurosci 33(17):7463–7474. doi:10.1523/JNEUROSCI.4678-12.2013
Pöppel E, Schill K, von Steinbuchel N (1990) Sensory integration within temporally neutral systems states: a hypothesis. Naturwissenschaften 77(2):89–91
Ratcliff R, Murdock BB (1976) Retrieval processes in recognition memory. Psychol Rev 83(3):190
Roach NW, Heron J, Whitaker D, McGraw PV (2011) Asynchrony adaptation reveals neural population code for audio–visual timing. Proc Biol Sci 278(1710):1314–1322. doi:10.1098/rspb.2010.1737
Roseboom W, Arnold DH (2011) Twice upon a time multiple concurrent temporal recalibrations of audiovisual speech. Psychol Sci 22(7):872–877
Roseboom W, Linares D, Nishida S (2015) Sensory adaptation for timing perception. Proc R Soc Lon B: Biol Sci 282:20142833. doi:10.1098/rspb.2014.2833
Sanders A (1983) Towards a model of stress and human performance. Acta Psychol (Amst) 53(1):61–97
Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519–R521
Spence C, Nicholls ME, Driver J (2001a) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63(2):330–336
Spence C, Shore DI, Klein RM (2001b) Multisensory prior entry. J Exp Psychol-Gen 130(4):799–832. doi:10.1037//0096-3445.130.4.799
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, Mass
Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266. doi:10.1038/nrn2331
Sternberg S, Knoll RL (1973) The perception of temporal order: Fundamental issues and a general model. Attention and Performance IV:629–685
Tünnermann J, Petersen A, Scharlau I (2015) Does attention speed up processing? Decreases and increases of processing rates in visual prior entry. J Vis 15(3):1–27
Ulrich R (1987) Threshold models of temporal-order judgments evaluated by a ternary response task. Percept Psychophys 42(3):224–239
Ulrich R, Miller J, Schroter H (2007) Testing the race model inequality: an algorithm and computer programs. Behav Res Methods 39(2):291–302
Van der Burg E, Olivers CN, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053–1065. doi:10.1037/0096-1523.34.5.1053
Vibell J, Klinge C, Zampini M, Spence C, Nobre AC (2007) Temporal order is coded temporally in the brain: early ERP latency shifts underlying prior entry in a crossmodal temporal order judgment task. J Cogn Neurosci 19:109–120
Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72(4):871–884. doi:10.3758/APP.72.4.871
Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Brain Res Cogn Brain Res 22(1):32–35. doi:10.1016/j.cogbrainres.2004.07.003
Wallace MT, Stevenson RA (2014) The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia 64:105–123. doi:10.1016/j.neuropsychologia.2014.08.005
Woodworth RS, Schlosberg H (1938) Experimental psychology. Holt, New York, pp 327–328
Yarrow K, Jahn N, Durant S, Arnold DH (2011) Shifts of criteria or neural timing? The assumptions underlying timing perception studies. Conscious Cogn 20(4):1518–1531
Yuan X, Bi C, Yin H, Li B, Huang X (2014) The recalibration patterns of perceptual synchrony and multisensory integration after exposure to asynchronous speech. Neurosci Lett 569:148–152
Acknowledgements
Thanks to Alice Newton-Fenner for help with data collection and Phil Jaekl for excellent manuscript revision suggestions. Vanessa Harrar was supported by the Mary Somerville Junior Research Fellowship (Somerville College, Oxford University, UK), and by the Banting Fellowship (Canadian Institute of Health Research, Canada). LRH was supported by the Natural Sciences and Engineering Research Council of Canada.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Harrar, V., Harris, L.R. & Spence, C. Multisensory integration is independent of perceived simultaneity. Exp Brain Res 235, 763–775 (2017). https://doi.org/10.1007/s00221-016-4822-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-016-4822-2