Abstract
Living in a complex and multisensory environment involves constant interaction between perception and action. There is evidence that multisensory integration is governed by temporal factors, such as physiological synchrony between cross-modal stimuli favouring multisensory benefit, and the existence of a range of asynchrony between the stimuli which affords their binding (the temporal window of integration). These factors were examined in this study in a bimanual sensorimotor synchronization task with cross-modal stimuli. Participants synchronized each hand to a pair of audio–tactile stimuli, in which the asynchrony between onsets of auditory and tactile stimuli was systematically manipulated. In cross-modal conditions, they were instructed to tap either to the auditory stimuli or to tactile stimuli. The results reported a temporal window of integration of 160 ms centred around 40 and 80 ms (tactile first). Moreover, the temporal interval between the auditory and tactile stimuli affected the stability of bimanual coordination and of synchronization exclusively when participants were instructed to synchronize with tactile stimuli. Overall, the results indicate that both physiological asynchrony and temporal window of integration apply to cross-modal integration in a bimanual synchronization task. In addition, it shows the effect of auditory dominance onto multisensory temporal processes. This study sheds light on the role of temporal factors in multisensory processes when perception and actions are rhythmic and coupled.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alais D, Newell FN, Mamassian P (2010) Multisensory processing in review: from physiology to behaviour. See Perceiving 23:3–38. doi:10.1371/journal.pone.0011283
Aschersleben G (2002) Temporal control of movements in sensorimotor synchronization. Brain Cogn 48:66–79. doi:10.1006/brcg.2001.1304
Aschersleben G, Bertelson P (2003) Temporal ventriloquism: crossmodal interaction on the time dimension. 2. Evidence from sensorimotor synchronization. Int J Psychophysiol 50:157–163. doi:10.1016/S0167-8760
Beek PJ, Peper CE, Stegeman DF (1995) Dynamical models of movement coordination. Hum Mov Sci 14:573–608. doi:10.1016/0167-9457(95)00028-5
Bell AH, Meredith MA, Van Opstal AJ, Munoz DP (2005) Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements. J Neurophysiol 93:3659–3673. doi:10.1152/jn.01214.2004
Bialunska A, Dalla Bella S, Jaskowski P (2011) Increasing stimulus intensity does not affect sensorimotor synchronization. Psychol Res 75:43–53. doi:10.1007/s00426-010-0282-7
Blais M, Albaret J-M, Tallet J (2015) Is there a link between sensorimotor coordination and inter-manual coordination? Differential effects of auditory and/or visual rhythmic stimulations. Exp Brain Res 233:3261–3269. doi:10.1007/s00221-015-4394-6
Byblow WD, Carson RG, Goodman D (1994) Expressions of asymmetries and anchoring in bimanual coordination. Hum Mov Sci 13:3–28
Carver FW, Fuchs A, Jantzen KJ, Kelso JAS (2002) Spatiotemporal analysis of the neuromagnetic response to rhythmic auditory stimulation: rate dependence and transient to steady-state transition. Clin Neurophysiol 113:1921–1931. doi:10.1016/S1388-2457(02)00299-7
Colonius H, Diederich A (2010) The optimal time window of visual-auditory integration: a reaction time analysis. Front Integr Neurosci. doi:10.3389/fnint.2010.00011
Colonius H, Diederich A, Steenken R (2009) Time-window-of-integration (TWIN) model for saccadic reaction time: effect of auditory masker level on visual-auditory spatial interaction in elevation. Brain Topogr 21:177–184. doi:10.1007/s10548-009-0091-8
Diederich A (1995) Intersensory facilitation of reaction time: evaluation of counter and diffusion coactivation models. J Math Psychol 39:197–215. doi:10.1006/jmps.1995.1020
Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66:1388–1404. doi:10.3758/BF03195006
Diederich A, Colonius H (2007a) Modeling spatial effects in visual-tactile saccadic reaction time. Percept Psychophys 69:56–67. doi:10.3758/BF03194453
Diederich A, Colonius H (2007b) Why two “Distractors” are better than one: modeling the effect of non-target auditory and tactile stimuli on visual saccadic reaction time. Exp Brain Res 179:43–54. doi:10.1007/s00221-006-0768-0
Diederich A, Colonius H (2009) Crossmodal interaction in speeded responses: time window of integration model. In: Raab M (ed) Progress in brain research. Elsevier, Amsterdam, pp 119–135
Diederich A, Colonius H (2015) The time window of multisensory integration: relating reaction times and judgments of temporal order. Psychol Rev 122:232–241. doi:10.1037/a0038696
Diedrichsen J, Shadmehr R, Ivry RB (2010) The coordination of movement: optimal feedback control and beyond. Trends Cogn Sci 14:31–39. doi:10.1016/j.tics.2009.11.004
Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on “sensory-specific” brain regions, neural responses, and judgments. Neuron 57:11–23. doi:10.1016/j.neuron.2007.12.013
Elliott MT, Wing AM, Welchman AE (2010) Multisensory cues improve sensorimotor synchronisation. Eur J Neurosci 31:1828–1835. doi:10.1111/j.1460-9568.2010.07205.x
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433. doi:10.1038/415429a
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169. doi:10.1016/j.tics.2004.02.002
Fink PW, Foo P, Jirsa VK, Kelso JS (2000) Local and global stabilization of coordination by sensory information. Exp Brain Res 134:9–20. doi:10.1007/s002210000439
Fraisse P (1980) Les sycnhronisations sensori-motrices aux rythmes [The sensorimotor sycnhronization of rhythms]. In: Requin J (ed) Anticipations et comportements. Centre National, Paris, pp 233–257
Frissen I, Ziat M, Campion G et al (2012) The effects of voluntary movements on auditory-haptic and haptic-haptic temporal order judgments. Acta Psychol (Amst) 141:140–148. doi:10.1016/j.actpsy.2012.07.010
Fujisaki W, Nishida S (2009) Audio-tactile superiority over visuo-tactile and audio-visual combinations in the temporal resolution of synchrony perception. Exp Brain Res 198:245–259. doi:10.1007/s00221-009-1870-x
Ghazanfar A, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cogn Sci 10:278–285. doi:10.1016/j.tics.2006.04.008
Haken H (1988) Information and self-organization: a macroscopic approach to complex systems. Springer, Berlin
Haken H, Kelso JAS, Bunz H (1985) A theoretical model of phase transitions in human hand movements. Biol Cybern 51:347–356. doi:10.1007/BF00336922
Harrar V, Harris LR (2005) Simultaneity constancy: detecting events with touch and vision. Exp Brain Res 166:465–473. doi:10.1007/s00221-005-2386-7
Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186:517–524. doi:10.1007/s00221-007-1253-0
Harris L, Harrar V, Jaekl P, Kopinska A (2009) Mechanisms of simultaneity constancy. In: Nijhawan R, Khurana B (eds) Space and time in perception and action. Cambridge university Press, Cambridge, pp 232–253
Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289
Howard IS, Ingram JN, Körding KP, Wolpert DM (2009) Statistics of natural movements are reflected in motor errors. J Neurophysiol 102:1902–1910. doi:10.1152/jn.00013.2009
Jones M, Dzhafarov EN (2014) Unfalsifiability and mutual translatability of major modeling schemes for choice reaction time. Psychol Rev 121:1–32
Kato M, Konishi Y (2006) Auditory dominance in the error correction process: a synchronized tapping study. Brain Res 1084:115–122. doi:10.1016/j.brainres.2006.02.019
Kelso JAS (1984) Phase transitions and critical behavior in human bimanual coordination. Am J Physiol 15:1000–1004
Kelso JAS (1995) Dynamic patterns: the self-organization of brain and behavior. MIT Press, Cambridge
Kelso JAS, Delcolle JD, Schöner G (1990) Action-perception as a pattern formation process. In: Jeannerod M (ed) Attention and performance XIII. Erlbaum, Hillsdale, pp 139–169
Kitagawa N, Zampini M, Spence C (2005) Audiotactile interactions in near and far space. Exp Brain Res 166:528–537. doi:10.1007/s00221-005-2393-8
Kolers PA, Brewster JM (1985) Rhythms and responses. J Exp Psychol Hum Percept Perform 11:150–167. doi:10.1037/0096-1523.11.2.150
Lagarde J, Kelso JAS (2006) Binding of movement, sound and touch: multimodal coordination dynamics. Exp Brain Res 173:673–688. doi:10.1007/s00221-006-0410-1
Lagarde J, Zelic G, Mottet D (2012) Segregated audio-tactile events destabilize the bimanual coordination of distinct rhythms. Exp Brain Res 219:409–419. doi:10.1007/s00221-012-3103-y
Lakatos P, Chen CM, O’Connell MN et al (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–292. doi:10.1016/j.neuron.2006.12.011
Leise T, Cohen A (2007) Nonlinear oscillators at our fingertips. Am Math Mon 114:14–28
Mégevand P, Molholm S, Nayak A, Foxe JJ (2013) Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS One 8:e71608. doi:10.1371/journal.pone.0071608
Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14:274–279
Müller K, Aschersleben G, Schmitz F et al (2008) Inter- versus intramodal integration in sensorimotor synchronization: a combined behavioral and magnetoencephalographic study. Exp Brain Res 185:309–318. doi:10.1007/s00221-007-1155-1
Murray MM, Molholm S, Michel CM et al (2005) Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15:963–974. doi:10.1093/cercor/bhh197
Nangini C, Ross B, Tam F, Graham SJ (2006) Magnetoencephalographic study of vibrotactile evoked transient and steady-state responses in human somatosensory cortex. Neuroimage 33:252–262. doi:10.1016/j.neuroimage.2006.05.045
Occelli V, Spence C, Zampini M (2011) Audiotactile interactions in temporal perception. Psychon Bull Rev 18:429–454. doi:10.3758/s13423-011-0070-4
Otto TU, Dassy B, Mamassian P (2013) Principles of multisensory behavior. J Neurosci 33:7463–7474. doi:10.1523/JNEUROSCI.4678-12.2013
Paillard J (1949) Quelques données psychophysiologiques relatives au déclenchement de la commande motrice [Some psychophysiological data relating to the triggering of motor commands]. Annee Psychol 48:28–47
Powers AR, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29:12265–12274. doi:10.1523/JNEUROSCI.3501-09.2009
R core Team (2013) R: a language and environment for statistical computing. R A Lang Environ Stat Comput
Raab DH (1962) Statistical facilitation of simple reaction time. Trans N Y Acad Sci 43:574–590. doi:10.1017/CBO9781107415324.004
Repp BH (2005) Sensorimotor synchronization: a review of the tapping literature. Psychon Bull Rev 12:969–992. doi:10.3758/BF03206433
Repp BH, Penel A (2002) Auditory dominance in temporal processing: new evidence from synchronization with simultaneous visual and auditory sequences. J Exp Psychol Hum Percept Perform 28:1085–1099. doi:10.1037/0096-1523.28.5.1085
Repp BH, Penel A (2004) Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Psychol Res 68:252–270. doi:10.1007/s00426-003-0143-8
Ronsse R, Miall RC, Swinnen SP (2009) Multisensory integration in dynamical behaviors: maximum likelihood estimation across bimanual skill learning. J Neurosci 29:8419–8428. doi:10.1523/JNEUROSCI.5734-08.2009
Rowland BA, Quessy S, Stanford TR, Stein BE (2007) Multisensory integration shortens physiological response latencies. J Neurosci 27:5879–5884. doi:10.1523/JNEUROSCI.4986-06.2007
Schöner G, Kelso JAS (1988a) A synergetic theory of environmentally-specified and learned patterns of movement coordination II. Component oscillator dynamics. Biol Cybern 58:81–89
Schöner G, Kelso JAS (1988b) A synergic theory of environmentally-specified and learned patterns of movement coordination I. Relative phase dynamics. Biol Cybern 58:71–80
Schöner G, Haken H, Kelso JAS (1986) A stochastic theory of phase transitions in human hand movement. Biol Cybern 53:247–257. doi:10.1007/BF00336995
Schöner G, Zanone PG, Kelso JAS (1992) Learning as change of coordination dynamics: theory and experiment. J Mot Behav 24:29–48
Sejdić E, Fu Y, Pak A et al (2012) The effects of rhythmic sensory cues on the temporal dynamics of human gait. PLoS One. doi:10.1371/journal.pone.0043104
Soto-Faraco S, Deco G (2009) Multisensory contributions to the perception of vibrotactile events. Behav Brain Res 196:145–154. doi:10.1016/j.bbr.2008.09.018
Spence C, Parise C (2010) Prior-entry: a review. Conscious Cogn 19:364–379. doi:10.1016/j.concog.2009.12.001
Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13:519–521. doi:10.1016/S0960-9822(03)00445-7
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge
Swinnen SP (2002) Intermanual coordination: from behavioural principles to neural–network interactions. Nat Rev Neurosci 3:348–359. doi:10.1038/nrn807
Todd JW (1912) Reaction to multiple stimuli. Arch Psychol 25:1–65
Todorov E, Jordan MI (2002) Optimal feedback control as theory of motor coordination. Nat Neurosci 5:1226–1235. doi:10.1038/nn963
Turvey MT (1990) Coordination. Am Psychol 45:938–953. doi:10.1037/0003-066X.45.8.938
Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72:871–884. doi:10.3758/APP.72.4.871
Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638–667. doi:10.1037/0033-2909.88.3.638
Wilson EC, Reed CM, Braida LD (2009) Integration of auditory and vibrotactile stimuli: effects of phase and stimulus-onset asynchrony. J Acoust Soc Am 126:1960–1974
Wing AM, Doumas M, Welchman AE (2010) Combining multisensory temporal information for movement synchronisation. Exp Brain Res 200:277–282. doi:10.1007/s00221-009-2134-5
Zampini M, Brown T, Shore DI et al (2005) Audiotactile temporal order judgments. Acta Psychol (Amst) 118:277–291. doi:10.1016/j.actpsy.2004.10.017
Zelic G, Mottet D, Lagarde J (2012) Behavioral impact of unisensory and multisensory audio-tactile events: pros and cons for interlimb coordination in juggling. PLoS One 7:e32308. doi:10.1371/journal.pone.0032308
Zelic G, Mottet D, Lagarde J (2016) Perceptuo-motor compatibility governs multisensory integration in bimanual coordination dynamics. Exp Brain Res 234:463–474. doi:10.1007/s00221-015-4476-5
Acknowledgments
We thank two anonymous reviewers for their careful reading of the manuscript and suggestions on earlier drafts of the manuscript. The study was supported by a grant from the European Commission (FP7-ICT-2013-10 “BeatHealth: Health and Wellness on the Beat”). The authors declare no competing financial interests. We would like to thank Bernard Gilbert, Déborah Varoqui and Robin Salesse for their help at various stages of this project.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Roy, C., Dalla Bella, S. & Lagarde, J. To bridge or not to bridge the multisensory time gap: bimanual coordination to sound and touch with temporal lags. Exp Brain Res 235, 135–151 (2017). https://doi.org/10.1007/s00221-016-4776-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-016-4776-4