Abstract
Very little is known about the coordination of movement in combination with stimuli such as sound and touch. The present research investigates the hypothesis that both the type of action (e.g., a flexion or extension movement) and the sensory modality (e.g., auditory or tactile) determine the stability of multimodal coordination. We performed a parametric study in which the ability to synchronize movement, touch and sound was explored over a broad range of stimulus frequencies or rates. As expected, synchronization of finger movement with external auditory and tactile stimuli was successfully established and maintained across all frequencies. In the key experimental conditions, participants were instructed to synchronize peak flexion of the index finger with touch and peak extension with sound (and vice-versa). In this situation, tactile and auditory stimuli were delivered counter-phase to each other. Two key effects were observed. First, switching between multimodal coordination patterns occurred, with transitions selecting one multimodal pattern (flexion with sound and extension with touch) more often than its partner. This finding indicates that the stability of multimodal coordination is influenced by both the type of action and the stimulus modality. Second, at higher rates, transitions from coherent to incoherent phase relations between touch, movement and sound occurred, attesting to the breakdown of multimodal coordination. Because timing errors in multimodal coordination were systematically altered when compared to unimodal control conditions we are led to consider the role played by time delays in multimodal coordination dynamics.
Similar content being viewed by others
Notes
The Kuiper test detects whether the observed distribution differs significantly from randomness. The null hypothesis that the observed distribution is uniform is tested against any alternative, including multi-peaked distributions with not necessarily equally spaced peaks. The Kuiper test has been recently applied to detect transient synchronization and desynchronization of coupled oscillators (Tass 2004).
References
Aschersleben G (2002) Temporal control of movements in sensorimotor synchronization. Brain and Cogn 48:66–79
Batschelet E (1981) Circular statistics in biology. Academic, London
Beek PJ (1989) Timing and phase locking in cascade juggling. Ecol Psychol 1:55–96
Beek PJ, Peper CE, Daffertshofer A (2002) Modeling rhythmic interlimb coordination: beyond the Haken-Kelso-Bunz model. Brain Cogn 48:149–65
Bernstein IH (1970) Can we see and hear at the same time? Some recent studies of intersensory facilitation of reaction time. Acta Psychol 33:21–35
Beuter A, Larocque D, Glass L (1989) Complex oscillations in a human motor system. J Mot Behav 21:277–289
Bresciani JP, Ernst MO, Drewing K, Bouyer G, Maury V, Kheddar A (2005) Feeling what you hear: auditory signals can modulate tactile tap perception. Exp Brain Res 162:172–180
Bressler SL, Kelso JAS (2001) Cortical coordination dynamics and cognition. Trends in Cog Sci 5:26–36
Brovelli A, Ding M, Ledberg A, Chen Y, Nakamura R, Bressler SL (2004) Beta oscillations in a large-scale sensorimotor cortical network: directional influences revealed by Granger causality. Proc Natl Acad Sci USA 101:9849–9854
Bushara KO, Hanakawa T, Toma K, Kansaku K, Hallet M (2003) Neural correlates of cross- modal binding. Natur Neurosci 6:190–195
Byblow WD, Carson RG, Goodman D (1994) Expressions of asymmetries and anchoring in bimanual coordination. Hum Mov Sci 13:3–28
Calvert GA (2001) Crossmodal Processing in the Human Brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123
Calvert GA, Spence C, Stein BE (2004) Handbook of multisensory processes. MIT Press, Cambridge
Carson RG (1995) The dynamics of isometric bimanual coordination. Exp Brain Res 105:465–476
Carson RG (1996) Neuromuscular-skeletal constraints upon the dynamics of perception-action coupling. Exp Brain Res 110:99–110
Carson RG (2004) Governing coordination: why do muscles matter? In: Jirsa VK, Kelso JAS (eds) Coordination dynamics: issues and trends, vol 1. Springer series in Understanding Complex Systems, Springer, Berlin Heidelberg New York, pp 141–154
Carson RG, Kelso JAS (2004) Governing coordination: behavioral principles and neural correlates. Exp Brain Res 154:267–274
Carson RG, Riek S (1998) Influence of joint position on the dynamics of perception-action coupling. Exp Brain Res 121:103–114
Chen Y, Ding M, Kelso JAS (1997) Long memory processes (1/fα type) in human coordination. Phys Rev Lett 79:4501–4504
Chen Y, Ding M, Kelso JAS (2003) Task-related power and coherence changes in neuromagnetic activity during visuomotor coordination. Exp Brain Res 148:105–116
Cheyne D, Roberts LE, Gaetz W, Bosnyak D, Nahmias C, Christoforou N, Weinberg H (1998) Somatotopic organization of human somatosensory cortex: a comparison of EEG, MEG and FMRI methods. In: Koga Y, Narata K, Hirata K (eds) Brain topography today. Elsevier, Amsterdam, pp 76–81
Daffertshoffer A, Peper CE, Beek PJ (2000) Spectral analysis of event-related electroencephalographic signals. Phys Lett A 266:290–302
DeGuzman GC, Kelso JAS (1991) Multifrequency behavioral patterns and the phase attractive circle map. Biol Cyb 64:485–495
DeGuzman GC, Kelso JAS (1992) The flexible dynamics of biological coordination: living in the niche between order and disorder. In: Mittenthal J, Baskin A (eds) Principles of organization in organisms. Santa Fe Institute studies in the science of complexity, vol 13. Addison-Wesley, Longman, pp 11–34
Dehner LR, Keniston LP, Clemo HR, Meredith MA (2004) Cross-modal circuitry between auditory and somatosensory areas of the cat anterior ectosylvian sulcal cortex: a ‘new’ inhibitory form of multisensory convergence. Cereb Cortex 14:387–403
Dhamala M, Assisi CG, Jirsa VK, Kelso, JAS (2003) Multisensory integration displayed in the human EEG and FMRI is parametrized by frequency and time delays. [CDROM] Program No. 912.14. 2003 Abstract Viewer/Itinerary Planner. Society for Neuroscience, Washington, DC
Dhamala M, Jirsa VK, Ding M (2004) Enhancement of neural synchrony by time delay. Phys Rev Lett 92:1–4
Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement of reaction time: effects of stimulus onset and intensity. Percept Psychophys 66:1388–1404
Driver J, Spence C (2000) Multisensory perception: beyond modularity and convergence. Curr Biol 10:R737–R735
Dunlap K (1910) Reactions to rhythmic stimuli, with attempt to synchronize. Psychol Rev 17:399–416
Edelman GM (2004) Wider than the sky. Yale University Press, New Haven
Edelman GM, Tononi G (2000) A universe of consciousness: how matter becomes imagination. Basic Books, New York
Engström DA, Kelso JAS, Holroyd T (1996) Reaction-anticipation transitions in human perception-action patterns. Hum Mov Sci 15:809–832
Fingelkurts AA, Fingelkurts AA (2004) Making complexity simpler: multivariability and metastability in the brain. Int J Neurosci 114:843–862
Fink PW, Kelso JAS, Foo P, Jirsa VK (2000) Local and global stabilization of coordination by sensory information. Exp Brain Res 134:9–20
Finney SA, Warren WH (2002) Delayed auditory feedback and rhythmic tapping: evidence for a critical interval shift. Percept Psychophys 64:896–908
Fogassi L, Gallese V (2004) Action as a binding key to multisensory integration. In: Calvert GA, Spence C, Stein BE (eds) The handboock of multisensory processes. MIT Press, Cambridge, pp 425–441
Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high density electrical mapping. Cogn Brain Res 10:77–83
Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol 88:540–543
Frank TD, Dafferstoffer A, Peper CE, Beek PJ, Haken H (2000) Toward a comprehensive theory of brain activity: coupled oscillators systems under external forces. Physica D 144:62–86
Freeman WJ, Holmes MD (2005) Metastability, instability and state transition in neocortex. Neural Networks, 18:497–504
Friston KJ (1997) Transients, metastability and neural dynamics. NeuroImage 5:164–171
Friston KJ (2000) The labile brain. I. Transients and spatio-temporal receptive fields. PhilosTrans R Soc Lond B Biol Sci 355:253–265
Fu KM, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE (2003) Auditory cortical neurons respond to somatosensory stimulation. J Neurosci 23:7510–7515
Fuchs A, Jirsa VK (2000) The HKB model revisited: how varying the degree of symmetry controls dynamics. Hum Mov Sci 19:425–444
Fuchs A, Kelso JAS, Haken H (1992) Phase transitions in the human brain: spatial mode dynamics. Int J Bif Chaos 2:917–939
Fuchs A, Jirsa VK, Kelso JAS (2000a) Theory of the relation between human brain activity (MEG) and hand movements. Neuroimage 11:359–369
Fuchs A, Mayville J, Cheyne D, Weinberg H, Deecke L, Kelso JAS (2000b) Spatiotemporal analysis of neuromagnetic events underlying the emergence of coordinative instabilities. Neuroimage 12:71–84
Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nature Neurosci 7:773–778
Gibson JJ (1966) The senses considered as perceptual systems. Houghton Mifflin, Boston
Glass L, Beuter A, Laroque D (1988) Time delays, oscillations, and chaos in physiological control systems. Math Biosci 90:111–125
Gobbelé R, Schürmann M, Forss N, Juottonen K, Buchner H, Hari R (2003) Activation of the human posterior parietal and temporal cortices during audiotactile interaction. Neuroimage 20:503–511
Graziano MS, Gross CG (1995) The representation of extrapersonal space: a possible role for bimodal, visual-tactile neurons. In: Gazzaniga MS (ed) The cognitive neurosciences. MIT Press, Cambridge, pp 1021–1034
Graziano MS, Cooke DF, Taylor CS (2000) Coding the location of the arm by sight. Science 290:1782–1786
Guckenheimer J, Holmes P (1990) Nonlinear oscillations, dynamical systems, and bifurcations of vector fields. Springer, Berlin Heidelberg New York
Guest S, Catmur C, Lloyd D, Spence C (2002) Audiotactile interactions in roughness perception. Exp Brain Res 147:161–171
Haken H (1983) Advanced synergetics. Instability hierarchies of self-organizing systems and devices. Springer. Berlin Heidelberg New York
Haken H (1996) Principles of brain functioning: a synergetic approach to brain activity, behavior and cognition. Springer, Berlin Heidelberg New York
Haken H, Kelso JAS, Bunz H (1985) A theoretical model of phase transitions in human bimanual coordination. Biol Cybern 51: 347–356
Haken H, Peper CE, Beek PJ, Daffertshofer A (1996) A model for phase transitions in human hand movements during multifrequency tapping. Physica D 90:179–196
Hamalainen H, Kokoni J, Sams M, Reinikainen K, Naatanen R (1990) Human somatosensory evoked potentials to mechanical pulses and vibration: contributions of SI and SII somatosensory cortices to P50 and P100 components. Electroenceph Clin Neurophysiol 75:13–21
Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293
Hummel F, Gerloff C (2005) Larger interregional synchrony is associated with greater behavioral success in a complex sensory integration task in humans. Cereb Cortex 15:970–978
Ishida F, Sawada YE (2004) Human hand moves proactively to the external stimulus: an evolutional strategy for minimizing transient error. Phys Rev Lett 93:1681051–1681053
Jantzen KJ, Steinberg FL, Kelso JAS (2004) Brain networks underlying human timing behavior are influenced by prior context. Proc Natl Acad Sci 101:6815–6820
Jeka JJ, Lackner JR (1994) Fingertip contact influences human postural control. Exp Brain Res 100:495–502
Jeka JJ, Schöner G, Dijkstra T, Ribeiro P, Lackner JR (1997) Coupling of fingertip somatosensory information to head and body sway. Exp Brain Res 113:475–483
Jirsa VK, Kelso JAS (2000) Spatiotemporal pattern formation in continuous systems with heterogeneous connection topologies. Phys Rev E 62:8462–8465
Jirsa VK, Kelso JAS (eds) (2004a) Coordination dynamics: issues and trends, vol 1. Springer series in Understanding Complex Systems. Springer, Berlin Heidelberg New York
Jirsa VK, Kelso JAS (2004b) Integration and segregation of perceptual and motor behavior. In: Jirsa VK, Kelso JAS (eds) Coordination dynamics: issues and trends, vol 1. Springer series in understanding complex systems. Springer, Berlin Heidelberg New York, pp 243–259
Jirsa VK, Fink PW, Foo P, Kelso JAS (2000) Parametric stabilization of biological coordination: a theoretical model. J Biol Phys 26:85–112
Jousmäki V, Hari R (1998) Parchment-skin illusion: sound-biased touch. Curr Biol 8:R190
Katsumata H, Zatiorsky V, Sternad D (2003) Control of ball-racket interactions in the rhythmic propulsion of elastic and non-elastic balls. Exp Brain Res 149:17–29
Kelso JAS (1984) Phase transitions and critical behavior in human bimanual coordination. Am J Physiol 15:R1000–R1004
Kelso JAS (1991) Behavioral and neural pattern generation: The concept of neurobehavioral dynamical system (NBDS). In: Koepchen HP (ed) Cardiorespiratory and Motor Coordination. Springer, Berlin Heidelberg New York
Kelso JAS (1992) Coordination dynamics of human brain and behavior. Springer Proc in Physics 69:223–234
Kelso JAS (1995) Dynamic patterns: the self-organization of brain and behavior. MIT Press, Cambridge
Kelso JAS (2001) How the brain changes its mind: metastable coordination dynamics. In: The emergence of the mind. Fondazione Carlo Erba, Milano pp 93–101
Kelso JAS, DeGuzman GC (1988) Order in time: how cooperation between the hands informs the design of the brain. In: Haken H (ed) Neural and synergetic computers. Springer, Berlin Heidelberg New York, pp 180–196
Kelso JAS, Scholz JP, Schöner G (1986) Non equilibrium phase transitions in coordinated biological motion: critical fluctuations. Phys Lett A 118:279–284
Kelso JAS, Schöner G, Scholz JP, Haken H (1987) Phase-locked modes, phase transitions and component oscillators in biological motion. Phys Scripta 35:79–87
Kelso JAS, Delcolle JD, Schöner G (1990) Action-perception as a pattern formation process. In: Jeannerod M (ed) Attention and performance XIII. Erlbaum, Hillsdale, pp 139–169
Kelso JAS, DeGuzman GC, Holroyd T (1991) Synergetic dynamics of biological coordination with special reference to phase attraction and intermittency. In: Haken H, Koepchen HP (eds) Rhythms in Physiological Systems, vol. 55. Springer Series in Synergetics. Springer, Berlin Heidelberg New York, pp 195–213
Kelso JAS, Bressler SL, Buchanan S, DeGuzman GC, Ding M, Fuchs A, Holroyd T (1992) A phase transition in human brain and behavior. Phys Lett A 169:134–144
Kelso JAS, Fink PW, DeLaplain CR, Carson RG (2001) Haptic information stabilizes and destabilizes coordination dynamics. Proc R Soc Lond B 268:1207–1213
Koch C (2004) The quest for consciousness. CO Roberts & Co, Englewood
Kuiper NH (1960) Tests concerning random points on a circle. Proc K Ned Akad Wet, Ser A Math Sci 63:38–47
Lam K, Kakigi R, Kaneoke Y, Naka D, Maeda K, Suzuki H (1999) Effects of visual and auditory stimulation on somatosensory evoked magnetic fields. Clin Neurophysiol 110:295–304
Langenberg U, Hefter H, Kessler KR, Cooke JD (1998) Sinusoidal forearm tracking with delayed visual feedback. I. Exp Brain Res 118:161–170
Lewkowicz DJ (2000) The development of intersensory temporal perception: an epigenetic systems/ limitations view. Psychol Bull 126:281–308
Lloyd DM, Shore DI, Spence C, Calvert GA (2003) Multisensory representation of limb position in human premotor cortex. Nat Neurosci 6:17–18
Lütkenhöner B, Lammertmann C, Simoes C, Hari R (2002) Magnetoencephalographic correlates of audiotactile interaction. NeuroImage 15:509–522
Mardia KV (1972) Statistics of directional data. Academic Press, London
Mayville JM, Bressler SL, Fuchs A, Kelso JAS (1999) Spatiotemporal reorganization of electrical activity in the human brain associated with a timing transition in rhythmic auditory-motor coordination. Exp Brain Res 127:371–381
Mayville JM, Jantzen KJ, Fuchs A, Steinberg F, Kelso JAS (2002) Cortical and subcortical networks underlying syncopated and synchronized coordination revealed using fMRI. Hum Brain Mapping 17:214–219
Meredith MA (2002) On the neural basis for multisensory convergence: a brief overview. Cogn Brain Res 14:31–40
Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221: 389–391
Meredith MA, Stein BE (1986) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365:350–354
Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7:3215–3229
Meyer-Lindenberg A, Ziemann U, Hajak G, Cohen L, Berman KF (2002) Transitions between dynamical states of differing stability in the human brain. Proc Natl Acad Sci USA 99:10948–10953
Miall RC, Weir DJ, Stein JF (1986) Manual tracking of visual targets by trained monkeys. Behav Brain Sci 20:185–201
Molhom S, Ritter W, Murray MM, Javitt DC, Schroder CE, Foxe JJ (2002) Multisensory audio-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cog Brain Res 14:115–128
Murray EA, Mishkin M (1985) Amygdalectomy impairs crossmodal association in monkeys. Science 228:604–606
Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ (2005) Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15:963–974
Nickerson RS (1973) Intersensory facilitation of reaction time: energy summation or preparation enhancement? Psychol Rev 80:489–509
Nunez PL, Wingeier BM, Silberstein B (2001) Spatial-temporal structures of human alpha rhythms: theory, microcurrent sources, multiscale measurements, and global binding of local networks. Hum Brain Mapp 13:125–164
Peper L, Beek PJ, Van Wieringen PCW (1995) Frequency induced transitions in bimanual tapping. Biol Cybern 73:301–309
Picton TW, Hillyard SA, Krausz HI, Galambos R (1974) Human auditory evoked potentials. I. Evaluation of components. Electroenceph clin Neurophysiol 36:179–190
Raab D (1962) Statistical facilitation of simple reaction time. Trans N Y Acad Sci 43:574–590
Schmidt RC, Shaw BK, Turvey MT (1993) Coupling dynamics in interlimb coordination. J Exp Psychol Hum Percept Perform 19:397–415
Schöner G, Kelso JAS (1988) A synergetic theory of environmentally specified and learned patterns of movement coordination. I. Relative phase dynamics. Biol Cybern 58:71–80
Schöner G, Haken H, Kelso JAS (1986) A stochastic model of phase transitions in human hand movement. Biol Cybern 53:247–257
Schöner G, Yiang WY, Kelso JAS (1990) A synergetic theory of quadrupedal gaits and gait transitions. J Theor Biol 142:359–391
Shore DI, Spry E, Spence C (2002) Confusing the mind by crossing the hands. Cogn Brain Res 14:153–163
Spence C, Nicholls ME, Gillespie N, Driver J (1998) Cross-modal links in exogenous covert spatial orienting between touch, audition, and vision. Percept Psychophys 60:544–557
Stein BE (1998) Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Exp Brain Res 123:124–135
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge
Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448:355–358
Stins JF, Michaels CF (1999) Strategy differences in oscillatory tracking: stimulus – hand versus stimulus-manipulandum coupling. J Exp Psychol Hum Percept Perform 25:1793–1812
Swinnen SP, Walter CB, Lee TD, Serrien DJ (1993) Acquiring bimanual skills: contrasting forms of information feedback for interlimb decoupling. J Exp Psychol Learn Mem Cogn 19:1328–1344
Tass PA (1995) Phase and frequency shifts of two nonlinearly coupled oscillators. Z Phys B 99:111–121
Tass PA (2004) Transmission of stimulus-locked responses in two coupled phase oscillators. Phys Rev E 69:051909
Tass PA, Kurths J, Rosenblum MG, Guasti G, Hefter H (1996) Delay- induced transitions in visually guided movements. Phys Rev E 54:R2224–R2227
Todd JW (1912) Reaction to multiple stimuli. Arch Psychol 25:1–65
Tuller B, Kelso JAS (1989) Environmentally specified patterns of movement coordination in normal and split-brain subjects. Exp Brain Res 75:306–316
Turvey MT (1990) Coordination. Am Psychol 45:938–953
Turvey MT (2004) Impredicativity, dynamics, and the perception–action divide. In: Jirsa VK and Kelso JAS (eds) Coordination dynamics: issues and trends, vol 1. Springer series in Understanding Complex Systems. Springer, Berlin Heidelberg New York, pp 1–20
Turvey MT, Schmidt RC (1994) A low dimensional nonlinear dynamic governing interlimb rhythmic coordination. In: Swinnen SP et al (eds) Interlimb coordination: neural, dynamical and cognitive constraints. Academic Press, San Diego, pp 278–300
Varela F, Lachaux JP, Rodriguez E, Martinerie J (2001) The brainweb: phase synchronization and large-scale integration. Nat Rev Neurosci 2:229–39
Vercher JL, Gauthier GM (1992) Oculo-manual coordination control: ocular and manual tracking of visual targets with delayed visual feedback of the hand motion. Exp Brain Res 90:599–609
Wallenstein GV, Kelso JAS, Bressler SL (1995) Phase transitions in spatiotemporal patterns of brain activity and behavior. Physica D 84:626–634
Wimmers RH, Beek PJ, van Wieringen FCW (1992) Phase transitions in rhythmic tracking movements: a case of unilateral coupling. Hum Mov Sci 11:217–226
Woodrow H (1932) The effect of rate of sequence upon the accuracy of synchronization. J Exp Psychol 15:357–379
Yeung MK, Strogatz SH (1999) Time delay in the Kuramoto model of coupled oscillators. Phys Rev Lett 82:648–651
Yuste R, MacLean JN, Smith J, Lansner A (2005) The cortex as a central pattern generator. Nat Rev Neurosci 6:477–483
Zanone PG, Kelso JAS (1992) Evolution of behavioral attractors with learning: nonequilibrium phase transitions. J Exp Psychol Hum Percept Perform 18:403–421
Acknowledgements
This work was supported by National Institute of Mental Health grants MH42900 and MH01386 to JASK. Julien Lagarde was supported by Enactive Interfaces, a network of excellence (IST contract #002114) of the Commission of the European Community during part of the writing of this work. We wish to thank Drs Gonzalo deGuzman, Viktor Jirsa, Collins Assisi, Mukesh Dhamala, Emmanuelle Tognoli and Olivier Oullier for fruitful discussion.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lagarde, J., Kelso, J. Binding of movement, sound and touch: multimodal coordination dynamics. Exp Brain Res 173, 673–688 (2006). https://doi.org/10.1007/s00221-006-0410-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-006-0410-1