Abstract
It has repeatedly been demonstrated that the presence of multiple cues in different sensory modalities can enhance behavioral performance by speeding responses, increasing accuracy, and/or improving stimulus detection. Despite an extensive knowledge base as to how the spatial, temporal, and physical (e.g., intensity) characteristics of multisensory stimuli influence such enhancements, little is known about the role of semantic or contextual congruence. Our hypothesis was that semantically congruent multisensory stimuli would result in enhanced behavioral performance, and that semantically incongruent multisensory stimuli would result in either no enhancement or a decrement in behavioral performance. The results from a redundant cue feature discrimination task clearly demonstrate that congruent cross-modal stimulation improves behavioral performance. This effect is specific to the multisensory stimuli, as no improvements are seen in the presence of redundant unimodal stimulus pairs. In contrast, incongruent stimulus pairs result in behavioral decrements for both multisensory and paired unimodal stimuli. These results highlight that in addition to such simple stimulus features as space, time and relative effectiveness, the semantic content of a multisensory stimulus plays a critical role in determining how it is processed by the nervous system.
Similar content being viewed by others
References
Ben-Artzi E, Marks LE (1995) Visual-auditory interaction in speeded classification: role of stimulus difference. Percept Psychophys 57:1151–1162
Ben-Artzi E, Marks LE (1999) Processing linguistic and perceptual dimensions of speech: interactions in speeded classification. J Exp Psychol Hum Percept Perform 25:579–595
Bushara KO, Hanakawa T, Immisch I, Toma K, Kansaku K, Hallett M (2003) Neural correlates of cross-modal binding. Nat Neurosci 6:190–195
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657
Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438
Cowan N (1989a) The reality of cross-modal Stroop effects. Percept Psychophys 45:87–88
Cowan N (1989b) A reply to Miles, Madden, and Jones (1989). Percept Psychophys 45:82–84
Cowan N, Barron A (1987) Cross-modal, auditory-visual Stroop interference and possible implications for speech memory. Percept Psychophys 41:393–401
Driver J, Spence C (1998) Cross-modal links in spatial attention. Philos Trans R Soc Lond B Biol Sci 353:1319–1331
Forster B, Cavina-Pratesi C, Aglioti SM, Berlucchi G (2002) Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time. Exp Brain Res 143:480–487
Frassinetti F, Bolognini N, Ladavas E (2002) Enhancement of visual perception by crossmodal visuo-auditory interaction. Exp Brain Res 147:332–343
Frens MA, Van Opstal AJ (1998) Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Res Bull 46:211–224
Frens MA, Van Opstal AJ, Van der Willigen RF (1995) Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept Psychophys 57:802–816
Hadjikhani N, Roland PE (1998) Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J Neurosci 18:1072–1084
Harrington LK, Peck CK (1998) Spatial disparity affects visual-auditory interactions in human sensorimotor processing. Exp Brain Res 122:247–252
Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, London
Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R (1994) Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J Exp Psychol Hum Percept Perform 20:131–153
Laurienti PJ, Burdette JH, Wallace MT, Yen Y-F, Field AS, Stein BE (2002) Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci 14:1-10
Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223
Lewald J, Guski R (2003) Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Brain Res Cogn Brain Res 16:468–478
Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17:447–453
Macaluso E, Frith CD, Driver J (2000) Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206–1208
MacLeod CM (1991) Half a century of research on the Stroop effect: an integrative review. Psychol Bull 109:163–203
McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748
Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365:350–354
Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662
Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857
Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7:3215–3229
Miles C, Jones DM (1998) The fallacy of the cross-modal Stroop effect: a rejoinder to Cowan (1989). Percept Psychophys 24:85–86
Miles C, Madden C, Jones DM (1989) Cross-modal, auditory-visual Stroop interference: a reply to Cowan and Barron (1987). Percept Psychophys 45:77–81; discussion 82–76
Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cognit Psychol 14:247–279
Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128
Ratcliff R (1979) Group reaction time distributions and an analysis of distribution statistics. Psychological Bulletin 86:446–461
Schroger E, Widmann A (1998) Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology 35:755–759
Shimada H (1990) Effect of auditory presentation of words on color naming: the intermodal Stroop effect. Percept Mot Skills 70:1155–1161
Snedecor G, Cochran W (1989) Statistical methods. Iowa State University Press, Ames, Iowa
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA
Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448:355–358
Stroop J (1935) Studies of interference in serial verbal reactions. J Exp Psychol 18:643–662
Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488
Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance, vol. 1: sensory processes and perception. Wiley, New York, pp 25.21–25.36
Acknowledgements
This work was supported in part by NS042568 to PJL. We would like to thank Drs. Dennis Slice and Pete Santago and David Hairston and Ersin Bayram for helpful discussions concerning the curve fitting methodologies and statistical analyses used for this work.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Vincent transformation
Each subject’s response times (n) were divided into (q) quantiles (q=21 representing from 0 to 100% of responses in increments of 5). Each response time was then replaced by q equal response times. For example, if the first response time was 350 ms and 21 quantiles were desired then the first 21 values would all be 350 ms. Repeating this procedure for all response times generates a list that contains n*q values. The response time for the first quantile was then calculated by summing the first n response times and dividing by n. The value for the second quantile was calculated by averaging the next n response times and dividing by n. This procedure was repeated q times to generate the Vincent transformed CDF. The cumulative probability curves were then averaged across subjects to generate group CDFs. The CDFs were statistically compared across conditions by performing paired t-tests between the response times at each quantile (p values were corrected for multiple comparisons using the Bonferroni method).
After applying the Vincent Averaging Method to generate group CDFs, common data points were transformed from the x axis (response time) to the y axis (quantile or probability). Thus, each CDF had a data point at every quantile and the response time for that quantile was known. However, there was not a data point for every response time, so the quantile for any one time bin may or may not have been known. Because the Miller Inequality Model requires that probabilities at each time bin be used to calculate the race model, the data had to be fit to generate common data points for each time bin. In addition, the error had to be transformed from the response time axis to the probability axis to perform statistics at each time bin.
Curve fitting
The Vincent transformed CDF for each subject was fit to the CDF of the gamma distribution,
where α and λ are the shape and scale parameters, respectively. The shift parameter, x0, was added to the account for the different starting times of each Vincent transformed CDF. The incomplete gamma function,γ(a,z), is defined as
Γ(α) is the complete gamma function, which is equal to the incomplete gamma function, γ(α,z), with z equal to 0 (Snedecor and Cochran 1989). Nonlinear regression using the Levenberg Marquardt algorithm provided in Mathematica (Wolfram Inc., Champaign, IL, USA) was used to fit the CDFγ function to the Vincent transformed CDFs. The fitting of the Vincent transformed CDFs was simplified from a three parameter fitting problem to a two parameter fitting problem by setting the shape parameter, α, equal to 4, which was chosen heuristically. However, our experience with selecting different values of α revealed little difference on the curve fits with the scale parameter, λ, and shift parameter, x0, compensating for different values of α. The adequacy of the fits was assessed using the Chi Squared goodness-of-fit test. We were unable to reject the null hypothesis (p>0.05) for any of the curve fits, indicating that the gamma distribution was a reasonable model for the data.
Propagation of error
Error propagation for the CDFγ(x,α,λ,x0) was determined from the error propagation formula
where \(\frac{{\partial CDF_{\gamma } }} {{\partial n}}\) is the partial derivative of the CDFγfunction with respect to n, σnis the standard deviation of the parameter n, where n is equal to the parameters, x, λ, and x0. \(\sigma ^{2}_{{\lambda ,x_{0} }} \) is the estimated covariance between the scaling parameter, λ, and the shifting parameter, x0. The partial derivatives of the CDF function were calculated with Mathematica and are as follows
and
where Γ(α) is the complete gamma function. The error for each bin was calculated and used to determine significance.
Miller inequality calculations
The fitted data for each subject were used to calculate the race model prediction using the Miller Inequality Equation ((Pstimulus 1+ P stimulus 2)−(P stimulus 1*P stimulus 2)) where P is the probability of response for any give time bin. This equation was solved for each time bin to generate predicted response time CDFs if the dual stimulus conditions were processed independently.
Rights and permissions
About this article
Cite this article
Laurienti, P.J., Kraft, R.A., Maldjian, J.A. et al. Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158, 405–414 (2004). https://doi.org/10.1007/s00221-004-1913-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-004-1913-2