Abstract
Language processing requires the combination of compatible (auditory-vocal and visual-manual) or incompatible (auditory-manual and visual-vocal) sensory-motor modalities, and switching between these sensory-motor modality combinations is very common in every-day life. Sensory-motor modality compatibility is defined as the similarity of stimulus modality and the modality of response-related sensory consequences. We investigated the influence of sensory-motor modality compatibility during performing language-related cognitive operations on different linguistic levels. More specifically, we used a variant of the task-switching paradigm, in which participants had to switch between compatible or between incompatible sensory-motor modality combinations during a verbal semantic categorization (Experiment 1) or during a word-form decision (Experiment 2). The data show higher switch costs (i.e., higher reaction times and error rates in switch trials compared to repetition trials) in incompatible sensory-motor modality combinations than in compatible sensory-motor modality combinations. This was true for every language-related cognitive operation, regardless of the individual linguistic level. Taken together, the present study demonstrates that sensory-motor modality compatibility plays an important role in modality switching during language processing.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Badets, A., Koch, I., & Philipp, A. M. (2014). A review of ideomotor approaches to perception, cognition, action, and language: advancing a cultural recycling hypothesis. Psychological Research, 1–15.
Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660.
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617–645.
Borghi, A. M., & Pecher, D. (2011). Introduction to the special topic embodied and grounded cognition. Frontiers in Psychology, 2, 187.
Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). Modality switching in a property verification task: an ERP study of what happens when candles flicker after high heels click. Frontiers in Psychology, 2, 10.
Greenwald, A. G. (1972). On doing two things at once: Time sharing as a function of ideomotor compatibility. Journal of Experimental Psychology, 94, 52–57.
Hald, L. A., Marshall, J. A., Janssen, D. P., & Garnham, A. (2011). Switching modalities in a sentence verification task: ERP evidence for embodied language processing. Frontiers in Psychology, 2, 93.
Hazeltine, E., Ruthruff, E., & Remington, R. W. (2006). The role of input and output modality pairings in dual-task performance: Evidence for content-dependent central interference. Cognitive Psychology, 52, 291–345.
Hickok, G., Houde, J., & Rong, F. (2011). Sensorimotor integration in speech processing: Computational basis and neural organization. Neuron, 69, 407–422.
Howard, D., & Franklin, S. (1988). Missing the meaning? A cognitive neuropsychological study of processing of words by an aphasic patient. Cambridge, MA: MIT Press.
International Phonetic Association. (1999). Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge, UK: Cambridge University Press.
Kiesel, A., Steinhauser, M., Wendt, M., Falkenstein, M., Jost, K., Philipp, A. M., & Koch, I. (2010). Control and interference in task switching—a review. Psychological Bulletin, 136, 849–874.
Koch, I., Gade, M., Schuch, S., & Philipp, A. M. (2010). The role of inhibition in task switching: a review. Psychonomic Bulletin & Review, 17, 1–14.
Lukas, S., Philipp, A. M., & Koch, I. (2010). Switching attention between modalities: further evidence for visual dominance. Psychological Research, 74, 255–267.
Meiran, N. (2010). Task switching: Mechanisms underlying rigid vs. flexible self control. In: Hassin, R., Ochsner, K., & Trope Y. (Eds.), Self control in society, mind, and brain (pp. 202–220). New York, NY: Oxford University Press.
Monsell, S. (2003). Task switching. Trends in Cognitive Sciences, 7, 134–140.
Morton, J. (1969). The interaction of information in word recognition. Psychological Review, 76, 165–178.
Norris, D. (2013). Models of visual word recognition. Trends in Cognitive Sciences, 17, 517–524.
Pecher, D., Zeelenberg, R., & Barsalou, L. W. (2003). Verifying different-modality properties for concepts produces switching costs. Psychological Science, 14, 119–124.
Price, C. J. (2012). A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading. Neuroimage, 62, 816–847.
Pulvermüller, F., & Fadiga, L. (2010). Active perception: sensorimotor circuits as a cortical basis for language. Nature Reviews Neuroscience, 11, 351–360.
Raij, T., Uutela, K., & Hari, R. (2000). Audiovisual integration of letters in the human brain. Neuron, 28, 617–625.
Spence, C., Nicholls, M. E., & Driver, J. (2001). The cost of expecting events in the wrong sensory modality. Perception and Psychophysics, 63, 330–336.
Stephan, D. N., & Koch, I. (2010). Central cross-talk in task switching: Evidence from manipulating input-output modality compatibility. Journal of Experimental Psychology, Learning, Memory, and Cognition, 36, 1075–1081.
Stephan, D. N., & Koch, I. (2011). The role of input–output modality compatibility in task switching. Psychological Research, 75, 491–498.
Vandierendonck, A., Liefooghe, B., & Verbruggen, F. (2010). Task switching: Interplay of reconfiguration and interference control. Psychological Bulletin, 13, 601–626.
Acknowledgments
This research was funded by DFG grant PH 156/3-1. We thank Anna M. Borghi and the anonymous reviewers for helpful comments on an earlier version of this paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Schaeffner, S., Koch, I. & Philipp, A.M. The role of sensory-motor modality compatibility in language processing. Psychological Research 80, 212–223 (2016). https://doi.org/10.1007/s00426-015-0661-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00426-015-0661-1