Abstract
The importance of visual cues in speech perception is illustrated by the McGurk effect, whereby a speaker’s facial movements affect speech perception. The goal of the present study was to evaluate whether the McGurk effect is also observed for sung syllables. Participants heard and saw sung instances of the syllables /ba/ and /ga/ and then judged the syllable they perceived. Audio-visual stimuli were congruent or incongruent (e.g., auditory /ba/ presented with visual /ga/). The stimuli were presented as spoken, sung in an ascending and descending triad (C E G G E C), and sung in an ascending and descending triad that returned to a semitone above the tonic (C E G G E C#). Results revealed no differences in the proportion of fusion responses between spoken and sung conditions confirming that cross-modal phonemic information is integrated similarly in speech and song.
Article PDF
References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262. doi:10.1016/j.cub.2004.01.029
Besson, M., Faïta, F., Peretz, I., Bonnel, A.-M., & Requin, J. (1998). Singing in the brain: Independence of lyrics and tunes. Psychological Science, 9, 494–498. doi:10.1111/1467-9280.00091
Bigand, E., Tillmann, B., Poulin, B., D’Adamo, D. A., & Madurell, F. (2001). The effect of harmonic context on phoneme monitoring in vocal music. Cognition, 81, B11-B20. doi:10.1016/S0010-0277(01)00117-2
Boersma, P., & Weenink, D. (2007). Praat: Doing phonetics by computer (Version 4.6.29) [Computer program]. Available at www.praat .org.
Bonnel, A.-M., Faïta, F., Peretz, I., & Besson, M. (2001). Divided attention between lyrics and tunes of operatic songs: Evidence for independent processing. Perception & Psychophysics, 63, 1201–1213.
Burnham, D., & Dodd, B. (2004). Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect. Developmental Psychobiology, 45, 204–220. doi:10.1002/dev.20032
Collister, L. B., & Huron, D. (2008). Comparison of word intelligibility in spoken and sung phrases. Empirical Musicology Review, 3, 109–125.
Dahl, S., & Friberg, A. (2007). Visual perception of expressiveness in musicians’ body movements. Music Perception, 24, 433–454. doi:10.1525/mp.2007.24.5.433
Davidson, J. W. (1993). Visual perception of performance manner in the movements of solo musicians. Psychology of Music, 21, 103–113. doi:10.1177/030573569302100201
Green, K. P., Kuhl, P. K., Meltzoff, A. N., & Stevens, K. N. (1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception & Psychophysics, 50, 524–536.
Gregg, J. W., & Scherer, R. C. (2006). Vowel intelligibility in classical singing. Journal of Voice, 20, 198–210. doi:10.1016/j.jvoice.2005.01.007
Hidalgo-Barnes, M., & Massaro, D. W. (2007). Read my lips: An animated face helps communicate musical lyrics. Psychomusicology, 19, 3–12.
Hollien, H., Mendes-Schwartz, A. P., & Nielsen, K. (2000). Perceptual confusions of high-pitched sung vowels. Journal of Voice, 14, 287–298. doi:10.1016/S0892-1997(00)80038-7
Jesse, A., & Massaro, D. W. (2010). Seeing a singer helps comprehension of the song’s lyrics. Psychonomic Bulletin & Review, 17, 323–328.
Krumhansl, C. L. (1990). Cognitive foundations of musical pitch. New York: Oxford University Press.
MacDonald, J., Andersen, S., & Bachmann, T. (2000). Hearing by eye: How much spatial degradation can be tolerated? Perception, 29, 1155–1168. doi:10.1068/p3020
Massaro, D. W. (2004). From multisensory integration to talking heads and language learning. In G. Calvert, C. Spence, & B. E. Stein (Eds.), Handbook of multisensory processes (pp. 153–176). Cambridge, MA: MIT Press.
Massaro, D. W., & Cohen, M. M. (1983). Evaluation and integration of visual and auditory information in speech perception. Journal of Experimental Psychology: Human Perception & Performance, 9, 753–771. doi:10.1037/0096-1523.9.5.753
Massaro, D. W., Cohen, M. M., & Smeele, P. M. T. (1996). Perception of asynchronous and conflicting visible and auditory speech. Journal of the Acoustical Society of America, 100, 1777–1786. doi:10.1121/1.417342
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. doi:10.1038/264746a0
Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58, 351–362.
Neely, K. K. (1956). Effect of visual factors on the intelligibility of speech. Journal of the Acoustical Society of America, 28, 1275–1277. doi:10.1121/1.1908620
Ostry, D. J., & Munhall, K. G. (1994). Control of jaw orientation and position in mastication and speech. Journal of Neurophysiology, 71, 1528–1545.
Paré, M., Richler, R. C., ten Hove, M., & Munhall, K. G. (2003). Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect. Perception & Psychophysics, 65, 553–567.
Patel, A. D. (2003). Language, music, syntax and the brain. Nature Neuroscience, 6, 674–681. doi:10.1038/nn1082
Patel, A. D. (2008). Music, language, and the brain. New York: Oxford University Press.
Patel, A. D. (2009). Music and the brain: Three links to language. In S. Hallam, I. Cross, & M. Thaut (Eds.), The Oxford handbook of music psychology (pp. 208–216). Oxford: Oxford University Press.
Peretz, I., & Coltheart, M. (2003). Modularity of music processing. Nature Neuroscience, 6, 688–691. doi:10.1038/nn1083
Saldaña, H. M., & Rosenblum, L. D. (1993). Visual influences on auditory pluck and bow judgments. Perception & Psychophysics, 54, 406–416.
Sams, M., Möttönen, R., & Sihvonen, T. (2005). Seeing and hearing others and oneself talk. Cognitive Brain Research, 23, 429–435. doi:10.1016/j.cogbrainres.2004.11.006
Schlaug, G., Marchina, S., & Norton, A. (2008). From singing to speaking: Why singing may lead to recovery of expressive language function in patients with Broca’s aphasia. Music Perception, 25, 315–323. doi:10.1525/mp.2008.25.4.315
Schutz, M., & Lipscomb, S. (2007). Hearing gestures, seeing music: Vision influences perceived tone duration. Perception, 36, 888–897. doi:10.1068/p5635
Smith, A. (1992). The control of orofacial movements in speech. Critical Reviews in Oral Biology & Medicine, 3, 233–267.
Straube, T., Schulz, A., Geipel, K., Mentzel, H.-J., & Miltner, W. H. R. (2008). Dissociation between singing and speaking in expressive aphasia: The role of song familiarity. Neuropsychologia, 46, 1505–1512. doi:10.1016/j.neuropsychologia.2008.01.008
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215. doi:10.1121/1.1907309
Sundberg, J. (1977). The acoustics of the singing voice. Scientific American, 3, 82–91.
’T Hart, J., Collier, R., & Cohen, A.(1990). A perceptual study of intonation: An experimental-phonetic approach to speech melody. Cambridge: Cambridge University Press.
Thompson, W. F., Graham, P., & Russo, F. A. (2005). Seeing music performance: Visual influences on perception and experience. Semiotica, 156, 203–227. doi:10.1515/semi.2005.2005.156.203
Thompson, W. F., Russo, F. A., & Livingstone, S. (2010). Facial expressions of pitch structure in music performance. Psychonomic Bulletin & Review, 17, 317–322. doi:10.3758/PBR.17.3.317
Thompson, W. F., Russo, F. A., & Quinto, L. (2008). Audio-visual integration of emotional cues in song. Cognition & Emotion, 22, 1457–1470. doi:10.1080/02699930701813974
Vines, B. W., Krumhansl, C. L., Wanderley, M. M., & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101, 80–113. doi:10.1016/j.cognition.2005.09.003
Zipse, L., Norton, A., Marchina, S., & Schlaug, G. (2009). Singing versus speaking in nonfluent aphasia. NeuroImage, 47, S39-S41. doi:10.1016/S1053-8119(09)71121-8
Author information
Authors and Affiliations
Corresponding author
Additional information
Data collection and manuscript preparation were supported by discovery grants from the Natural Sciences and Engineering Research Council of Canada and by an Australian Research Council grant to W.F.T.
Rights and permissions
About this article
Cite this article
Quinto, L., Forde Thompson, W., Russo, F.A. et al. A comparison of the McGurk effect for spoken and sung syllables. Attention, Perception, & Psychophysics 72, 1450–1454 (2010). https://doi.org/10.3758/APP.72.6.1450
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/APP.72.6.1450