Top

Gepubliceerd in:

01-01-2007 | Original Article

Hearing lips in a second language: visual articulatory information enables the perception of second language sounds

Auteurs: Jordi Navarra, Salvador Soto-Faraco

Gepubliceerd in: Psychological Research | Uitgave 1/2007

Abstract

We investigated the effects of visual speech information (articulatory gestures) on the perception of second language (L2) sounds. Previous studies have demonstrated that listeners often fail to hear the difference between certain non-native phonemic contrasts, such as in the case of Spanish native speakers regarding the Catalan sounds /ɛ/ and /e/. Here, we tested whether adding visual information about the articulatory gestures (i.e., lip movements) could enhance this perceptual ability. We found that, for auditory-only presentations, Spanish-dominant bilinguals failed to show sensitivity to the /ɛ/–/e/ contrast, whereas Catalan-dominant bilinguals did. Yet, when the same speech events were presented audiovisually, Spanish-dominants (as well as Catalan-dominants) were sensitive to the phonemic contrast. Finally, when the stimuli were presented only visually (in the absence of sound), none of the two groups presented clear signs of discrimination. Our results suggest that visual speech gestures enhance second language perception at the level of phonological processing especially by way of multisensory integration.

vorige artikel Taking the brain serious: introduction to the special issue on integration in and across perception and action

volgende artikel Kinematic cues in perceptual weight judgement and their origins in box lifting

Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15, 839–843.CrossRefPubMed

Arnold, P., & Hill, F. (2001). Bisensory augmentation: a speechreading advantage when speech is clearly audible and intact. British Journal of Psychology, 92, 339–355.CrossRef

Auer, E. T., & Bernstein, L. (1997). Speechreading and the structure of the lexicon: computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. Journal of the Acoustical Society of America, 102, 3704–3710.CrossRefPubMed

Best, C. (1995). A direct realist view of cross-language speech perception. In W. Strange (Ed.). Speech perception and linguistic experience. Issues in cross-language research(pp. 171–204) Baltimore, MD: York Press.

Bosch, L., Costa, A., & Sebastian-Gallés, N. (2000). First and second language vowel perception in early bilinguals. European Journal of Cognitive Psychology, 12, 189–222.CrossRef

Burnham, D. K. (1998) Language specificity in the development of auditory-visual speech perception. In R. Campbell, B. Dodd& D.Burnham (Eds.) Hearing by eye II: advances in the psychology of speechreading and auditory–visual speech(pp. 27–60). Hove, UK: Psychology Press.

Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Cross, C., & Vatikiotis-Bateson, E. (2003). Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport, 14, 2213–2218.CrossRefPubMed

Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport, 10, 2619–2623.PubMedCrossRef

Calvert, G., Spence, C.,& Stein, B. E. (Eds.). (2004). The handbook of multisensory processes. Cambridge: MIT Press.

Campbell, R. (2004). Audiovisual speech processing. In K. Brown (Ed.). The Encyclopedia of language and linguistics, 2nd edition—Editor-in-chief Keith Brown. Oxford, UK: Elsevier.

Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. (2002). Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory. Clinical Neurophysiology, 113, 495–506.CrossRefPubMed

Davis, C., & Kim, J. (2004). Audio-visual interactions with intact clearly audible speech. Quarterly Journal of Experimental Psychology, 57, 1103–1121.PubMed

Ernst, M. O., & Banks, M. S. (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433.CrossRefPubMed

Ernst, M. O., & Bulthoff, H. H. (2004). Merging the senses into a robust percept TRENDS in Cognitive Sciences, 8, 162–169.

Fadiga, L., Craighero, L., Buccino, G., & Rizzolatti, G. (2002). Speech listening specifically modulates the excitability of tongue muscles: a TMS study. European Journal of Neuroscience, 15, 399–402.CrossRefPubMed

Flege, J. (1995). Second language speech learning: Theory, findings and problems. In W. Strange (Ed.). Speech perception and linguistic experience. Issues in cross-language research (pp. 233–272), Baltimore, MD: York Press.

Forster, K. L., & Forster, J. C. (2003). DMDX: a windows display program with millisecond accuracy. Behavioural Research Methods, Instruments,& Computers, 35, 116–124.

Garner, W. R. (1978). Selective attention to attributes and to stimuli. Journal of Experimental Psychology: General, 107, 287–308.CrossRef

Ghazanfar, A. A., & Logothetis, N. K. (2003). Facial expressions linked to monkey calls. Nature, 423, 937–938.CrossRefPubMed

Hommel, B. (in press). Feature integration across perception and action: Event files affect response choice. Psychological Research, Psychologische Forschung.

Kerzel, D. (2002). Evidence for effects of phonological correspondence between visible speech and written syllables. Psychological Research, 66, 195–200.CrossRefPubMed

Kerzel, D., & Bekkering, H. (2000). Motor activation from visible speech: evidence from stimulus response compatibility. Journal of Experimental Psychology. Human Perception and Performance, 26, 634–647.CrossRefPubMed

Klucharev, V., Möttönen R., & Sams, M. (2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research, 18, 65–75.CrossRefPubMed

Kuhl, P. K. (1991). Human adults and human infants show a "perceptual magnet effect" for the prototypes of speech categories, monkeys do not. Perception and Psychophysics, 50, 93–107.PubMed

Kuhl, P. K., & Meltzoff, A. N. (1982). The bimodal perception of speech in infancy. Science, 218, 1138–1141.PubMedCrossRef

Kuhl, P. K., Tsao, F. M., & Liu, H. M. (2003). Foreign-language experience in infancy: effects of short-term exposure and social interaction on phonetic learning. Proceedings of the National Academy of Sciences USA, 100, 9096–9101.CrossRef

Liberman, A. M., Cooper, F. S., Shankweiler, D. P., & Studdert-Kennedy, M. (1967). Perception of the speech code. Psychological Review, 74, 431–461.PubMedCrossRef

Massaro, D. W. (1987). Speech perception by ear and eye. In B. Dodd& R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 53–83). London, England UK: Lawrence Erlbaum Associates, Inc.

Massaro, D. W. (1998). Perceiving talking faces. Cambridge, MA: MIT Press

Massaro, D. W., Tsuzaki, M., Cohen, M., Gesi, A., & Heredia, R. (1993). Bimodal speech perception: An examination across languages. Journal of Phonetics, 21:445–478

Mattys, S. L., Bernstein, L. E., & Auer, E. T. (2002). Stimulus-based lexical distinctiveness as a general word-recognition mechanism. Perception & Psychophysics, 64, 667–679.

McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 265, 746–748.CrossRef

Mehler, J., Sebastián-Gallés, N., & Nespor, M., (2004). Biological Foundation of Language Acquisition: Evidence from Bilingualism. In M. S. Gazzaniga (Ed.), Cognitive Neurosciences III (pp. 825–836). Cambridge, MA: MIT press.

Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual–auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452–465.CrossRefPubMed

Navarra, J., Sebastián-Gallés, N., & Soto-Faraco, S. (2005). The perception of second language sounds in early bilinguals: New evidence from an implicit measure. Journal of Experimental Psychology: Human Perception and Performance, 31(5), 912.CrossRefPubMed

Pallier, C., Colomé, A., & Sebastián-Gallés, N. (2001). The influence of native-language phonology on lexical access: Exemplar-based versus abstract lexical entries. Psychological Science, 12, 445–449.CrossRefPubMed

Prinz, W. (1990). A common coding approach to perception and action. In O. Neumann & W. Prinz (Eds.), Relationships between perception and action: Current approaches (pp. 167–201). Berlin: Springer.

Reisberg, D., McLean, J., & Goldfield, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In B. Dodd & R. Campbell (Eds.). Hearing by eye: The psychology of lip-reading. Hillsdale: LEA

Sebastián-Gallés, N., & Baus, C. (2005). On the relationship between perception and production in L2 categories. In A. Cutler (Ed.) Twenty-First Century Psycholinguistics. Hillsdale: Erlbaum.

Sebastián-Gallés, N., & Soto-Faraco, S. (1999). Online processing of native and non-native phonemic contrasts in early bilinguals. Cognition, 72, 111–123.CrossRefPubMed

Sekiyama K., & Tohkura Y. (1993) Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, 427–444.

Soto-Faraco, S., Navarra, J., & Alsius, A. (2004). Assessing automaticity in audiovisual speech integration: Evidence from the speeded classification task. Cognition, 92, B13-B23.CrossRefPubMed

Soto-Faraco, S., Navarra, J., Voloumanos, A., Sebastián-Gallés, N., Weikum, & Werker, J. F. (submitted). Discriminating languages by speechreading. Perception and Psychophysics.

Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: The MIT Press.

Strange, W. (1995). Cross-language studies of speech perception. A historical review. In W. Strange (Ed.) Speech perception and linguistic experience. Issues in cross-language research (pp. 3–45). Baltimore, MD: York Press.

Sumby, W., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.CrossRef

Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd, R. Campbell (Eds.). Hearing by Eye: The Psychology of Lip-Reading (pp. 3–51). London, UK: LEA.

Tomiak, G. R., Mullennix, J. W., & Sawusch, J. R. (1987). Integral processing of phonemes: Evidence for a phonetic mode of perception. Journal of the Acoustical Society of America, 81, 755–764.CrossRefPubMed

Van Wassenhove, V., Grant, K.W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the Natural Academy of Sciences USA, 102(4), 1181–1186.CrossRef

Watkins, K. E., & Paus, T. (2004). Modulation of motor excitability during speech perception: The role of Broca’s area. Journal of Cognitive Neuroscience, 16(6), 978–987.CrossRefPubMed

Watkins, K. E., Strafella, A.P., & Paus, T. (2003). Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia, 41(8), 989–994.CrossRefPubMed

Werker, J. F., & Tees, R. C. (1983). Developmental changes across childhood in the perception of non-native speech sounds. Canadian Journal of Psychology, 37(2), 278–286.PubMed

Werker, J. F., Gilbert, J. H. V., Humphrey, K., & Tees, R.C. (1981). Developmental aspects of cross-language speech perception. Child Development, 52, 349–355.PubMedCrossRef

Wilson, S. M., Pinar Saygin, A., Sereno, M. I., & Iacobini, M. (2004). Listening to speech activates motor areas involved in speech production. Nature, 7(7), 701–702.

Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J., & McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex, 13, 1034–1043.CrossRefPubMed

Titel: Hearing lips in a second language: visual articulatory information enables the perception of second language sounds
Auteurs: Jordi Navarra
Salvador Soto-Faraco
Publicatiedatum: 01-01-2007
Uitgeverij: Springer-Verlag
Gepubliceerd in: Psychological Research / Uitgave 1/2007
Print ISSN: 0340-0727
Elektronisch ISSN: 1430-2772
DOI: https://doi.org/10.1007/s00426-005-0031-5

Bohn Stafleu van Loghum

Deel dit onderdeel of sectie (kopieer de link)

Abstract

Log in om toegang te krijgen

Andere artikelen Uitgave 1/2007

Feature binding and episodic retrieval in blindness for congruent stimuli: evidence from analyses of sequential congruency

Intentional control of attention: action planning primes action-related stimulus dimensions

Instruction-induced feature binding

Repetition costs in word identification: evaluating a stimulus–response integration account

Kinematic cues in perceptual weight judgement and their origins in box lifting

Outsourcing control to the environment: effects of stimulus/response locations on task selection