Look who's talking: The deployment of visuo-spatial attention during multisensory speech processing under noisy environmental conditions

doi:10.1016/j.neuroimage.2008.06.046

NeuroImage

Volume 43, Issue 2, 1 November 2008, Pages 379-387

https://doi.org/10.1016/j.neuroimage.2008.06.046 Get rights and content

Abstract

In a crowded scene we can effectively focus our attention on a specific speaker while largely ignoring sensory inputs from other speakers. How attended speech inputs are extracted from similar competing information has been primarily studied in the auditory domain. Here we examined the deployment of visuo-spatial attention in multiple speaker scenarios. Steady-state visual evoked potentials (SSVEP) were monitored as a real-time index of visual attention towards three competing speakers. Participants were instructed to detect a target syllable by the center speaker and ignore syllables from two flanking speakers. The study incorporated interference trials (syllables from three speakers), no-interference trials (syllable from center speaker only), and periods without speech stimulation in which static faces were presented. An enhancement of flanking speaker induced SSVEP was found 70–220 ms after sound onset over left temporal scalp during interference trials. This enhancement was negatively correlated with the behavioral performance of participants - those who showed largest enhancements had the worst speech recognition performance. Additionally, poorly performing participants exhibited enhanced flanking speaker induced SSVEP over visual scalp during periods without speech stimulation. The present study provides neurophysiologic evidence that the deployment of visuo-spatial attention to flanking speakers interferes with the recognition of multisensory speech signals under noisy environmental conditions.

Section snippets

Participants

Seventeen neurologically normal paid volunteers participated in the study. Four participants were excluded from the analysis on the basis of extensive eye movement artifacts. The remaining thirteen participants (all right handed, mean age 25.5 y, range 19–35 y, 6 females) reported normal hearing and had normal or corrected-to-normal vision. The Institutional Review Board of the Nathan Kline Institute for Psychiatric Research approved the experimental procedures, and each subject provided

Behavioral results

An analysis of variance (ANOVA) for RTs triggered to the onset of lip movements in the multisensory task showed significantly shorter RTs in the no-interference (689 ms) compared to the interference trials (728 ms; F(1,12) = 16.22, p < .002). In addition, there was a higher hit-rate (HR) in the no-interference (93%) compared to interference trials (76%; F(1,12) = 11.36, p < .007), whereas the false alarm (FA) rate did not differ significantly between no-interference (0.5%) and interference trials

Discussion

In this study we examined the effects of visou-spatial attention on multisensory speech processing in a multiple speaker scenario. By monitoring SSVEP as a real-time index for the allocation of visual attention, we observed that the deployment of attention towards the visual inputs from flanking speaker interferes with speech recognition performance. We discuss these findings in detail below.

Summary and conclusion

The present study provides, to our knowledge, the first neurophysiologic evidence for the important role of visuo-spatial attention in speech recognition during multiple speaker interference conditions. Participants who showed stronger responses to the visual inputs from flanking speakers were more distracted in speech recognition performance than those who showed weaker responses to these inputs. This raises the question about the general nature of visual information processing in multiple

Acknowledgments

This work was supported by a grant from the U.S. National Institute of Mental Health (MH65350) to Dr. J.J. Foxe. Dr. D. Senkowski received support from a NARSAD young investigator award and the German Research Foundation (SE 1859/1-1). We would like to thank Dr. Simon Kelly and Dr. Alexander Maye for discussions of the data and analysis, Marina Shpaner and Jennifer Montesi for their technical assistance, and three anonymous reviewers for their helpful comments during revision of this article.

References (46)

AlsiusA. et al.
Audiovisual integration of speech falters under high attention demands
Curr. Biol.
(2005)
BusseL. et al.
The ERP omitted stimulus response to “no-stim” events and its implications for fast-rate event-related fMRI designs
NeuroImage
(2003)
CalvertG.A. et al.
Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex
Curr. Biol.
(2000)
CarlyonR.P.
How the brain separates sounds
Trends. Cogn. Sci.
(2004)
GhazanfarA.A. et al.
Is neocortex essentially multisensory?
Trends Cogn. Sci.
(2006)
HerrmannC.S. et al.
Gamma responses and ERPs in a visual classification task
Clin. Neurophysiol.
(1999)
HertrichI. et al.
Sequential audiovisual interactions during speech perception: a whole-head MEG study
Neuropsychologia
(2007)
MullerM.M. et al.
Effects of spatial selective attention on the steady-state visual evoked potential in the 20–28 Hz range
Brain Res. Cogn. Brain Res.
(1998)
MullerN.G. et al.
The attentional field has a Mexican hat distribution
Vision Res.
(2005)
RossL.A. et al.
Impaired multisensory processing in schizophrenia: deficits in visual enhancement of speech comprehension under noisy environmental conditions
Schizophr. Res.
(2007)

Saint-AmourD. et al.

Seeing voices: high-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion

Neuropsychologia

(2007)

SenkowskiD. et al.

Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations

Neuropsychologia

(2007)

AhveninenJ. et al.

Task-modulated “what” and “where” pathways in human auditory cortex

Proc. Natl. Acad. Sci. U. S. A.

(2006)

AlsiusA. et al.

Attention to touch weakens audiovisual speech integration

Exp. Brain Res.

(2007)

AsariH. et al.

Sparse representations for the cocktail party problem

J. Neurosci.

(2006)

BeckD.M. et al.

Stimulus context modulates competition in human extrastriate cortex

Nat. Neurosci.

(2005)

BesleJ. et al.

Bimodal speech: early suppressive visual effects in human auditory cortex

Eur. J. Neurosci.

(2004)

BusseL. et al.

The spread of attention across modalities and space in a multisensory object

Proc. Natl. Acad. Sci. U. S. A.

(2005)

CallanD.E. et al.

Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information

J. Cogn. Neurosci.

(2004)

ConwayA.R. et al.

The cocktail party phenomenon revisited: the importance of working memory capacity

Psychon. Bull. Rev.

(2001)

DesimoneR. et al.

Neural mechanisms of selective visual attention

Annu. Rev. Neurosci.

(1995)

EsceraC. et al.

Involuntary attention and distractibility as evaluated with event-related brain potentials

Audiol. Neuro-otol.

(2000)

FoxeJ.J. et al.

The case for feedforward multisensory convergence during early cortical processing

NeuroReport

(2005)

Cited by (44)

Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation
2022, NeuroImage
Citation Excerpt :
Despite the fact that most of us are generally poor lip readers (Tye-Murray et al., 2007), the enhancing effects of visual speech can be dramatic, rendering mostly indecipherable vocalizations clearly audible (Ross, Saint-Amour, Leavitt, Javitt, et al., 2007; Sumby, 1954). This well-known “principle of inverse effectiveness” (Meredith & Stein, 1986; Stein et al., 1988; Stein & Meredith, 1993) holds that multisensory enhancement generally increases with the degradation of the unisensory signals and has been shown across species (Stein et al., 1993) and experimental approaches (Sumby, 1954; van de Rijt et al., 2019) (James, 2012; Ross, Saint-Amour, Leavitt, Javitt, et al., 2007; Stevenson et al., 2012). In the human brain the effect of congruent visual information can be observed at the neural level where low frequency neural activity phase locks to the temporal envelope of speech (Zion Golumbic et al., 2013) and has been shown to be enhanced in degraded auditory speech conditions (Crosse et al., 2016).
This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.
The multisensory cocktail party problem in children: Synchrony-based segregation of multiple talking faces improves in early childhood
2022, Cognition
Extraction of meaningful information from multiple talkers relies on perceptual segregation. The temporal synchrony statistics inherent in everyday audiovisual (AV) speech offer a powerful basis for perceptual segregation. We investigated the developmental emergence of synchrony-based perceptual segregation of multiple talkers in 3–7-year-old children. Children either saw four identical or four different faces articulating temporally jittered versions of the same utterance and heard the audible version of the same utterance either synchronized with one of the talkers or desynchronized with all of them. Eye tracking revealed that selective attention to the temporally synchronized talking face increased while attention to the desynchronized faces decreased with age and that attention to the talkers' mouth primarily drove responsiveness. These findings demonstrate that the temporal synchrony statistics inherent in fluent AV speech assume an increasingly greater role in perceptual segregation of the multisensory clutter created by multiple talking faces in early childhood.
The multisensory cocktail party problem in adults: Perceptual segregation of talking faces on the basis of audiovisual temporal synchrony
2021, Cognition
Social interactions often involve a cluttered multisensory scene consisting of multiple talking faces. We investigated whether audiovisual temporal synchrony can facilitate perceptual segregation of talking faces. Participants either saw four identical or four different talking faces producing temporally jittered versions of the same visible speech utterance and heard the audible version of the same speech utterance. The audible utterance was either synchronized with the visible utterance produced by one of the talking faces or not synchronized with any of them. Eye tracking indicated that participants exhibited a marked preference for the synchronized talking face, that they gazed more at the mouth than the eyes overall, that they gazed more at the eyes of an audiovisually synchronized than a desynchronized talking face, and that they gazed more at the mouth when all talking faces were audiovisually desynchronized. These findings demonstrate that audiovisual temporal synchrony plays a major role in perceptual segregation of multisensory clutter and that adults rely on differential scanning strategies of a talker's eyes and mouth to discover sources of multisensory coherence.
The interactions of multisensory integration with endogenous and exogenous attention
2016, Neuroscience and Biobehavioral Reviews
Citation Excerpt :
The superior colliculus (SC) is part of the midbrain and contains a large number of multisensory neurons that play an important role in the integration of information from the somatosensory, visual and auditory modalities (Fairhall and Macaluso, 2009; Meredith and Stein, 1996; Wallace et al., 1998). The superior temporal sulcus (STS), which is an association cortex, mediates multisensory benefits at the level of object recognition (Werner and Noppeney, 2010b), especially for biologically relevant stimuli from different modalities; such stimuli include speech (Senkowski et al., 2008), faces/voices (Ghazanfar et al., 2005), and real-life objects (Beauchamp et al., 2004; Werner and Noppeney, 2010a). Posterior parietal regions such as the superior parietal lobule (SPL) and intraparietal sulcus (IPS) can mediate behavioral multisensory facilitation effects (Molholm et al., 2006; Werner and Noppeney, 2010a) through anticipatory motor control (Krause et al., 2012b).
Stimuli from multiple sensory organs can be integrated into a coherent representation through multiple phases of multisensory processing; this phenomenon is called multisensory integration. Multisensory integration can interact with attention. Here, we propose a framework in which attention modulates multisensory processing in both endogenous (goal-driven) and exogenous (stimulus-driven) ways. Moreover, multisensory integration exerts not only bottom-up but also top-down control over attention. Specifically, we propose the following: (1) endogenous attentional selectivity acts on multiple levels of multisensory processing to determine the extent to which simultaneous stimuli from different modalities can be integrated; (2) integrated multisensory events exert top-down control on attentional capture via multisensory search templates that are stored in the brain; (3) integrated multisensory events can capture attention efficiently, even in quite complex circumstances, due to their increased salience compared to unimodal events and can thus improve search accuracy; and (4) within a multisensory object, endogenous attention can spread from one modality to another in an exogenous manner.
How prior expectations shape multisensory perception
2016, NeuroImage
The brain generates a representation of our environment by integrating signals from a common source, but segregating signals from different sources. This fMRI study investigated how the brain arbitrates between perceptual integration and segregation based on top-down congruency expectations and bottom-up stimulus-bound congruency cues.
Participants were presented audiovisual movies of phonologically congruent, incongruent or McGurk syllables that can be integrated into an illusory percept (e.g. “ti” percept for visual «ki» with auditory /pi/). They reported the syllable they perceived. Critically, we manipulated participants' top-down congruency expectations by presenting McGurk stimuli embedded in blocks of congruent or incongruent syllables.
Behaviorally, participants were more likely to fuse audiovisual signals into an illusory McGurk percept in congruent than incongruent contexts. At the neural level, the left inferior frontal sulcus (lIFS) showed increased activations for bottom-up incongruent relative to congruent inputs. Moreover, lIFS activations were increased for physically identical McGurk stimuli, when participants segregated the audiovisual signals and reported their auditory percept. Critically, this activation increase for perceptual segregation was amplified when participants expected audiovisually incongruent signals based on prior sensory experience.
Collectively, our results demonstrate that the lIFS combines top-down prior (in)congruency expectations with bottom-up (in)congruency cues to arbitrate between multisensory integration and segregation.
Early visual and auditory processing rely on modality-specific attentional resources
2013, NeuroImage
Citation Excerpt :
However our exploratory analyses links attentional modulation of brain activity to behavioral performance, which supports an active suppression mechanism (at least in the visual case): The group of participants that showed stronger reduction of visual processing during attention to auditory input performed better in the auditory task. Senkowski et al. (2008) reported a similar link between SSR amplitude modulation and behavioral performance in a multisensory attention task. In their study, participants had to attend to a frequency-tagged audiovisual speaker presentation (i.e. a flickering video of a face uttering syllables) that was flanked by two other frequency-tagged speakers.
Many everyday situations require focusing on visual or auditory information while ignoring the other modality. Previous findings suggest an attentional mechanism that operates between sensory modalities and governs such states. To date, evidence is equivocal as to whether this ‘intermodal’ attention relies on a distribution of resources either common or specific to sensory modalities. We provide new insights by investigating consequences of a shift from simultaneous (‘bimodal’) attention to vision and audition to unimodal selective attention. Concurrently presented visual and auditory stimulus streams were frequency-tagged to elicit steady-state responses (SSRs) recorded simultaneously in electro- and magnetoencephalograms (EEG/MEG). After the shift, decreased amplitudes of the SSR corresponding to the unattended sensory stream indicated reduced processing. We did not observe an amplitude increase of the SSR corresponding to the attended sensory stream. These findings are incompatible with a common-resources account. A redistribution of attentional resources between vision and audition would result in simultaneous processing gain in the attended sensory modality and reduction in the unattended sensory modality. Our results favor a modality-specific-resources account, which allows for independent modulation of early cortical processing in each sensory modality.

View all citing articles on Scopus

View full text

Look who's talking: The deployment of visuo-spatial attention during multisensory speech processing under noisy environmental conditions

Abstract

Section snippets

Participants

Behavioral results

Discussion

Summary and conclusion

Acknowledgments

Curr. Biol.

NeuroImage

Curr. Biol.

Trends. Cogn. Sci.

Trends Cogn. Sci.

Clin. Neurophysiol.

Neuropsychologia

Brain Res. Cogn. Brain Res.

Vision Res.

Schizophr. Res.

Neuropsychologia

Neuropsychologia

Task-modulated “what” and “where” pathways in human auditory cortex

Proc. Natl. Acad. Sci. U. S. A.

Attention to touch weakens audiovisual speech integration

Exp. Brain Res.

Sparse representations for the cocktail party problem

J. Neurosci.

Stimulus context modulates competition in human extrastriate cortex

Nat. Neurosci.

Bimodal speech: early suppressive visual effects in human auditory cortex

Eur. J. Neurosci.

The spread of attention across modalities and space in a multisensory object

Proc. Natl. Acad. Sci. U. S. A.

Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information

J. Cogn. Neurosci.

The cocktail party phenomenon revisited: the importance of working memory capacity

Psychon. Bull. Rev.

Neural mechanisms of selective visual attention

Annu. Rev. Neurosci.

Involuntary attention and distractibility as evaluated with event-related brain potentials

Audiol. Neuro-otol.

The case for feedforward multisensory convergence during early cortical processing

NeuroReport