Research ReportA new method for detecting interactions between the senses in event-related potentials
Introduction
Perception relies heavily on the integration of input from different sensory systems (Welch and Warren, 1986). Basic mechanisms of multisensory integration have been studied with event-related potentials (ERPs). In many of these studies, unimodal and bimodal stimuli were used (e.g. auditory, visual, and auditory–visual stimuli), and the ERP to the bimodal stimulus (AV) was compared to the sum of the ERPs to the unimodal stimuli (A, V): if the senses operate independently (that is, they form separate ‘mental modules’; Sternberg, 2001), the ERP to the bimodal stimulus should be equal to the sum of the ERPs to the unimodal stimuli (Barth et al., 1995): AV = A + V, or AV − (A + V) = 0. By contrast, if the ERP to the bimodal stimulus differs from the sum of the ERPs to unimodal stimuli (AV ≠ A + V), it is concluded that the senses interact (Barth et al., 1995). The time point at which the expression AV − (A + V) starts to differ from zero is thought to indicate the processing stage at which the inputs of the different sensory systems are integrated. Using this approach, several studies have demonstrated interactions of the auditory and the visual system (Fort et al., 2002, Giard and Peronnet, 1999, Molholm et al., 2002), the auditory and the somatosensory system (Foxe et al., 2000, Gobbelé et al., 2003), and the visual and the somatosensory system (Schürmann et al., 2002). In some of these studies, AV − (A + V) differed from zero as early as 50 ms after stimulus onset (e.g. Foxe et al., 2000, Giard and Peronnet, 1999, Molholm et al., 2002), which has been interpreted as evidence for multisensory interactions at early sensory processing stages.
This analysis method has recently been criticized by Teder-Sälejärvi et al. (2002): the authors emphasized that the AV − (A + V) subtraction requires that A, V, and AV do not elicit any common ERP activity (C). If such common activity exists, it is subtracted twice (AC, VC) from the bimodal ERP (AVC): AVC − (AC + VC) = −C. Therefore, the resulting term not only reflects interactions of the auditory and visual system, but is also the inverse of this common activity. Two types of common activity can be distinguished: If the auditory and the visual processing pathways converge at jointly used neural structures, this might be considered as a ‘real’ multisensory interaction. Problems arise if A, V, and AV contain unspecific common activity, e.g. activity related to the expectation of the stimulus, or motor preparation. Teder-Sälejärvi et al. demonstrated that the onset of a significant multisensory interaction in ERPs is influenced systematically by the duration of the baseline: Using a baseline correction interval of −100 ms to 0 ms before stimulus onset, AV differed from A + V starting at 60 ms after stimulus onset. By contrast, a baseline of −100 ms to −50 ms before stimulus onset moved the first significant auditory–visual interaction to 18 ms after stimulus onset. The authors suggested that these signs of multisensory interactions did not origin from multisensory processes proper but rather were due to superimposed slow waves such as the contingent negative variation (CNV, Walter et al., 1964). Since the CNV is equally present in A, V, and AV, it will affect the result of AV − (A + V) during the entire ERP interval, even before stimulus onset. Teder-Sälejärvi et al. suggested the use of a high-pass filter which eliminates the stimulus-preceding slow deflections in the ERPs to A, V, and AV. Indeed, after high-pass filtering, they found a first significant AV − (A + V) interaction at central and parietal electrodes, starting at around 160 ms after stimulus onset rather than at 50 ms as reported in earlier studies.
Nevertheless, the stimulus-preceding CNV activity is only one candidate for common activity. For example, it is plausible to assume that processes associated with the P300 (e.g. stimulus evaluation) are active during target detection. These and other processes are part of ‘C’, and only a subset of them is eliminated by the high-pass filter. Moreover, the high-pass filter might eliminate low frequency ERP components unique to A, V, or AV, which are not part of C.
The central aim of the present study is to introduce a new approach to assess auditory–visual interactions in ERPs. In a first step, the ERP to an omitted stimulus (O, ‘null-stimulus’) is added to the minuend side of the comparison: (O + AV) − (A + V). If the omitted stimulus elicited only C, it would be eliminated because two ERPs are subtracted from two others. Unfortunately, omitted stimuli may elicit rather specific ERP deflections (a prolonged CNV, and the so-called ‘missing stimulus related potential’ Busse and Woldorff, 2003, Simson et al., 1976). Therefore, in a second step, each stimulus is presented together with an additional tactile stimulus (T).
The ERP comparison is now (T + TAV) − (TA + TV). Unisensory ERP activity and common activity are eliminated in this comparison. Theoretically, the trimodal stimulus elicits additional ERP activity due to the interaction of the auditory and the visual system, the auditory and the tactile system, the visual and the tactile system, and possibly even trisensory interactions (Table 1). However, auditory–tactile and visuo-tactile interactions should be eliminated because both should be equally visible in TA and TV. Therefore, auditory–visual and—if present—trisensory interactions are isolated in the comparison.
An ERP study was run to compare the two methods. ERPs to uni-, bi-, and trimodal auditory, visual, and tactile stimuli were recorded, while participants had to make speeded responses to infrequent target stimuli. Multisensory interactions were investigated using the new comparison (T + TAV) − (TA + TV). The results were compared with those of the ‘classical’ analysis approach, AV − (A + V). A new variant of the race model test was developed to check for redundancy gains due to trisensory interactions (Gondan and Röder, in revision; for the general procedure, see Miller, 1982).
Section snippets
Reaction time data
False alarms and misses were below 10% on average (i.e., less than 4 misses per target condition) and were not further analyzed. Reaction times for the seven types of target stimuli are shown in Table 2: responses to trimodal targets were fastest followed by responses to bimodal targets, and responses to unimodal targets were slowest (unimodal vs. bimodal: t18 = 12.3, P < 0.01; bimodal vs. trimodal: t18 = 7.9, P < 0.01). The reaction time gain in bimodal redundant targets (AV, TA, TV) was
Acknowledgment
This study was supported by the Emmy Noether grant Ro 1226/4-1/2/3 to BR of the German Research Foundation (DFG).
References (23)
- et al.
The spatiotemporal organization of auditory, visual, and auditory–visual evoked potentials in rat cortex
Brain Res.
(1995) - et al.
The ERP omitted stimulus response to “no-stim” events and its implications for fast-rate event-related fMRI designs
NeuroImage
(2003) - et al.
Early auditory–visual interactions in human cortex during nonredundant target identification
Cogn. Brain Res.
(2002) - et al.
Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping
Cogn. Brain Res.
(2000) - et al.
Activation of the human posterior parietal and temporoparietal cortices during audiotactile interaction
NeuroImage
(2003) Divided attention: evidence for coactivation with redundant signals
Cognit. Psychol.
(1982)- et al.
Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study
Cogn. Brain Res.
(2002) - et al.
The scalp topography of potentials associated with missing visual or auditory stimuli
Electroencephalogr. Clin. Neurophysiol.
(1976) Separate modifiability, mental modules, and the use of pure and composite measures to reveal them
Acta Psychol.
(2001)- et al.
An analysis of audio–visual crossmodal integration by means of event-related potential (ERP) recordings
Cogn. Brain Res.
(2002)
Probability inequalities for testing separate activation models of divided attention
Percept. Psychophys.
Cited by (52)
Testing trisensory interactions
2021, Journal of Mathematical PsychologyMultiple phases of cross-sensory interactions associated with the audiovisual bounce-inducing effect
2020, Biological PsychologyCitation Excerpt :ERP processing was carried out using “Scan” software (version 4.5). ERP components associated with cross-modal interaction were isolated by calculating cross-modal difference (CMdiff) waveforms, which were obtained by subtracting the summed ERPs elicited by the unimodal V and A stimuli from ERPs evoked by the bimodal VA stimuli (c.f. Giard & Peronnet, 1999; Molholm et al., 2002; Fort, Delpuech, Pernier, & Giard, 2002; Teder-Sälejärvi, McDonald, Di Russo, & Hillyard, 2002; Teder-Sälejärvi, Di Russo, McDonald, & Hillyard, 2005; Talsma & Woldorff, 2005; Gondan & Röder, 2006; Bonath et al., 2007; Talsma, Doty, & Woldorff, 2007; Mishra, Martinez, Sejnowski, & Hillyard, 2007; Mishra, Martinez, & Hillyard, 2008, 2010; Li, Wu, & Touge, 2010; Senkowski, Saint-Amour, Höfle, & Foxe, 2011; Van der Burg, Talsma, Olivers, Hickey, & Theeuwes, 2011; Yang et al., 2013; Gao et al., 2014; Zhao et al., 2018). In order to examine whether the variations of early cross-modal neural activities are responsible for the occurrence of ABE, these cross-modal difference waveforms were calculated separately for VA_bouncing trials [CMdiff_bou = VA_bou - (V + A–N)] and VA_streaming trials [CMdiff_str = VA_str - (V + A–N)].
A matter of attention: Crossmodal congruence enhances and impairs performance in a novel trimodal matching paradigm
2016, NeuropsychologiaCitation Excerpt :The mechanisms of these interactions are still far from being completely understood. One on-going challenge in the field of multisensory research is the question of how crossmodal interactions can be identified and quantified (Gondan and Röder, 2006,Stevenson et al., 2014). In many cases, crossmodal interactions have been investigated by means of redundant signal detection paradigms in which performance in unimodal trials is compared to performance in redundant multimodal trials (Diederich and Colonius, 2004).