A new method for detecting interactions between the senses in event-related potentials

Abstract

Event-related potentials (ERPs) can be used in multisensory research to determine the point in time when different senses start to interact, for example, the auditory and the visual system. For this purpose, the ERP to bimodal stimuli (AV) is often compared to the sum of the ERPs to auditory (A) and visual (V) stimuli: AV − (A + V). If the result is non-zero, this is interpreted as an indicator for multisensory interactions. Using this method, several studies have demonstrated auditory–visual interactions as early as 50 ms after stimulus onset. The subtraction requires that A, V, and AV do not contain common activity: This activity would be subtracted twice from one ERP and would, therefore, contaminate the result. In the present study, ERPs to unimodal, bimodal, and trimodal auditory, visual, and tactile stimuli (T) were recorded. We demonstrate that (T + TAV) − (TA + TV) is equivalent to AV − (A + V), but common activity is eliminated because two ERPs are subtracted from two others. With this new comparison technique, the first auditory–visual interaction starts around 80 ms after stimulus onset for the present experimental setting. It is possible to apply the new comparison method to other brain imaging techniques, as well, e.g. functional magnetic resonance imaging.

Introduction

Perception relies heavily on the integration of input from different sensory systems (Welch and Warren, 1986). Basic mechanisms of multisensory integration have been studied with event-related potentials (ERPs). In many of these studies, unimodal and bimodal stimuli were used (e.g. auditory, visual, and auditory–visual stimuli), and the ERP to the bimodal stimulus (AV) was compared to the sum of the ERPs to the unimodal stimuli (A, V): if the senses operate independently (that is, they form separate ‘mental modules’; Sternberg, 2001), the ERP to the bimodal stimulus should be equal to the sum of the ERPs to the unimodal stimuli (Barth et al., 1995): AV = A + V, or AV − (A + V) = 0. By contrast, if the ERP to the bimodal stimulus differs from the sum of the ERPs to unimodal stimuli (AV ≠ A + V), it is concluded that the senses interact (Barth et al., 1995). The time point at which the expression AV − (A + V) starts to differ from zero is thought to indicate the processing stage at which the inputs of the different sensory systems are integrated. Using this approach, several studies have demonstrated interactions of the auditory and the visual system (Fort et al., 2002, Giard and Peronnet, 1999, Molholm et al., 2002), the auditory and the somatosensory system (Foxe et al., 2000, Gobbelé et al., 2003), and the visual and the somatosensory system (Schürmann et al., 2002). In some of these studies, AV − (A + V) differed from zero as early as 50 ms after stimulus onset (e.g. Foxe et al., 2000, Giard and Peronnet, 1999, Molholm et al., 2002), which has been interpreted as evidence for multisensory interactions at early sensory processing stages.

This analysis method has recently been criticized by Teder-Sälejärvi et al. (2002): the authors emphasized that the AV − (A + V) subtraction requires that A, V, and AV do not elicit any common ERP activity (C). If such common activity exists, it is subtracted twice (AC, VC) from the bimodal ERP (AVC): AVC − (AC + VC) = −C. Therefore, the resulting term not only reflects interactions of the auditory and visual system, but is also the inverse of this common activity. Two types of common activity can be distinguished: If the auditory and the visual processing pathways converge at jointly used neural structures, this might be considered as a ‘real’ multisensory interaction. Problems arise if A, V, and AV contain unspecific common activity, e.g. activity related to the expectation of the stimulus, or motor preparation. Teder-Sälejärvi et al. demonstrated that the onset of a significant multisensory interaction in ERPs is influenced systematically by the duration of the baseline: Using a baseline correction interval of −100 ms to 0 ms before stimulus onset, AV differed from A + V starting at 60 ms after stimulus onset. By contrast, a baseline of −100 ms to −50 ms before stimulus onset moved the first significant auditory–visual interaction to 18 ms after stimulus onset. The authors suggested that these signs of multisensory interactions did not origin from multisensory processes proper but rather were due to superimposed slow waves such as the contingent negative variation (CNV, Walter et al., 1964). Since the CNV is equally present in A, V, and AV, it will affect the result of AV − (A + V) during the entire ERP interval, even before stimulus onset. Teder-Sälejärvi et al. suggested the use of a high-pass filter which eliminates the stimulus-preceding slow deflections in the ERPs to A, V, and AV. Indeed, after high-pass filtering, they found a first significant AV − (A + V) interaction at central and parietal electrodes, starting at around 160 ms after stimulus onset rather than at 50 ms as reported in earlier studies.

Nevertheless, the stimulus-preceding CNV activity is only one candidate for common activity. For example, it is plausible to assume that processes associated with the P300 (e.g. stimulus evaluation) are active during target detection. These and other processes are part of ‘C’, and only a subset of them is eliminated by the high-pass filter. Moreover, the high-pass filter might eliminate low frequency ERP components unique to A, V, or AV, which are not part of C.

The central aim of the present study is to introduce a new approach to assess auditory–visual interactions in ERPs. In a first step, the ERP to an omitted stimulus (O, ‘null-stimulus’) is added to the minuend side of the comparison: (O + AV) − (A + V). If the omitted stimulus elicited only C, it would be eliminated because two ERPs are subtracted from two others. Unfortunately, omitted stimuli may elicit rather specific ERP deflections (a prolonged CNV, and the so-called ‘missing stimulus related potential’ Busse and Woldorff, 2003, Simson et al., 1976). Therefore, in a second step, each stimulus is presented together with an additional tactile stimulus (T).

The ERP comparison is now (T + TAV) − (TA + TV). Unisensory ERP activity and common activity are eliminated in this comparison. Theoretically, the trimodal stimulus elicits additional ERP activity due to the interaction of the auditory and the visual system, the auditory and the tactile system, the visual and the tactile system, and possibly even trisensory interactions (Table 1). However, auditory–tactile and visuo-tactile interactions should be eliminated because both should be equally visible in TA and TV. Therefore, auditory–visual and—if present—trisensory interactions are isolated in the comparison.

An ERP study was run to compare the two methods. ERPs to uni-, bi-, and trimodal auditory, visual, and tactile stimuli were recorded, while participants had to make speeded responses to infrequent target stimuli. Multisensory interactions were investigated using the new comparison (T + TAV) − (TA + TV). The results were compared with those of the ‘classical’ analysis approach, AV − (A + V). A new variant of the race model test was developed to check for redundancy gains due to trisensory interactions (Gondan and Röder, in revision; for the general procedure, see Miller, 1982).