Integration of auditory and visual information in the recognition of realistic objects

Abstract

Recognizing a natural object requires one to pool information from various sensory modalities, and to ignore information from competing objects. That the same semantic knowledge can be accessed through different modalities makes it possible to explore the retrieval of supramodal object concepts. Here, object-recognition processes were investigated by manipulating the relationships between sensory modalities, specifically, semantic content, and spatial alignment between auditory and visual information. Experiments were run under realistic virtual environment. Participants were asked to react as fast as possible to a target object presented in the visual and/or the auditory modality and to inhibit a distractor object (go/no-go task). Spatial alignment had no effect on object-recognition time. The only spatial effect observed was a stimulus–response compatibility between the auditory stimulus and the hand position. Reaction times were significantly shorter for semantically congruent bimodal stimuli than would be predicted by independent processing of information about the auditory and visual targets. Interestingly, this bimodal facilitation effect was twice as large as found in previous studies that also used information-rich stimuli. An interference effect was observed (i.e. longer reaction times to semantically incongruent stimuli than to the corresponding unimodal stimulus) only when the distractor was auditory. When the distractor was visual, the semantic incongruence did not interfere with object recognition. Our results show that immersive displays with large visual stimuli may provide large multimodal integration effects, and reveal a possible asymmetry in the attentional filtering of irrelevant auditory and visual information.

Appendix

An additional experiment was performed because we could not conclude from the results of our main experiment whether shorter RTs to semantically congruent stimuli than to semantically incongruent stimuli were due to semantic congruence or simply to redundancy of information. In this new experiment, the target stimuli were the sound of a frog (A+_f) and the image of a phone (V+_p). Participants had to respond to A+_f, V+_p, or when both were presented simultaneously (A+_fV+_p). In this case, the redundant target condition was also a semantically incongruent stimulus, whereas the non-redundant target condition was a semantically congruent stimulus. If the RSE observed in the main experiment was related to the semantic congruence between the auditory and the visual parts of the stimulus, there should be no bimodal integration for the incongruent stimuli (redundant targets) in the present control experiment. In addition, if semantically congruent trials benefited from crossmodal integration, mean RTs in semantically congruent trials (non-redundant target) should be shorter than mean RTs in semantically incongruent trials (redundant targets).

Eleven volunteers (5 women; mean age 30.9 ± 8 years; all but one right-handed) participated in the experiment. All were naïve with respect to the purpose of the experiment. None of them reported having hearing problems, and all reported normal or corrected-to-normal vision. All participants provided informed consent to participate in the study. Apparatus and stimuli were exactly the same as in the main experiment. Procedure was also highly similar, except for the definition of the go and no-go conditions. There were five go conditions: auditory frog alone (A+_f), visual phone alone (V+_p), auditory frog with a visual phone (A+_fV+_p), auditory frog with a visual frog (A+_fV−_f), and auditory phone with a visual phone (A−_pV+_p). The A+_fV+_p condition was the only redundant target; the other four conditions were non-redundant targets. The no-go conditions were an auditory phone alone (A−_p), a visual frog alone (V−_f) and an auditory phone with a visual frog (A−_pV−_f). Each go condition was presented 48 times and each no-go condition was presented 20 times. In this additional experiment, there are no inter-stimulus contingencies. Thus, in the case of a potential RSE, this would not be due to the contingencies. The entire experiment for each participant consisted of 300 stimuli of which 240 (80%) were task-relevant stimuli (go responses). Statistical analyses were similar as the ones performed in the main experiment (log-transformation and ANOVA on the mean ln(RTs)), except that we did not remove RTs greater than 1,000 ms, due to the difficulty of the task that lead to RTs of the order of 650 ms on average.

Nonparametric repeated-measures ANOVA (Friedman’s test) revealed a significant effect of condition (A−, V−, A−V−_i) on percentage of false alarms (χ ²(2) = 9.5; P < 0.01). Percentage of false alarms was higher with a bimodal stimulus A−_pV−_f (39.5 ± 6.7%) than with a unimodal one (21.4 ± 4.4% for A−_p and 21.8 ± 4.9% for V−_f). Only 0.9 ± 0.1% of misses were observed. Overall, the larger number of false alarms here in comparison to the main experiment (around three times more) could reveal the difficulty of the task (attend to two different objects at the same time).

RTs of this additional experiment are represented in Fig. 5. The distribution of the residuals of the ANOVA was not different from a normal distribution (Kolmogorov–Smirnov test: d = 0.09; N = 55; P > 0.2). Overall, RTs observed in this additional experiment were much longer than those of the main experiment (more than 600 ms here, compared to around 350 ms in the main experiment). This confirms the difficulty of the task. To identify between-condition differences in mean ln(RTs), a repeated-measures ANOVA was conducted with the five conditions as a within-subjects factor (A+_f, V+_p, A+_fV+_p, A+_fV−_f, A−_pV+_p). It revealed a significant main effect of condition (F _4,60 = 6.14; ε = 0.8; P < 0.001). Post hoc Tukey HSD tests revealed that this effect was due to a difference between A+_f and A+_fV+_p (P < 0.001) and a difference between A+_fV+_p and A+_fV−_f (P < 0.004). Importantly, there was no significant difference between the shortest of the unimodal conditions (here, V+_p) and the redundant target (A+_fV+_p) (P = 0.5). In other words, we observed no bimodal facilitation effect for redundant target (semantically incongruent stimulus). It is of course difficult to interpret unambiguously a null effect; however, it strongly suggests that semantic incongruence of redundant stimuli prevents any redundant facilitation effect.

Fig. 5

RTs of the semantically incongruent bimodal (redundant target, A+_fV+_p), semantically congruent bimodal (non-redundant target, A+_fV−_f and A−_pV+_p), and unimodal (A+_f and V+_p) conditions are presented. RTs were first transformed to a log scale and then averaged across all participants. The error bars represent one standard error of the mean. The log scale is converted back to ms for displays purposes. There was no bimodal facilitation effect (RTs to the A+_fV+_p condition are similar to RTs to the shortest unimodal condition, i.e. V+_p). RTs to the semantically congruent conditions (A+_fV−_f and A−_pV+_p) were not shorter than RTs to the semantically incongruent condition (A+_fV+_p). The only significant differences observed are due to shorter RTs to the visual target alone compared to the auditory target alone