Abstract
In our everyday lives, we perceive emotional information via multiple sensory channels. This is particularly evident for emotional faces and voices in a social context. Over the past years, a multitude of studies have addressed the question of how affective cues conveyed by auditory and visual channels are integrated. Behavioral studies show that hearing and seeing emotional expressions can support and influence each other, a notion which is supported by investigations on the underlying neurobiology. Numerous electrophysiological and neuroimaging studies have identified brain regions subserving the integration of multimodal emotions and have provided new insights into the neural processing steps underlying the synergistic confluence of affective information from voice and face. In this paper we provide a comprehensive review covering current behavioral, electrophysiological and functional neuroimaging findings on the combination of emotions from the auditory and visual domains. Behavioral advantages arising from multimodal redundancy are paralleled by specific integration patterns on the neural level, from encoding in early sensory cortices to late cognitive evaluation in higher association areas. In summary, these findings indicate that bimodal emotions interact at multiple stages of the audiovisual integration process.
About the authors
Martin Klasen studied psychology at the University of Trier and obtained his Ph.D. in neuroscience from RWTH Aachen University, where he is currently working as a postdoctoral fellow. He is interested in the neural basis of social behaviour and communication in a multimodal context. His current research focuses on the multisensory combination of emotional cues as well as the processing of aggression-relevant information in violent video games.
Yu-Han Chen obtained her Ph.D. from RWTH Aachen University. She is currently a postdoctoral fellow at University of New Mexico, Department of Psychiatry, studying auditory processing abnormality in individuals with schizophrenia and bipolar disorder. Her research focuses on applying neuroimaging and electrophysiology techniques, such as magnetoencephalography (MEG), electroencephalography (EEG), and diffusion tensor imaging (DTI), to study brain structure and function.
Professor Klaus Mathiak, M.D., Ph.D., studied mathematics, medicine and neuroscience. He develops methods for neuroimaging and applies them in fMRI and MEG to study complex and interactive neural processes. As psychiatrist he is interested in the causation of complex social disorders in psychological diseases and attempts to develop new treatment strategies such as fMRI-based neurofeedback.
©2012 by Walter de Gruyter Berlin Boston
