Take a look at the Recent articles

Emotion processing in different media types: Realism, complexity, and immersion

Schlochtermeier LH

Cluster of Excellence “Languages of Emotion”, Freie Universität Berlin, Berlin, Germany

Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany

E-mail : l.schlochtermeier@fu-berlin.de

Pehrs C

Cluster of Excellence “Languages of Emotion”, Freie Universität Berlin, Berlin, Germany

Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany

Kappelhoff H

Cluster of Excellence “Languages of Emotion”, Freie Universität Berlin, Berlin, Germany

Department of Philosophy and Humanities, Freie Universität Berlin, Berlin, Germany

Kuchinke L

Cluster of Excellence “Languages of Emotion”, Freie Universität Berlin, Berlin, Germany

Department of Psychology, Experimental Psychology, Ruhr-Universität Bochum, Germany

Jacobs AM

Cluster of Excellence “Languages of Emotion”, Freie Universität Berlin, Berlin, Germany
Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
Dahlem Institute for Neuroimaging of Emotion, Freie Universität Berlin, Germany
Center for Cognitive Neuroscience Berlin, Germany

DOI: 10.15761/JSIN.1000109

Article
Article Info
Author Info
Figures & Data

Abstract

With the aim to improve the ecological validity when studying real life phenomena, research has increasingly been employing more complex and realistic materials, from pictorial and verbal (e.g., movies vs. pictures, narratives vs. single words), to interactive or virtual settings. This article has the objective to understand the emotional impact of these different types of media. It first summarizes neuroimaging findings on emotion processing, focusing on the development toward more realistic and complex materials. The presented literature shows that all media types, simple words or complex movies, may induce consistent emotional responses, mirrored in activations in core emotion regions. Regions related to the (embodied) simulation of another’s bodily state, and mentalizing, the cognitive representation of another’s mental state, are particularly reported in response to more complex, narrative or social materials. Other media-specific responses are described in sensory or language brain regions, while dynamic and multimodal stimuli are reported to yield behavioral advantages together with increased emotional brain responses. Finally, the article discusses the role of immersive processes for emotional engagement in different media settings. The potential to make the viewer immerse into fictitional or artificial worlds, is proposed as a crucial modulator for emotional responses in different media types, leading to the formulation of open questions and implications for future research on emotion processing.

Key words

neuroimaging, stimulus materials, emotion induction, FMRI

Introduction

Commonly used emotion induction instruments are sets of single words [1,2], sentences, vignettes [3-5], music [6], facial emotional expressions [7,8], static pictures [9], cartoons [10], movie clips [11-15], imaginative case scenarios [16], poems and narratives [17-19], or virtual environments and video games [20,21]. Emotion induction instruments usually conceptualize emotions either as basic categories or in a two-dimensional space along the axes “valence” and “arousal”. These conceptualizations are based on theories assuming the existence of biological categories of emotion, [22,23] or on theories assuming a neurophysiological core affect system constituted by valence (pleasure/displeasure) and arousal (activation/deactivation) [24].

A recent aim of affective neuroscientific research is to use more complex, dynamic and social stimulus materials to create more naturalistic settings and thus overcome the problem of studying real life phenomena in an artificial laboratory context [25-27]. Focusing on media reception, the particularly powerful emotion effects of watching narrative movies [28], or reading literature [29-31] have been emphasized. This review aims to analyze the emotional impact of different media types, focusing on the development from simple to more complex and realistic materials. It therefore illustrates similarities and dissociations of neural networks underlying emotional processing in different media types. It then discusses how the potential to make viewers or listeners dive into and immerse into artificial or virtual worlds [29,32,33] may play a crucial moderating role for emotional responses to different media types. This article focuses on review articles and neuroimaging studies to capture main regularities as well as important findings of single studies (based on a search in “pubmed” and “web of science” published after 2000). We used several keywords, i.e. emotion processing, fMRI, narratives, movies, films, virtual reality, multimodal, dynamic, complex. Rather than giving an exhaustive review of the literature, we present those studies that best illustrate a comparison between media types. As this procedure and the present state of research appear insufficient to reach final conclusions, outstanding research questions are formulated instead.

The basic question we address is how the response to affective information is moderated by the medium it is presented in. As an example consider a picture (i.e., photo of snake) from the International Affective Picture System (IAPS, [9]) and its corresponding verbal representation from the Affective Norms for English Text (ANET, [34]) (ANET: “You watch a giant snake coiled in a display case. You freeze, as the snake’s eyes move in your direction, and a red forked-tongue darts out.” Is a perceptually more realistic pictorial representation generally more engaging? What if a verbal description leads to a deeper attentive and semantic processing, engaging immersive processes like being transported into and absorbed by a story world [e.g., 30,31,35-38]. We will examine this issue below by considering verbal, pictorial, dynamic audiovisual, virtual and interactive stimulus classes, followed by a discussion on the role of immersion in the different media settings.

Emotion processing with different media types

Verbal stimuli: Words, sentences, poems and narratives

Simple words, sentences, scripts, poems or narratives have been used as stimuli to induce emotional responses. Although being perceptually non-realistic, emotional content in verbal stimuli can evoke strong and reliable emotional responses on behavioral and brain level, engaging manifold brain networks, related to language, mentalizing, reward and emotion processing [31-39]. In single word recognition, typically an emotion processing network including the anterior cingulate (ACC), orbitofrontal cortex (OFC), hippocampus (HP), and extrastriate areas is activated [e.g., 40-44,39], while amygdala (AMY) activations are usually only found for highly arousing words [45]. Research on more complex verbal materials, such as sentences, poems, and narratives suggests an involvement of reward areas [3,19] as well as interactions between mentalizing, the cognitive representation of the other’s mind (also referred to as theory of mind), and emotion processing [e.g., [46]. Activations are reported consistently in the emotional network together with regions related to mentalizing, such as temporoparietal junction (TPJ), superior temporal sulcus (STS), middle temporal gyrus (MTG) and medial prefrontal cortex (MPFC) [4,37,47,48]. Furthermore, intense emotional content in written narratives is reported to cause a joint recruitment of both emotional and language regions of the brain [48,49] and while engaging in stories both, mentalizing and premotor networks that are related to the embodied simulation of another’s actions, are activated [50]. When listening to emotional stories, in addition to emotion processing, language and mentalizing networks [51], the left anterior temporal lobe plays a key role [52]. Emotional prosody further activates the human-selective voice area and the posterior part of the STS, corresponding to the processing of the emotional content of the speakers voice [53,54]. Synchronizations between subjects in sound and speech comprehension networks (as well as embodied simulation networks) are reported to being modulated by valence and arousal when listening to language [55].

Pictorial stimuli: Pictures of scenes, static and dynamic faces

Compared to words, perceptually more realistic pictures are often reported to have behavioral advantages and evoke stronger brain responses [39,56]. The literature indicates that complexity contributes to these findings: When controlling for complexity, Schlochtermeier et al. [57] did not replicate this effect, suggesting that a picture’s perceptual complexity may account for its increased emotional brain response compared to a word. Pictorial stimuli usually depict complex sceneries or facial expression, which are presented statically or dynamically. They produce powerful responses in emotional brain regions [58], engaging processes related to the depicted content in face, object or social processing regions. Compared to photos of facial emotional expressions, photos of natural sceneries show specific activations in the lateral occipital cortex and thalamus as well as common activations in emotional networks. Faces, on the other hand, activate specific regions such as the FG (fusiform gyrus) and the STS [e.g., 59]. Increasing their complexity and realism by presenting facial expressions dynamically rather than statically, results in more pronounced activations for dynamic emotional expressions. They are reported to evoke stronger activations in regions, such as AMY, MTG, FG, STS and IFG (inferior frontal gyrus), which together with findings of faster and better emotion recognition responses, have been attributed to emotional responding being facilitated through increased facial, movement or social processing [review: 60, meta-analysis: 61].

Dynamic audiovisual stimuli: Silent film clips, audiovisual movies and 3D movies

Apart from short sequences of facial expressions, several stimulus sets taken from fictional movies have been developed with the aim to induce powerful emotional responses on behavioral and brain level [e.g., 13]. Regarding neural correlates of emotions elicited by film, Goldin et al. [62] report different emotional networks for sadness including activations in MPFC, IFG, STG, and AMY, and happiness including medial frontal gyrus (MFG), IFG, dorsolateral PFC, temporal lobes (TL), and the caudatus (CAUD). Film stimuli have been used without sound, but also, increasing the stimulus’ complexity, with music including the complete, movie composition. When comparing uni- and multimodal stimuli, several studies report emotional modulations in regions associated with cross-modal convergence in the TL for multimodal stimuli [e.g., 63,64] as well as stronger responses in limbic and paralimbic structures, for congruent facial and vocal emotion expressions [e.g., 65]. It is reported that emotional multisensory information is integrated at multiple levels, from unisensory cortices to higher order association areas [15,26], which may lead to additive emotional effects [66]. Beside emotional brain activations, networks related to simulation and mentalizing are reported during empathic engagement in audiovisual movies [67], as well as correlations of subjective valence ratings with activations in emotional and sensory simulation regions [68]. Revealing the powerful emotional effects of film stimuli, Nummenmaa et al. [69] showed that while viewing movies their valence modulated widespread synchronizations in emotion and simulation networks, and arousal modulated activation of dorsal attention networks.

Interactive settings and virtual environments

A recent plea to overcome the spectatorial perspective of participants passively watching, reading or listening to emotional or social situations, has lead to the introduction of methods that include virtual, real-life and interactive settings [20,70,71]. Several studies indicate that the inclusion of bodies, interaction, a further spatial dimension, or further modalities changes the way emotions are processed. Research on emotions in multimodal settings points to the relevance of all human senses for emotion processing [26]. Pleading for the inclusion of bodies in naturalistic emotion research, de Gelder et al. [27] reported that the observation of bodily expressions activates more brain areas compared to facial expressions, including the FG and motor-related areas. In addition, the variation of spatial proximity modulates activation in visuospatial brain regions in threat situations indicating that a spatial dimension may add to the understanding of social emotions [72]. Using video games, Mathiak et al. [21] showed that emotional as well as spatial and self-referential regions played a role in complex affect changes in a virtual environment, while active engagement in a video game influences striatal (CAUD, PUT) reward circuits [73]. Interactive compared to passive perspectives are suggested to differentially involve simulation networks [20], indicating that besides recruiting additional brain areas and facilitating emotional responding, interactive or virtually realistic settings, as compared to passive perceptive ones, might change how we understand another person’s feelings or thoughts and thus change the way we evaluate and experience emotional situations.

In all, hard and fast distinctions between the processing of emotional information in different media types are conceivably difficult. Rather, the literature suggests a complex interplay of emotional information with language and sensory properties of a stimulus (a tentative overview of processing differences is illustrated in Figure 1). While common activations in core affective regions are consistently reported, emotional information engages several specific processes related to the setting, such as speech, language, movement, object and face recognition as well as multimodal integration, mentalizing, simulation and empathizing. Such interactions of emotional responses with the medium or setting may be interpreted within constructivist accounts of emotion [16,74,75]. Latter theories propose that emotional experiences recruit situation-specific distributed modal systems of the brain, including the perception of the external environment, internal bodily sensations, and mentalizing, which may explain the media-specific involvement of sensory, sensorimotor or mentalizing processes for emotion processing. Providing a framework for the role of language in emotion processing, Kölsch et al.’s Quartet Theory [76] appears to be the first to include language in its model of the human emotion system. It proposes that language may function to communicate, evoke or regulate emotions. Beside stimulus-specific brain activations, the above-presented research suggests some processing advantages in the sense of behavioral advantages and additional and more pronounced emotional brain activations for more complex materials. If emotional information is presented in more than one modality [26] or includes movement [61], the additional information is integrated in multisensory brain regions, probably facilitating emotional responses, as shown by advantages in emotion recognition and increased connectivity with and activations in emotional brain regions [15,26,66]. In a similar vein, information gained within dynamic interaction and from an additional spatial dimension is suggested to facilitate and modulate the understanding of other people and emotional responding [20,27], while music and affective prosody in language induce sensations directly [76,77]. Beside some advantages of more complex materials, the reported studies show that more realistic stimuli as well as purely symbolic language or artificial movies, are able to elicit reliable and strong emotional responses.

Figure 1. Main differences in emotion effects between the various media types and specific emotion effects evoked by more complex media types (as described by overview articles or the majority of single studies). For clarity, regions are assigned to the networks adapted from the constructivist model by Lindquist et al. [75] and descriptions of mentalizing (MENT) and embodied simulation (ES) networks by Gallese& Guerra et al. [28], Schilbach [20] and Raz et al. [67], coded with different colors (core affect: OFC, ACC, PUT/CAUD, AMY; conceptualization: PCC, mPFC; visual: OCC, FG; language: IFG, TP, STS; mentalizing: TPJ, STS, mPFC, PCC, embodied simulation: IFG, IPL/PRECUN, INS, ACC/MCC, PM/SS (premotor/somatosensory cortex). It should be noted that this is a simplification, i.e. only key brain regions are depicted and the networks do overlap.

Along this line, several studies even suggest that especially fictional settings or artificial movies may have a high emotional impact due to their potential to let the reader simulate a story world [47] or create a sense of reality [28], by recruiting the brain’s mental and embodied simulation systems. Immersion, the concept of being absorbed by or diving into an artificial world [30,31,36-38,78-80] provides a framework for this phenomenon. Its role for emotional engagement, its relation to realism and its neural mechanisms in different stimulus categories is therefore discussed in the next section.

Immersion

First of all it should be noted that the concept of immersion is far from unified [30]. It has been defined as the psychological state of constructing a mental picture of and being transported into and absorbed by a story world [e.g., 29,36,78,79]. Immersion potential in this context refers to the potential of a medium to induce immersive processes [30,31,79]. A slightly different position, suggested by Slater [81], states that immersion should refer to the technological property of a medium to make an artificial setting seem real, whereas the psychological response should be referred to as “feeling of presence”. While still lacking a unified conceptualization, immersive processes have been reported to being highly correlated [31] and interrelated bi-directionally with emotional engagement [82]. To better understand the relation of emotional engagement, immersion and realism, Blascovichs’ social threshold model [83] may be helpful. It proposes that the degree to which an artificial, immersive setting impacts a participant cognitively or emotionally can be referred to as social influence. Social influence, here, is determined by the subjective belief that characters are real (social presence), together with the degree that a presented behavior is realistic (behavioral realism). In other words, the immersion potential of a medium, the realism of the presented behavior, and the emotionality of the content, determine the degree of emotional engagement. In this sense watching a beach ball could be comparably engaging as watching a real human being, if the setting was immersive and its behavioral aspects realistic and emotional [83]. Supporting the notion that characters do not have to be real to have an emotional impact, studies have shown that expressions of emotions in avatars are perceived and processed similar to human emotions [27,84]. It is worth noting that the immersion potential of written narratives can be high beyond objective realism [85] and was even reported to being the highest compared to film and music [36]. This issue is being tackled by an ongoing discussion on neurocognitive mechanisms underlying immersion and the sense of presence and on technological or compositional factors that may modulate the immersion potential of a medium. According to Ryan [79], the most likely subjective cause for immersion in verbal narratives is theory of mind. It is also suggested to rely on symbol grounding, the grounding of language in sensory-motor brain areas [86] or the closely related concept of neuronal re-use [87,88], the functional re-use of brain areas, originally related to face, pattern and object recognition, for language [29]. With regard to movies, it has similarly been suggested that their “reality effect” relies on embodied simulation [28]. In a similar line, presence in immersive virtual settings is proposed to rely on embodiment processes [89], which here may lead to multisensory body ownership illusions, the belief that virtual body parts are one’s own [90]. In virtual environments, sensory enrichment, visual scale or haptic feedback, have been proposed to increase the sense of presence by facilitating embodiment processes [87]. In literature, inducing fiction feelings like empathy, sympathy, identification, suspense, or vicarious fear or joy facilitate immersion [31], as evidenced by activations in mentalizing regions being enhanced by fictional rather than realistic contextual knowledge [47] and suspenseful plot [30,31,36,91]. Backgrounding elements of a story or poem, such as familiarity or situational embedding also facilitate immersion in reading [18], while in movies, compositional features may enhance immersive processes by facilitating embodiment [28]. Prototypical facial expressions or close-ups of faces, for example, may enhance facial emotional processing [60,92]. Attractive faces capture visual attention [93] and close-ups of tools engage motor simulation [92]. Bakels [94] suggests that lighting, frequency of cuts, or changes in camera angle modulate embodied simulations. To illustrate our assumption that less realistic language stimuli as well as more realistic settings have a high immersion potential, Figure 2 shows the hypothetical immersion potential of the different stimulus categories in relation to realism and complexity in a 3-D space.

Figure 2. The graph illustrates the hypothetical immersion potential of the discussed stimulus categories (and studies using the materials for emotion induction) in relation to realism and complexity in a 3-D space. It should be noted that the localization of the categories in the 3-D space is not based on conclusive empirical findings. It rather represents a heuristic hypothesis, based on the literature on immersion and illustrates our assumption that perceptually non-realistic, complex language stimuli as well as very realistic settings may have a high immersion potential.

Conclusion and outstanding questions

This article summarized recent neuroimaging findings focusing on emotion processing in more complex or realistic materials. All considered media types evoke reliable emotional responses, involving manifold brain networks. Media-specific activations are reported in sensory and socio-emotional regions related to mentalizing and embodied simulation. Additional brain activations and a modulation of socio-emotional effects are found for dynamic, multimodal and interactive stimuli. Further, aesthetically composed movies, as well as materials that rely on our imagination, such as narratives or scripts, are able to evoke strong emotional responses. We therefore propose that the immersion potential of a given stimulus material is a crucial factor determining emotional responses in different media settings. Specific technological or compositional features of a material/medium and its context control its immersion potential, e.g., by facilitating mentalizing and embodiment processes. Experimental evidence for neural correlates of the psychological state of immersion [16,37], and the significance of simulation processes for passive compared to interactive settings [20] are still scarce and inconclusive. In sum, the current literature highlights the importance of investigating the emotional impact of imaginative and passive perceptive as well as interactive or real-life settings, to advance our understanding of the human emotion system. Differences in the processing of and interactions between media types may help understand the complex interplay of symbolic language and pictures as well as of self-referential and (inter-) action-related aspects for emotion processing.

References

2021 Copyright OAT. All rights reserv
  1. Bradley MM, Lang PJ (1999) Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical Report C-1. The Center for Research in Psychophysiology, University of Florida, 1-45.
  2. Võ ML, Conrad M, Kuchinke L, Urton K, Hofmann MJ, et al. (2009) The Berlin Affective Word List Reloaded (BAWL-R). Behav Res Methods 41: 534-538. [Crossref]
  3. Bohrn IC, Altmann U, Lubrich O, Menninghaus W, Jacobs AM (2013) When we like what we know--a parametric fMRI analysis of beauty and familiarity. Brain Lang 124: 1-8. [Crossref]
  4. Hervé PY, Razafimandimby A, Jobard G, Tzourio-Mazoyer N (2013) A Shared Neural Substrate for Mentalizing and the Affective Component of Sentence Comprehension. PloS one 8: e54400. [Crossref]
  5. Sabatinelli D, Lang PJ, Bradley MM, Flaisch T (2006) The neural basis of narrative imagery: emotion and action. Prog Brain Res 156: 93-103. [Crossref]
  6. Zentner M, Grandjean D, Scherer KR (2008) Emotions evoked by the sound of music: characterization, classification, and measurement. Emotion 8: 494-521. [Crossref]
  7. Ekman P, Friesen WV (1977) Facial action coding system.
  8. Lundqvist D, Flykt A, Öhman A(1998) The Karolinska directed emotional faces (KDEF). CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, 91-630.
  9. Lang PJ, Bradley MM, Cuthbert BN (2008) International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8. University of Florida, Gainesville, FL, USA.
  10. Brink TT, Urton K, Held D, Kirilina E, Hofmann MJ, et al. (2011) The role of orbitofrontal cortex in processing empathy stories in 4- to 8-year-old children. Front Psychol 2: 80. [Crossref]
  11. Dziobek I, Fleck S, Kalbe E, Rogers K, et al. (2006) Introducing MASC: a movie for the assessment of social cognition. J Autism Dev Disord 36: 623-636. [Crossref]
  12. Gross JJ, Levenson RW (1995) Emotion elicitation using films. Cognition and Emotion 9: 87-108.
  13. Schaefer A, Nils F, Sanchez X, Philippot P (2010) Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers. Cognition and Emotion 24: 1153-1172.
  14. Schnell K, Bluschke S, Konradt B, Walter H (2011) Functional relations of empathy and mentalizing: an fMRI study on the neural basis of cognitive empathy. Neuroimage 54: 1743-1754. [Crossref]
  15. Pehrs C, Deserno L, Bakels JH, Schlochtermeier LH, Kappelhoff H, et al. (2014) How music alters a kiss: superior temporal gyrus controls fusiform-amygdalar connectivity. Soc Cogn Affect Neurosci 9: 1770-1778. [Crossref]
  16. Wilson-Mendenhall CD, Barrett LF, Barsalou LW (2013) Neural evidence that human emotions share core affective properties. Psychol Sci 24: 947-956. [Crossref]
  17. Altmann U, Bohrn IC, Lubrich O, Menninghaus W, Jacobs AM (2012) The power of emotional valence—from cognitive to affective processes in reading. Front Hum Neurosci 6: 192. [Crossref]
  18. Lüdtke, J., Meyer-Siekendieck, B., & Jacobs, A. M., (2014) Immersing in the Stillnss of an Early Morning: Testing the Mood Empathy Hypothesis of Poetry Reception. Psychology of Aestetics, Creativity, and the Arts 8: 363-377.
  19. Zeman, A., Milton, F., Smith, A., & R. Rylance. (2013) By heart: An fMRI study of brain activation by poetry and prose. Journal of Consciousness Studies 20: 132-158.
  20. Schilbach L, Timmermans B, Reddy V, Costall A, Bente G, et al. (2013) Toward a second-person neuroscience. Behav Brain Sci 36: 393-414. [Crossref]
  21. Mathiak KA, Klasen M, Zvyagintsev M, Weber R, Mathiak K (2013) Neural networks underlying affective states in a multimodal virtual environment: contributions to boredom. Front Hum Neurosci 7: 820. [Crossref]
  22. Ekman P (1992) An argument for basic emotions. Cognition and Emotion 6: 169-200.
  23. Panksepp J (1982) Toward a general psychobiological theory of emotions. Behavioral and Brain Sciences 5: 407-422
  24. Russell JA (2003) Core affect and the psychological construction of emotion. Psychol Rev 110: 145-172. [Crossref]
  25. Fischer AH, Van Kleef GA (2010) Where have all the people gone? A plea for including social interaction in emotion research. Emotion Review 2: 208-211.
  26. Klasen M, Kreifelts B, Chen YH, Seubert J, Mathiak K (2014) Neural processing of emotion in multimodal settings. Front Hum Neurosci 8: 822. [Crossref]
  27. de Gelder B, de Borst AW, Watson R (2015) The perception of emotion in body expressions. Wiley Interdiscip Rev Cogn Sci 6: 149-158. [Crossref]
  28. Gallese V, Guerra M (2012) Embodying movies: Embodied simulation and film studies. Cinema: Journal of Philosophy and the Moving Image 3: 183-210.
  29. Schrott R, Jacobs A (2011) Gehirn und Gedicht. Wie wir unsere Wirklichkeiten konstruieren. München: Carl Hanser.
  30. Jacobs AM (2015) Towards a Neurocognitive Poetics Model of literary reading, in Towards a Cognitive Neuroscience of Natural Language Use, ed. R. M. Willems (Cambridge: Cambridge University Press). 135-159.
  31. Jacobs AM (2015) Neurocognitive poetics: methods and models for investigating the neuronal and cognitive-affective bases of literature reception. Front Hum Neurosci 9: 186. [Crossref]
  32. Slater M, Wilbur S (1997) A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators and Virtual Environments 6: 603-616.
  33. Zwaan RA (1993) Aspects of Literary Comprehension: A Cognitive Approach. Amsterdam: Benjamins.
  34. Bradley MM, Lang PJ (2007) Affective Norms for English Text (ANET): Affective ratings of text and instruction manual. Technical Report D-1. University of Florida, Gainesville, FL.
  35. Gerrig RJ (1993) Experiencing Narrative Worlds. On the Psychological Activities of Reading. New Haven: Yale University Press.
  36. Hakemulder F (2013) "Travel experiences: A typology of transportation and other absorption states in relation to types of aesthetic responses,” in Wie gebannt: Aesthetische Verfahren der affektiven Bindung von Aufmerksamkeit, ed. J. Luedtke (Berlin: Freie Universitaet) 163-182.
  37. Hsu CT, Conrad M, Jacobs AM (2014) Fiction feelings in Harry Potter: Haemodynamic response in the mid-cingulate cortex correlates with immersive reading experience. Neuroreport 25: 1356–1361. [Crossref]
  38. Jacobs AM (2011) "Neurokognitive Poetik: Elemente eines Modells des literarischen Lesens (Neurocognitive poetics: elements of a model of literary reading)," in Gehirn und Gedicht: Wie wir unsere Wirklichkeiten konstruieren (Brain and Poetry: How We Construct Our Realities), ed. Schrott R, Jacobs AM (München: Carl Hanser Verlag) 492–520.
  39. Citron FM (2012) Neural correlates of written emotion word processing: a review of recent electrophysiological and hemodynamic neuroimaging studies. Brain Lang 122: 211-226. [Crossref]
  40. Hamann S, Mao H (2002) Positive and negative emotional verbal stimuli elicit activity in the left amygdala. Neuroreport 13: 15-19. [Crossref]
  41. Jacobs AM, Võ ML, Briesemeister BB, Conrad M, Hofmann MJ, et al. (2015) 10 years of BAWLing into affective and aesthetic processes in reading: what are the echoes? Front Psychol 6: 714. [Crossref]
  42. Posner J, Russell JA,Gerber A, Gorman D, Colibazzi T, et al. (2009) The neurophysiological bases of emotion: An fMRI study of the affective circumplex using emotion-denoting words. Hum Brain Mapp 30: 883-895. [Crossref]
  43. Lewis PA, Critchley HD, Rotshtein P, Dolan RJ (2007) Neural correlates of processing valence and arousal in affective words. Cereb Cortex 17: 742-748. [Crossref]
  44. Straube T, Sauer A, Miltner WH (2011) Brain activation during direct and indirect processing of positive and negative words. Behav Brain Res 222: 66-72. [Crossref]
  45. Kuchinke L, Jacobs AM, Grubich C, Võ ML, Conrad M, et al. (2005) Incidental effects of emotional valence in single word processing: an fMRI study. Neuroimage 28: 1022-1032. [Crossref]
  46. Walter H (2012) Social cognitive neuroscience of empathy: concepts, circuits, and genes. Emotion Review 4: 9-17.
  47. Altmann U, Bohrn IC, Lubrich O, Menninghaus W, Jacobs AM (2014). Fact vs fiction—how paratextual information shapes our reading processes. Soc Cogn Affect Neurosci 9: 22-29. [Crossref]
  48. Hsu CT, Jacobs AM, Altmann U, Conrad M (2015) The Magical Activation of Left Amygdala when Reading Harry Potter: An fMRI Study on How Descriptions of Supra-Natural Events Entertain and Enchant. PLoS One10: e0118179. [Crossref]
  49. Wallentin M, Nielsen AH, Vuust P, Dohn A, Roepstorff A, et al. (2011) Amygdala and heart rate variability responses from listening to emotionally intense parts of a story. Neuroimage 58: 963-973. [Crossref]
  50. Nijhof AD, Willems RM (2015) Simulating fiction: individual differences in literature comprehension revealed with FMRI. PLoS One 10: e0116492. [Crossref]
  51. Hervé PY, Razafimandimby A, Vigneau M, Mazoyer B, Tzourio-Mazoyer N (2012) Disentangling the brain networks supporting affective speech comprehension. Neuroimage 61: 1255-1267. [Crossref]
  52. Ferstl EC, von Cramon DY (2007) Time, space and emotion: fMRI reveals content-specific activation during text comprehension. Neurosci Lett 427: 159-164. [Crossref]
  53. Beaucousin V, Lacheret A, Turbelin MR, Morel M, Mazoyer B, et al. (2007) FMRI study of emotional speech comprehension. Cereb Cortex 17: 339-352. [Crossref]
  54. Schirmer A, Kotz SA (2006) Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends in Cognitive Sciences 10: 24-30.
  55. Nummenmaa L, Saarimäki H, Glerean E, Gotsopoulos A, Jääskeläinen IP, et al. (2014). Emotional speech synchronizes brains across listeners and engages large-scale dynamic brain networks. NeuroImage 102: 498-509. [Crossref]
  56. Kensinger EA, Schacter DL (2006) Processing emotional pictures and words: effects of valence and arousal. Cogn Affect Behav Neurosci 6: 110-126. [Crossref]
  57. Schlochtermeier LH, Kuchinke L, Pehrs C, Urton K, Kappelhoff H, et al. (2013) Emotional picture and word processing: an FMRI study on effects of stimulus complexity. PLoS One 8: e55619. [Crossref]
  58. Britton JC, Taylor SF, Sudheimer KD, Liberzon I (2006) Facial expressions and complex IAPS pictures: common and differential networks. Neuroimage 31: 906-919. [Crossref]
  59. Sabatinelli D, Fortune EE, Li Q, Siddiqui A, Krafft C, et al. (2011) Emotional perception: meta-analyses of face and natural scene processing. Neuroimage 54: 2524-2533. [Crossref]
  60. Arsalidou M, Morris D, Taylor MJ (2011) Converging evidence for the advantage of dynamic facial expressions. Brain Topogr 24: 149-163. [Crossref]
  61. Krumhuber EG, Kappas A, Manstead AS (2013) Effects of dynamic aspects of facial expressions: a review. Emotion Review 5: 41-46.
  62. Goldin PR, Hutcherson CA, Ochsner KN, Glover GH, Gabrieli JD, et al. (2005) The neural bases of amusement and sadness: a comparison of block contrast and subject-specific emotion intensity regression approaches. Neuroimage 27: 26-36. [Crossref]
  63. Klasen M, Chen YH, Mathiak K (2012) Multisensory emotions: perception, combination and underlying neural processes. Rev Neurosci 23: 381-392. [Crossref]
  64. Eldar E, Ganor O, Admon R, Bleich A, Hendler T (2007) Feeling the real world: limbic response to music depends on related content. Cereb Cortex 17: 2828-2840. [Crossref]
  65. Dolan RJ, Morris JS, de Gelder B (2001) Crossmodal binding of fear in voice and face. Proc Natl Acad Sci USA 98: 10006-10010. [Crossref]
  66. Zaki J (2013) Cue Integration: A Common Framework for Social Cognition and Physical Perception. Perspect Psychol Sci 8: 296-312. [Crossref]
  67. Raz G, Jacob Y, Gonen T, Winetraub Y, Flash T, et al. (2014) Cry for her or cry with her: context-dependent dissociation of two modes of cinematic empathy reflected in network cohesion dynamics. Soc Cogn Affect Neurosci 9: 30-38. [Crossref]
  68. Viinikainen M, Glerean E, Jääskeläinen IP, Kettunen J, Sams M, Nummenmaa L (2012) Nonlinear neural representation of emotional feelings elicited by dynamic naturalistic stimulation. Open Journal of Neuroscience 2-4.
  69. Nummenmaa L, Glerean E, Viinikainen M, Jääskeläinen IP, Hari R, et al. (2012) Emotions promote social interaction by synchronizing brain activity across individuals. Proceedings of the National Academy of Sciences109: 9599-9604.
  70. de Gelder B, Hortensius R (2014) The many faces of the emotional body. New Frontiers in Social Neuroscience. Research and Perspectives in Neurosciences153-164.
  71. Hasson U, Honey CJ (2012) Future trends in Neuroimaging: Neural processes as expressed within real-life contexts. Neuroimage 62: 1272-1278. [Crossref]
  72. Lloyd DM, Morrison CI (2008) 'Eavesdropping' on social interactions biases threat perception in visuospatial pathways. Neuropsychologia 46: 95-101. [Crossref]
  73. Kätsyri J, Hari R, Ravaja N, Nummenmaa L (2013) Just watching the game ain't enough: striatal fMRI reward responses to successes and failures in a video game during active and vicarious playing. Front Hum Neurosci 7: 278. [Crossref]
  74. Kober H, Barrett LF, Joseph J, Bliss-Moreau E, Lindquist K, et al. (2008) Functional grouping and cortical–subcortical interactions in emotion: a meta-analysis of neuroimaging studies. Neuroimage 42: 998-1031. [Crossref]
  75. Lindquist KA, Wager TD, Kober H, Bliss-Moreau E, Barrett LF (2012) The brain basis of emotion: a meta-analytic review. Behav Brain Sci 35: 121-143. [Crossref]
  76. Koelsch S, Jacobs AM, Menninghaus W, Liebal K, Klann-Delius G, et al. (2015) The quartet theory of human emotions: An integrative and neurofunctional model. Phys Life Rev 13: 1-27. [Crossref]
  77. Koelsch S (2014) Brain correlates of music-evoked emotions. Nat Rev Neurosci 15: 170-180. [Crossref]
  78. Appel M, Richter T (2010) Transportation and need for affect in narrative persuasion: A mediated moderation model. Media Psychology 13: 101-135.
  79. Ryan M (2001) Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Baltimore: Johns Hopkins UP.
  80. Visch VT, Tan ES, Molenaar D (2010) The emotional and cognitive effect of immersion in film viewing. Cognition and Emotion 24: 1439-1445.
  81. Slater M (2003) A note on presence terminology. Presence Connect 3: 1-5.
  82. Riva G, Mantovani F, Capideville CS, Preziosa A, Morganti F, et al. (2007) Affective interactions using virtual reality: the link between presence and emotions. Cyberpsychol Behav 10: 45-56. [Crossref]
  83. Blascovich J (2002) ‘Social influence within immersive virtual environments in The social life of avatars, ed. Schroeder, R. (London: Springer). 127-145.
  84. Moser E, Derntl B, Robinson S, Fink B, Gur RC, et al. (2007) Amygdala activation at 3T in response to human and avatar facial expressions of emotions.J Neurosci Methods 161: 126-133. [Crossref]
  85. Green MC (2004) Transportation into narrative worlds: The role of prior knowledge and perceived realism. Discourse Processes 38: 247-266.
  86. Barsalou L (2003) Situated simulation in the human conceptual system. Language and Cognitive Processes 18: 513-562.
  87. Anderson ML (2010) Neural reuse: a fundamental organizational principle of the brain. Behav Brain Sci 33: 245-266. [Crossref]
  88. Ponz A, Montant M, Liegeois-Chauvel C, Silva C, Braun M, et al. (2014) Emotion processing in words: a test of the neural re-use hypothesis using surface and intracranial EEG. Soc Cogn Affect Neurosci 9: 619-627. [Crossref]
  89. Costa MR, Kim SY, Biocca F (2013) Embodiment and Embodied Cognition. In Virtual Augmented and Mixed Reality. Designing and Developing Augmented and Virtual Environments. ed. Schumaker R (Springer Berlin Heidelberg) 333-342.
  90. Slater M, Sanchez-Vives MV (2014) Transcending the self in immersive virtual reality. Computer 47: 24-30.
  91. Lehne M, Engel P, Rohrmeier M, Menninghaus W, Jacobs AM, et al. (2015) Reading a suspenseful literary text activates brain areas related to social cognition and predictive inference. PLoS One 10: e0124550. [Crossref]
  92. Hasson U, Nir Y, Levy I, Fuhrmann G, Malach R (2004) Intersubject synchronization of cortical activity during natural vision. Science 303: 1634-1640.
  93. Leder H, Tinio PP, Fuchs IM, Bohrn I (2010) When attractiveness demands longer looks: The effects of situation and gender. Q J Exp Psychol (Hove) 63: 1858-1871. [Crossref]
  94. Bakels JH (2014) “Embodying audio-visual media. Concepts and transdisciplinary perspectives,“ in Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, ed. Müller C, Cienki A, Fricke E, Cienki A, Ladewig SH, McNeill D, Teßendorf S (Berlin: De Gruyter Mouton)  2048-2061.

Editorial Information

Editor-in-Chief

George Perry
The University of Texas at San Antonio

Article Type

Review Article

Publication history

Received: September 22, 2015
Accepted: October 07, 2015
Published: October 10, 2015

Copyright

©2015 Schlochtermeier LH. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Citation

Schlochtermeier LH, Pehrs C, Kappelhoff H, Kuchinke L, Jacobs AM (2015). Emotion processing in different media types: Realism, complexity, and immersion. J Syst Integr Neurosci 1: doi: 10.15761/JSIN.1000109

Corresponding author

Lorna Schlochtermeier

Freie Universität Berlin, Habelschwerdter Allee 45, 14195 Berlin, Germany, Tel: +49 (0)176 64760649, Fax: +49 (0)30 838 55620.

E-mail : l.schlochtermeier@fu-berlin.de

Figure 1. Main differences in emotion effects between the various media types and specific emotion effects evoked by more complex media types (as described by overview articles or the majority of single studies). For clarity, regions are assigned to the networks adapted from the constructivist model by Lindquist et al. [75] and descriptions of mentalizing (MENT) and embodied simulation (ES) networks by Gallese& Guerra et al. [28], Schilbach [20] and Raz et al. [67], coded with different colors (core affect: OFC, ACC, PUT/CAUD, AMY; conceptualization: PCC, mPFC; visual: OCC, FG; language: IFG, TP, STS; mentalizing: TPJ, STS, mPFC, PCC, embodied simulation: IFG, IPL/PRECUN, INS, ACC/MCC, PM/SS (premotor/somatosensory cortex). It should be noted that this is a simplification, i.e. only key brain regions are depicted and the networks do overlap.

Figure 2. The graph illustrates the hypothetical immersion potential of the discussed stimulus categories (and studies using the materials for emotion induction) in relation to realism and complexity in a 3-D space. It should be noted that the localization of the categories in the 3-D space is not based on conclusive empirical findings. It rather represents a heuristic hypothesis, based on the literature on immersion and illustrates our assumption that perceptually non-realistic, complex language stimuli as well as very realistic settings may have a high immersion potential.