Visual and linguistic narrative comprehension in autism spectrum disorders: Neural evidence for modality-independent impairments

Highlights

• We compared comprehension of linguistic and visual narratives in adults with ASD.

• The ASD group showed reduced N400 effects for both linguistic and visual narratives.

• Comprehension impairments were observed for both linguistic and visual narratives.

• Narrative difficulties in ASD stem from general impairments in narrative comprehension.

• ASD individuals use a bottom-up processing style during narrative comprehension.

Abstract

Individuals with autism spectrum disorders (ASD) have notable language difficulties, including with understanding narratives. However, most narrative comprehension studies have used written or spoken narratives, making it unclear whether narrative difficulties stem from language impairments or more global impairments in the kinds of general cognitive processes (such as understanding meaning and structural sequencing) that are involved in narrative comprehension. Using event-related potentials (ERPs), we directly compared semantic comprehension of linguistic narratives (short sentences) and visual narratives (comic panels) in adults with ASD and typically-developing (TD) adults. Compared to the TD group, the ASD group showed reduced N400 effects for both linguistic and visual narratives, suggesting comprehension impairments for both types of narratives and thereby implicating a more domain-general impairment. Based on these results, we propose that individuals with ASD use a more bottom-up style of processing during narrative comprehension.

Introduction

The ability to understand sequential events is thought to be unique, even fundamental, to the human condition. The comprehension of narrative sequences is central to school, work, and everyday life, but poses significant challenges for individuals with communication and neurodevelopmental disorders like autism. This study explores whether such comprehension deficits in individuals with autism are language-specific or represent global difficulties with understanding narrative sequences.

Narrative theory has a long history in philosophy, literature, and psychology, with many different specific definitions of what constitutes a narrative (e.g. Bruner, 1991, Bruner, 1997, Richardson, 2000, Walsh, 2007). For the purposes of the current study, we conceptualize “narrative” or a “story” according to several characteristics included in many of these definitions: a narrative consists of an actor or actors with goals and motivations (Bruner, 1997) and a temporal sequence of events (Bruner, 1991, Richardson, 2000) with causal connections between those events (Richardson, 2000).

Although narrative production and narrative comprehension are interrelated, the two should be considered separately, as these two functions can comprise different processes. Narrative production may include narrative comprehension at its heart, as a story must first be understood to be told/retold. But narrative production also requires other skills, such as framing the story in a coherent manner that is understandable to the listener and using appropriate morphological and syntactic structures, which distinguish it as a separate process from comprehension. Narrative production studies have occasionally tested spontaneous narrative production in which participants tell a story such as a personal experience (e.g. Losh & Capps, 2003), which involves describing experienced events within a self-generated narrative structure. Spontaneous narrative is also often elicited by asking participants to recount the plot of a wordless picture book (AbdulSabur et al., 2014, Diehl et al., 2006, Losh and Capps, 2003, Tager-Flusberg, 1995; see reviews in Berman, 2009, Liles, 1993). Still other studies have examined retell, in which participants recount the plot of a spoken narrative (Diehl et al., 2006). Note that in retell, and also in spontaneous production of wordless picture books, the original narratives (whether orally or visually presented) have a narrative structure that is then internalized and re-produced, whereas in truly spontaneous production (e.g. describing a personal experience) this narrative structure is created on-line by the storyteller. In both types of production studies, narratives are commonly analyzed in terms of the presence of linguistic features such as the use of references, temporality, and syntactic connectivity; these production abilities are used to examine cognitive, social, and linguistic development (Berman, 2009, Liles, 1993).

In contrast, “narrative comprehension” refers to the access of semantic information – how various themes, characters, and plotlines fit together – mediated by narrative structures (Cohn, 2013a, Cohn, 2013b). Narrative comprehension has been assessed by, for example, asking participants to answer comprehension questions following presentation of a written or spoken narrative (Horowitz-Kraus et al., 2016, Nuske and Bavin, 2011, Schmithorst et al., 2006, Yarkoni et al., 2008); to judge the coherence or comprehensibility of a written story (Ferstl and von Cramon, 2001, Martín-Loeches et al., 2008); to verbally recall or retell a story (Martín-Loeches et al., 2008); or to arrange sentences to create a coherent story (Jolliffe & Baron-Cohen, 2000; see Mar, 2004, for a review).

Autism spectrum disorder (ASD) is a developmental disorder defined by deficits in social communication and interaction as well as restricted and repetitive behaviors or interests (APA, 2013). In practice, ASD often presents with a wide constellation of deficits in motor, sensory, cognitive, and social domains. Deficits in language processing, although no longer a diagnostic criterion (APA, 2013), often co-occur in ASD, and can have great functional consequences for many individuals. Language deficits are particularly prominent in higher-level processes such as semantics, syntax, and narrative production and comprehension (Groen et al., 2008, Tager-Flusberg et al., 2005).

Narrative production abilities in individuals with ASD have received considerable attention; many studies report deficits in personal and/or fictional storytelling in children and adults with ASD (Diehl et al., 2006, Losh and Capps, 2003, Tager-Flusberg and Sullivan, 1995). As a recent meta-analysis summarizes, children with ASD show poorer performance on variables related to both microstructure and macrostructure of narrative production, as well as lower use of internal state language, compared to TD peers (Baixauli, Colomer, Roselló, & Miranda, 2016).

Studies of narrative comprehension in individuals with ASD, although less numerous than production studies, have also suggested deficits in children and adults with ASD, including: difficulty arranging written sentences into coherent narratives (Jolliffe & Baron-Cohen, 2000); impairments in making inferences about stories (Jolliffe and Baron-Cohen, 2000, Kaland et al., 2007, Nuske and Bavin, 2011); and atypical brain activation and connectivity when reading passages requiring inferences (Mason, Williams, Kana, Minshew, & Just, 2008). These difficulties with making inferences during comprehension, extracting the main theme of a narrative, and connecting meaningful elements of the discourse have been attributed to an inability to connect pieces of information to thematically link together linguistic elements (Jolliffe and Baron-Cohen, 2000, Vermeulen, 2015).

Since successful narrative comprehension underlies successful narrative production, impairments in narrative production in individuals with ASD could arise, in part, from impaired comprehension. However, in comparison to the number of narrative production studies, investigations of narrative comprehension abilities in individuals with ASD are relatively sparse. Our understanding of the narrative comprehension abilities of individuals with ASD is incomplete, making it difficult to fully interpret production abilities. In the current study, we seek to address this relative paucity in the literature and gain a more comprehensive picture of narrative abilities in individuals with ASD.

Previous neuroimaging studies in typically developing (TD) individuals have demonstrated that narrative production and comprehension recruit largely overlapping brain regions, including parts of the language network (e.g. left inferior frontal gyrus and temporal lobe); areas involved in the construction and maintenance of a mental representation of the story (e.g. dorsomedial prefrontal cortex, precuneus, inferior parietal lobes); and areas involved in theory of mind (ToM; the ability to understand the mental states of others; e.g. medial prefrontal cortex, temporo-parietal junction) (AbdulSabur et al., 2014, Ferstl et al., 2008, Mar, 2004). However, production and comprehension also elicit dissociable patterns of brain activation, such as a greater reliance on motor regions in narrative production (AbdulSabur et al., 2014) and a more bilateral involvement of perisylvian language areas in narrative comprehension (AbdulSabur et al., 2014, Mar, 2004). Other studies have also proposed a more bilateral or right-hemisphere dominant pattern of activity for narrative comprehension (e.g., Karunanayaka et al., 2007, Schmithorst et al., 2006), although this conjecture has not always been empirically supported (e.g., Ferstl and von Cramon, 2001, Yarkoni et al., 2008).

Several of the neural regions underlying narrative comprehension are abnormally activated or connected in ASD during language processing, which may contribute to some of the observed differences in narrative comprehension in this population. For example, consistent with evidence of impaired ToM in autism, Mason et al. (2008) observed atypical activation of the ToM network during narrative comprehension in individuals with ASD. Individuals with ASD also showed reduced connectivity between the ToM network and left-hemisphere language areas during narrative comprehension (Mason et al., 2008). This research therefore suggests a neural basis for the observed narrative comprehension impairments in individuals with ASD.

Of particular relevance to the current study, electrophysiological studies have also documented impairments in several aspects of language comprehension in individuals with ASD (Braeutigam et al., 2008, Dunn and Bates, 2005, Dunn et al., 1999, McCleery et al., 2010, Pijnacker et al., 2010, Strandburg et al., 1993). The N400 event-related potential (ERP) component, a negative-going deflection peaking approximately 400 ms (ms) after stimulus presentation, has been established as an index of semantic processing (Kutas & Federmeier, 2011). In TD individuals, N400 amplitude is reduced for semantically congruent stimuli (e.g., contextually congruent sentence-final words) relative to semantically incongruent stimuli (e.g., contextually incongruent sentence-final words; Kutas and Federmeier, 2011, Kutas and Hillyard, 1980, Lau et al., 2008). The amplitude difference between congruent and incongruent conditions is referred to here as the N400 effect. (For current purposes, the N400 is taken to reflect semantic processing and integration [Kutas and Hillyard, 1980, Lau et al., 2008] although other functional interpretations exist [e.g., Brouwer, Fitz, & Hoeks, 2012; see Kutas & Federmeier, 2011 for a broader discussion.]) The N400 has been localized to areas of the frontal and temporal cortices (Kutas & Federmeier, 2011), which aligns with neuroimaging studies of narrative comprehension. Following the N400, a later positive component (LPC, alternatively called the P600) is also sometimes observed (Pijnacker et al., 2010, Sassenhagen et al., 2014, van de Meerendonk et al., 2010). This component is characterized by a more positive-going waveform for incongruent conditions compared to congruent conditions and generally occurs between approximately 500 and 900 ms. The LPC has been linked to later linguistic reanalysis (Sassenhagen et al., 2014, van de Meerendonk et al., 2010).

Individuals with ASD often show reduced or absent N400 effects for language compared to TD individuals (Braeutigam et al., 2008, Dunn and Bates, 2005, Dunn et al., 1999, McCleery et al., 2010, Pijnacker et al., 2010, Strandburg et al., 1993), suggesting difficulty with integrating lexical meaning with semantic context to arrive at a holistic understanding. For instance, Pijnacker et al. (2010) found that incongruent sentence-final words (e.g. “Finally, the climbers reached the top of the tulip”) elicited smaller N400 effects compared to congruent sentence-final words (e.g. “Finally, the climbers reached the top of the mountain”) for individuals with ASD compared to TD individuals. In contrast, the ASD group showed a larger LPC than the TD individuals, which the authors interpreted as reflecting a later re-evaluation process triggered by less-automatic initial semantic integration. Such difficulties with semantic integration may underlie many observable language deficits in this population, including those in narrative comprehension (Jolliffe and Baron-Cohen, 2000, Kaland et al., 2007, Mason et al., 2008, Nuske and Bavin, 2011).

Notably, language has been central to previous studies of narrative comprehension in individuals with ASD, which have predominantly used written or spoken narratives. However, because individuals with ASD often experience language deficits, using verbal materials to assess comprehension may misrepresent their abilities and could drive narrative impairments. Many studies account for this possibility by matching participant groups on language measures (Jolliffe and Baron-Cohen, 2000, Nuske and Bavin, 2011) or including language ability as a covariate in statistical analyses (Kaland et al., 2007). The current work takes a different approach by exploring whether deficits in narrative comprehension are observed in a non-linguistic modality. (Here and throughout, we use the term “linguistic” to mean “involving or related to written or spoken language.”) Given their non-linguistic nature, picture sequences (i.e., visual narratives, as in comics) could bypass verbal bottlenecks, which may make them preferable for individuals with language difficulties. Indeed, the assumption that visual narratives such as wordless picture books are transparent to such populations underlies their use in narrative elicitation tasks (Diehl et al., 2006, Losh and Capps, 2003, Tager-Flusberg, 1995).

Basic, low-level visual processing abilities like discrimination appear to be relatively intact in individuals with ASD; for example, a recent meta-analysis of low-level visual processing abilities in individuals with ASD found that for many tasks (including visual search, block design, and embedded figures tasks), individuals with ASD do not differ in performance from their TD peers (Van der Hallen, Evers, Brewaeys, Noortgate, & Wagemans, 2015). Several studies have also suggested that semantic processing of non-linguistic stimuli is intact in individuals with ASD (Coderre et al., 2017, Kamio and Toichi, 2000, McCleery et al., 2010, Sahyoun et al., 2009). For example, children with ASD performed better on a picture-word priming task than on a word-word priming task, suggesting an advantage for visuo-semantic processing (Kamio & Toichi, 2000). McCleery et al. (2010) reported an N400 effect for linguistic stimuli (pairs of pictures and spoken words) in TD children but not in children with ASD, reflecting difficulties with lexico-semantic processing. However, for non-linguistic stimuli (pairs of pictures and environmental sounds), N400 effects were similar for children with ASD and TD children, suggesting intact semantic processing of non-linguistic stimuli. This work suggests a language-specific deficit in semantic processing, which could explain the observed narrative comprehension difficulties (which have previously relied solely on linguistic stimuli). Given that basic visual discrimination and visuo-semantic processing appear intact in individuals with ASD, would higher-level comprehension remain intact for narratives presented visually?

In the context of narrative theory, it has long been recognized that narratives are not restricted to the linguistic modality but can also be portrayed non-linguistically or non-verbally (Bruner, 1991, Richardson, 2000, Walsh, 2007). Discourse and narrative research has demonstrated that similar cognitive processes are at work in comprehending narratives across different modalities, particularly written, drawn, and filmed (Cohn and Kutas, 2015, Cohn and Wittenberg, 2016, Cohn et al., 2012, Gernsbacher et al., 1990, Magliano et al., 2012, Magliano et al., 2016, Magliano et al., 2013, Radvansky and Zacks, 2014). Like language, sequential image comprehension is characterized by two processing streams. First, readers must integrate semantic information across images and update a mental model of the unfolding scene (Magliano and Zacks, 2011, Zwaan and Radvansky, 1998). Second, a “narrative grammar” organizes and constrains this semantic information with categorical roles embedded in hierarchic constituent structures, similar to the syntactic structure of sentences (Cohn et al., 2012, Cohn, 2013b). While these categories are similar to traditional notions of narrative (e.g., set up, rising action, climax, resolution), such roles are not descriptors of meaning itself but are determined through interactions between the bottom-up semantic content in images and the top-down sequential context (Cohn, 2013b, Cohn, 2014). Just as meaningful information in a sentence can be independent of its syntax, meaningful information in a story is independent from the narrative structure ordering that meaning (e.g., Brewer, 1985, Cohn, 2013b). Thus, this narrative grammar is separate from semantics, just as syntactic structure is separate from – yet interfaced with – meaning in sentences (Cohn et al., 2012).

Although relatively little research examines the neurobiological basis of picture arrangement and visual narrative comprehension, some studies have indicated the importance of frontal and temporal areas (Cohn and Maher, 2015, McFie and Thompson, 1972, Nagai et al., 2007, Osaka et al., 2014): similar areas that are recruited by linguistic narratives. Lesion studies have also highlighted the role of the right hemisphere (Boone et al., 1999, Huber and Gleber, 1982, McFie and Thompson, 1972).

Further support for the similarity between visual and linguistic narratives comes from studies examining ERPs to sequential images using manipulations like those in sentence processing studies. Just as incongruent sentence-final words elicit an enhanced N400 relative to congruent words (Kutas & Hillyard, 1980), an enhanced N400 is evoked when the final panel of a visual narrative is incongruent with its preceding context (West & Holcomb, 2002). Coherent visual narratives also typically yield attenuated N400 amplitudes across ordinal positions of the sequence (Cohn et al., 2012), similar to that observed across ordinal words in sentences (van Petten & Kutas, 1991). This attenuation is not observed for scrambled or grammar-only sentences/sequences (Cohn et al., 2012, van Petten and Kutas, 1991), suggesting a “build up” of meaning. Thus, much like linguistic narratives, semantic processing in visual narratives integrates meaning across disparate information units. This growing literature suggests that visual narrative comprehension balances a narrative structure and the semantic content of individual images, largely paralleling linguistic narrative comprehension.

Visual narratives thus provide a means of assessing narrative comprehension abilities in individuals with ASD in the absence of language, one that may allow an alternative assessment of meaning-making while bypassing documented language deficits. Studies using visual narratives and sequential image comprehension in individuals with ASD (Baron-Cohen et al., 1986, Davis et al., 2007, Zalla et al., 2010) have suggested deficits in sequential image comprehension, in line with studies of narrative comprehension in the linguistic domain. For instance, children with ASD have shown impairments compared to TD children in picture arrangement for stories requiring ToM (Baron-Cohen et al., 1986), inferring missing panels from visual event sequences (Davis et al., 2007), and predicting the final image of event sequences (Zalla et al., 2010).

Altogether, these studies hint that similar deficits may appear in visual and linguistic narrative comprehension in individuals with ASD. However, these methods also have limitations. For instance, performance on picture arrangement tasks is modulated by age and experience reading comics (Nakazawa, 2016), and reconstructions often marked as “errors” to a presumed “correct” sequence may still be narratively well-formed (Cohn, 2014). Furthermore, these prior studies of visual narrative comprehension of individuals with ASD have all been performed behaviorally, whereas the addition of neuroimaging methods may be better suited to illuminating subtle differences in cognitive processing between groups (especially those that might be masked by relatively intact behavioral performance; see for example, Braeutigam et al., 2008, Dunn and Bates, 2005, Dunn et al., 1999, Strandburg et al., 1993, for reports of atypical N400 effects in individuals with ASD in the absence of group differences in behavioral performance). In the current study, we employ ERPs to investigate narrative comprehension abilities in individuals with ASD.

We examined the comprehension of linguistic narratives (written sentences) and visual narratives (comic strips) in adults with ASD and TD adults to explore whether narrative comprehension deficits in ASD also occur in non-linguistic modalities. Our stimuli were adapted from previous studies using Peanuts comic strips for visual narratives (Cohn and Paczynski, 2013, Cohn et al., 2012, Cohn et al., 2014); linguistic narratives were short sentences translated from these comic strips. Narratives ended with either a semantically congruent or incongruent word, for written narratives; or a semantically congruent or incongruent comic panel, for visual narratives (Fig. 1). Narrative comprehension was assessed using the N400 ERP component. (Participants performed a behavioral task on a subset of trials to maintain attention. However, our main outcome measure was the N400 component.)

For linguistic narratives, we expected to replicate previous findings on N400 effects to linguistic stimuli in individuals with ASD, predicting a reduced or absent N400 effect in the ASD group compared to the TD group. As interpreted by prior studies, this result would suggest impaired semantic processing for linguistic narratives and a difficulty with semantic integration in the global context of the narrative. For visual narratives, we predicted an N400 effect for the TD group (replicating the findings of West and Holcomb, 2002). However, for visual narratives in the ASD group, we proposed two alternative outcomes.

First, if individuals with ASD have intact visuo-semantic processing, as several studies have suggested, visual narrative comprehension may be unimpaired. This may suggest that narrative comprehension remains generally intact in ASD, but difficulties with linguistic narratives may be driven by language processing deficits. For instance, visual narratives may recruit a slightly different brain network for comprehension – one that is perhaps less affected by underconnectivity in the long-range networks required for language processing. A language-specific deficit in narrative comprehension would predict an asymmetrical pattern of results in the current study: similar N400 effects in the visual narratives between groups and a reduced or absent N400 effect for the ASD group compared to the TD group in linguistic narratives only.

Alternatively, visual narratives require integrating information across the sequential narrative context to identify semantic incongruity and appear to use similar cognitive and neural mechanisms as linguistic processing. If individuals with ASD have trouble with integrating contextual information, this may lead to similar impairments in narrative comprehension regardless of modality of presentation. This may suggest that visual and linguistic narrative comprehension rely on largely shared neural substrates, and impaired narrative comprehension in individuals with ASD stems from modality-independent difficulties with integrating information into a broader context. Consequently, this would predict a symmetrical pattern of results: reduced or absent N400 effect for both the linguistic and visual narratives in the ASD group compared to the TD group.