Assessment of Narrative Production
Narrative production was assessed by analysing language samples obtained during a narrative production task of the Batteria per la Valutazione del Linguaggio in bambini dai 4 ai 12 anni (BVL_4-12; Marini et al.,
2015), a standardized battery of tests designed to assess language development in children aged from 4 to 12 years old. Namely, the participants were shown a cartoon story made of six drawings presented on the same page (the “Nest Story” originally by Paradis,
1987). The experimenter told them: “
Now you will see a picture story. I don’t know this story. You must tell me it. There is no right or wrong way to tell it. You can talk a lot or a little. I only ask you not to use words such as “here” or “this”. Try to be clear.”
Administration and transcription procedures followed the criteria outlined in Marini et al., (
2011a,
2011b). Each story was audio-recorded and subsequently transcribed verbatim by three trained students who were in their final stage of the master’s degree of Cognitive Sciences of Communication at Roma Tre University and who did not know which group participants belonged to. The transcriptions included phonological fillers, pauses, false starts, and extraneous utterances. These transcriptions were manually compared to obtain highly reliable texts for analysis. Discrepancies were discussed and resolved before the narratives were analysed further.
Narrative assessment was conducted adopting a multilevel procedure including micro- and macro-linguistic variables. As for microstructure, five measures often employed in the literature (Baixauli et al.,
2016) were used: the
number of words and the
number of utterances, which were considered indicators of narrative length; the
percentage of subordinated clauses included in the narratives, which was considered as a measure of syntactic complexity; the percentage of
phonological errors, which was considered assessing the participants’ ability to retrieve phonologically well-formed words (Marini et al.,
2011a,
2011b)
; the percentage of
semantic paraphasias, which was considered as an indication of lexical-semantic processing (Andreetta et al.,
2012).
As for narrative length, we rated both the number of words and the number of utterances. As for the number of words, the total number of well-formed words was calculated for each story. To compute the number of utterances, each story was segmented into utterances, and the total number of utterances (including those containing unintelligible words) was assessed following criteria detailed in Marini et al., (
2011a,
2011b) and Ferretti et al. (
2018). Accordingly, we adopted several criteria for segmenting text into utterances: acoustic, semantic, grammatical, and phonological ones. As explained by Marini et al. (
2011, p. 1379), since “it is hardly possible to provide a coherent segmentation by adopting just one criterion”, it is desirable to jointly adopt acoustic, semantic, grammatical, and phonological criteria. According to the acoustic criterion, an utterance is an emission of phonemes delimited by pauses that can be easily identified. Let’s consider the following sequence: “
ci sono … (silent pause of 3 s) una donna e un uomo (“there are … a woman and a man”). In this case, since a clear pause can be perceived between the first chunk “
there are” and the second one “
a woman and a man,” the sequence can be segmented in two distinct utterances: /there are/a woman and a man/. According to the semantic criterion, an utterance is a conceptually homogeneous piece of information—i.e., a proposition, defined as a semantic unit consisting of the main predicate with its arguments and all embedded predicates and argument(s) associated with it. For example, the sequence “
Ci stanno un signore e una signora che stanno fissando un nido con un uccello. Poi il signore si arrampica” (“There are a man and a lady who are staring at a nest with a bird. Then the man climbs up”) can be split in two distinct utterances: /There are a man and a lady who are staring at a nest with a bird/Then the man climbs up/. According to the grammatical criterion, a set of words can be considered an utterance when, in absence of clear pauses (acoustic criterion) and of propositional violations (semantic criterion), it forms a grammatically complete sentence (eventually also including subordinate clauses). For example, the sequence “
il ragazzo decide di arrampicarsi sull’albero per prendere il nido di uccelli (“the boy decides to climb the tree to get the bird's nest”) can be considered a single utterance. However, if the speaker utters two or more coordinated sentences, such as “
il ragazzo decide di arrampicarsi sull’ albero per prendere il nido di uccelli ma il ramo si spezza e cade” (“the boy decides to climb the tree to get the bird's nest but the branch breaks and he falls”), these can be divided in three separate utterances: /the boy decides to climb the tree to get the bird's nest/but the branch breaks/and he falls/. Lastly, the phonological criterion allows dividing the utterances when there is a phonological interruption between them: an utterance is considered abruptly interrupted when it contains an interrupted word (i.e., there is a false start). For example, the sequence “
una signora e un sig- …e un ragazzo” (/ a lady and a gent- /… and a boy/) can be split in two distinct utterances. In the statistical analyses, the total number of utterances was considered.
To assess syntactic complexity, for each story the total number of subordinated clauses used by the participants in their narratives was calculated. A subordinated clause is a clause that cannot stand alone as a complete sentence but is linked to the main clause by a subordinating conjunction. For example, in the sequence il ragazzo decide di arrampicarsi sull’albero per prendere il nido di uccelli (“the boy decides to climb the tree to get the bird's nest”), the subordinated clause is per prendere il nido di uccelli (“to get the bird's nest”). Then, the percentage of subordinated clauses was calculated by dividing the total number of such clauses by the number of utterances and then multiplying by 100.
The percentage of phonological errors was calculated following the criteria described in Andreetta et al. (
2012). False starts, phonological and phonetic paraphasias and neologisms were counted as phonological errors. To compute the percentage, the total number of phonological errors was divided by the number of units (each word, non-word or syllabic false start uttered by the speaker) and then multiplying this value by 100.
The last micro-linguistic variable was that measuring the percentage of semantic paraphasias, which represent a way to evaluate children’s ability to select semantically appropriate words. Following the criteria described in Andreetta et al. (
2012), when a target word was replaced by a semantically related word a semantic paraphasia was counted. For example, in the following sequence
la mamma chiama l’ambulanza (“the mother calls an ambulance”), the word
mamma/mother was considered as a semantic paraphasia as the speaker implied
moglie/wife. Lexical-semantic processing was measured in terms of the percentage of occurrences of semantic paraphasias on the total number of content words. Higher values represent more semantic errors per word.
As for macrostructure, to assess the children’s ability to construct a global representation of the narrative, the analysis focused on the units of information produced to convey the essential story components, i.e., the core story details; the cohesive connectives linking the story events according to causal and temporal principles, i.e., first–second-third order connectives and temporal markers; and the percentage of local and global coherence errors. Moreover, the analysis also included a variable evaluating children’s ability to infer implicit events, i.e., inferred events and a last variable evaluating the children’s ability to interpret the characters’ mental states and emotions, i.e., internal states terms.
As for the
core story details, a list of measures was created adapting a semantic-pragmatic evaluation employed by Kenan et al. (
2019). In particular, the list included the following categories:
Settings The number of references to settings where the story events take place was counted for each participant. A list of these items was prepared in advance; a total of 4 settings were established: garden/house/ambulance/hospital. Synonyms were scored as correct (e.g., ‘park’ instead of ‘garden’).
Objects The number of references to concrete objects that are visible from the pictures of the story was counted for each participant (e.g., nest, bed, pillow, window).
Characters The number of references to the story characters was counted for each participant. A list was prepared in advance: 9 individual characters were present in the story (3 birds and 6 persons).
Visible events This measure focused on the children’s use of clauses to refer to concrete events that were visible from the pictures of the story, requiring the direct interpretation of the visual stimuli. An example of visible event was the following: Ci stanno un signore e una signora che stanno fissando un nido con un uccello (“There are a man and a lady who are staring at a nest with a bird”). The number of visible events generated in the children’s narratives was counted for each participant.
To assess the children’s ability to connect the story events, the use of connectives (e.g., because, and then) serving to signal the causal and temporal relations between sentences was counted. In particular, we evaluated the generation of cohesive elements used to mark different levels of discourse: (a) first-order connectives: connective elements used by children to link events that were included in the same drawing of the six frames comprising the cartoon story; this use would reflect the processing of local stimulus properties; (b) second-order connectives: connectors used by children to connect events referring to two distinct drawings of the cartoon story; this use would require the ability to relate to information conveyed in previous pictures by interpreting a variety of story details; (c) third-order connectives: connectors used by children to link two events, of which at least one was not present in the stimuli, i.e., inferred event; this use would reflect a complex integration of story details into meaningful wholes to construct a coherent representation of the narrative scene. The number of connectives, for each type, used in the children’s narratives was counted for each participant.
As for the temporal connections between events, we assessed the usage of temporal markers: the number of indicators used to signal the temporal relationships between events, e.g., soon, later, was counted for each participant.
To determine the extent to which each utterance of the story was conceptually related to the previous one, we measured local coherence. Following the criteria described in Andreetta et al. (
2012), we evaluated
local coherence errors, which included the production of words without a clear referent and topic switching. The percentage of local coherence errors was calculated by dividing the number of local coherence errors by the number of utterances and multiplying this value by 100.
To determine the extent to which each utterance of the story was conceptually related to the main topic of the story, narrative global coherence was evaluated. Also in this case, we counted the percentage of
errors of global coherence (Andreetta et al.,
2012). Errors of global coherence included the production of utterances that may be tangential (containing a derailment in the flow of discourse with respect to the information provided in previous utterances), conceptually incongruent with the story (including ideas not directly addressed by the stimulus), propositional repetitions or simple fillers. The percentage of global coherence errors was calculated by dividing the number of global coherence errors by the number of utterances and multiplying this value by 100.
The children’s ability to infer implicit events, i.e., events that were not apparent in the stimuli, from the integration of story details in a relevant and accurate fashion was assessed measuring inferred events. For example, the fourth drawing of the story depicts a broken branch, the nest with the birds on the ground, a man lying on the ground with a broken leg, and in the background three people pointing at the man. In the fifth scene, the same man has a bandaged leg and is on the stretcher about to be loaded into an ambulance. The event that must be inferred to coherently connect these scenes is that someone called an ambulance. Therefore, an utterance such as chiamarono l’ambulanza (“They called an ambulance”) was considered as inferred event. The number of inferred events generated in the children’s narratives was counted for each participant.
The children’s ability to mention the characters’ emotional and cognitive states was measured in terms of
internal states terms (ISTs): the number of unique lexical items expressing negative or positive emotions (e.g.,
sad) and mental state verbs (e.g.,
think, wonder) (Peristeri et al.,
2017), was counted for each participant.
Both micro-and macrolinguistic analyses were performed independently by two trained students (who were different from those who transcribed the stories) attending the final year of the master’s degree of Cognitive Sciences of Communication at Roma Tre University who knew the main aim of the study but did not know which group the children belonged to. Micro-and macrolinguistic analyses resulted in substantial agreement: the inter-coder reliability for macrolinguistic variables was 0.69 < r < 0.91; p < 0.001; for macrolinguistic variables was 0.48 < r < 0.90; p < 0.001 (only first-order connectives resulted low agreement r = 0.28; p = 0.011). Discrepancies were resolved through discussion by the two evaluators.
Statistical Analyses
To compare the performance of the two groups (ASD vs. TD), a series of independent t-tests on the variables related to non-word repetition task, attention skills, and cognitive assessment (i.e., scores obtained on tasks assessing digit forward and backward repetition, attention, and theory of mind) and on the micro- and macrolinguistic measures (i.e., number of words, number of utterances, percentage of subordinate clauses, number of settings, objects, characters, visible events, first–second-third-order connectives, temporal markers, percentage of local coherence errors, inferred events, ISTs) were performed. Moreover, for three variables (% of phonological errors, % of semantic paraphasias and % of global coherence errors) non-parametric Mann–Whitney statistics were performed. Bonferroni's correction for multiple comparisons was applied on the categories that included multiple variables: for the micro-linguistic variables (number of words, number of utterances, percentage of subordinate clauses, percentage of phonological errors, and percentage of semantic paraphasias) p < 0.001 was accepted; for the variables relating to core story details (settings, objects, characters, and visible events) p < 0.017 was accepted; for the narrative cohesion/coherence variables (first-order connectives, second-order connectives, third-order connectives, temporal markers, percentage of local coherence errors, percentage of global coherence errors) p < 0.008 was accepted. For inferred events and ISTs p < 0.05 was accepted.
To evaluate the potential relation between cognitive (forward digit span, backward digit span, and theory of mind) and micro- and macrolinguistic variables, a series of Pearson’s product-moment correlation analyses were performed within each group. Moreover, in the group of participants with a diagnosis of ASD the relations between severity scores and micro- and macrolinguistic variables were also analysed.