Agreement in multi-informant assessment of behaviour and emotional problems and social functioning in adolescents with Autistic and Asperger's Disorder
Highlights
► Adolescents and parents generally had moderate levels of agreement on ratings of behaviour and emotional problems and social functioning. ► Little agreement was found between adolescent and teacher ratings of behaviour and emotional problems and social functioning, whereas agreement between parent and teacher ratings was variable. ► Patterns of agreement between multiple informants frequently varied to those previously observed in non-ASD populations.
Introduction
The ability of informants to provide a meaningful report of phenomena is limited by their capacity to observe individuals in specific settings (Fagan & Fantuzzo, 1999) and certain behaviours may be absent or evident with varying extent depending on the environment in which they occur (Renk & Phares, 2004). Different informants are also likely to colour their observations according to different biases, expectations and standards (Achenbach & Rescorla, 2001). Multi-informant assessment can capture the observations of numerous sources to provide a comprehensive and balanced assessment of an individual's behaviour, experiences and functioning (Macintosh & Dissanayake, 2006). Despite this advantage, few studies have examined the properties and characteristics of multi-informant assessment in populations with Autism Spectrum Disorders (ASD). This includes broad measures of behaviour and emotional problems and social functioning that are routinely used in research and clinical practice. Such information is valuable for those seeking to fully and accurately assess the wide variety of behavioural and emotional difficulties young people with ASD experience and the qualities of their social function (Matson and Nebel-Schwalm, 2007, Simonoff et al., 2008).
Consistent evidence of the discrepancy between self, parent and teacher report of behaviour and emotional problems, in children and adolescents from various populations, suggests that multi-informant assessment is preferable to obtaining information from a single informant (Bird et al., 1992, De Los Reyes and Kazdin, 2005). For example, children and adolescents from clinical, paediatric hospital and typically developing samples tend to report having more behaviour and emotional problems than other informants, whilst parents report more problems than teachers (Clay et al., 2008, Montano et al., 2011, Sawyer et al., 1992, van der Ende and Verhulst, 2005, van der Ende et al., 2011). Correlations between different informants are also generally low as reported by Achenbach, McConaughy and Howell's (1987) meta-analysis of 119 studies which found a mean correlation of 0.28 between informant ratings in young people. However, agreement varies depending on the rater dyads being compared. Ratings from informants who have witnessed behaviour in the same context (such as fathers and mothers) tend to produce moderate agreement whereas lower agreement is found between raters across contexts (Achenbach et al., 1987, Duhig et al., 2000, van der Ende et al., 2011). Low to moderate correlations between parent and teacher ratings have been found using a range of instruments in young people from general, clinical and intellectually disabled populations (Aman et al., 1996, Clay et al., 2008, Dekker et al., 2002, Freund and Reiss, 1991, Salbach-Andrae et al., 2009b, Stone et al., 2010). Weak to moderate correlations between self and parent and self and teachers ratings have been observed when assessing behaviour and emotional problems in adolescents with and without Axis I disorders (e.g. substance, mood, personality and conduct disorders) (Salbach-Andrae et al., 2009a, Salbach-Andrae et al., 2009b, van der Ende and Verhulst, 2005). Agreement between self and teacher ratings are generally lower than those between self and parents ratings (Salbach-Andrae et al., 2009b, van der Ende and Verhulst, 2005).
Preliminary evidence specific to the agreement of multi-informant ratings in young people with ASD is also available. Hurtig et al. (2009) assessed self, parent and teacher reported behaviour and emotional problems in 46 adolescents with Autistic or Asperger's Disorder, aged 11–17 years. Use of the Youth Self Report (YSR), Teacher Report Form (TRF) and parent-rated Child Behaviour Checklist (CBCL) allowed comparison of ratings across 11 broad and specific problem scales. Weak to moderate correlations were found between self and parent ratings with nine of the scales correlating significantly (r = .27–.51). Some agreement was also found between self and teacher ratings, and parent and teacher ratings, with significant correlations found on seven (r = .36–.66) and five scales (r = .39–.58), respectively. Agreement within both dyads tended to occur on scales concerning externalised behaviour problems. Self and parent agreement was also examined in typically developing controls where significant correlation was found on all but one scale. However, correlations were often weaker than self-parent correlations in the ASD sample. Kanne, Abbacchi, and Constantino (2009) compared levels of agreement between parent and teacher ratings of 177 children and adolescents aged 3–18 years with ASD to that found in their typically developing siblings (n = 148) who were aged 1–16 years. Significant, moderate to strong correlations (r = .37–.73) between parent and teachers ratings were observed on six of seven DSM-oriented subscales in the sibling sample. In contrast, only four subscales in the ASD sample correlated significantly, at a low to moderate level (r = .26–.49). Such findings suggest that there is more variability in inter-rater agreement in ASD compared to typically developing or intellectually disabled populations. Yet, Hurtig et al. findings suggest inter-rater agreement is roughly comparable and, in some instances, higher in young people with ASD compared to typically developing individuals. Agreement may vary according to the measures used and specific problems examined.
Multi-informant assessment of social functioning is also preferable given further evidence of low agreement between self, parent and teacher report (Kalyva, 2010). Renk and Phares (2004) conducted a meta-analysis of 74 studies that assessed multi-informant agreement of social competence in children and adolescents. Typically developing, clinical and physically and developmentally disabled samples were included. Low to moderate mean inter-rater correlations were found; self and parent ratings (r = .21) had the lowest correspondence, while parent–teacher agreement (r = .38) was significantly higher than that of self-teacher agreement (r = .25). Other examinations of children and adolescents from the general population have found weak to moderate correlations between self, parent and teacher ratings of specific areas of social skills (r = .02–.39) such as cooperation, empathy and engagement (Gresham, Cook, Vance, Elliot, & Kettler, 2010). Assessment of social functioning in young people with intellectual disability is limited to parents and teachers given the difficulty of children and adolescents with ID to provide self report (Bielecki & Swender, 2004). Low correlations have generally been found between parent and teacher ratings on scales of social competence and social relating problems (Aman et al., 1996, Dekker et al., 2002, Tasse and Lecavalier, 2000). Evidence of low agreement in these populations may reflect variability in social behaviour across contexts or perhaps the differing perspectives of informants (Renk & Phares, 2004).
Although social function impairments represent a core component of ASD diagnoses, there is a lack of evidence concerning multi-rated and cross-contextual social functioning in young people with ASDs (Murray, Ruble, Willis, & Molloy, 2009). Available data are inconsistent with inter-rater agreement appearing to vary according to the instruments and samples examined. Murray et al. (2009) used the TRIAD Social Skills Assessment to examine 45 children and adolescents aged 5–14 years with Autistic Disorder, Asperger's Disorder or Pervasive Developmental Disorder-Not Otherwise Specified, who were referred for social skills intervention. Correlations between parent and teacher ratings of positive social behaviour ranged from low to weak and statistically non-significant (r = .15–.38). Macintosh and Dissanayake (2006) found no significant correlations between parent and teacher rated total and subscale social skill scores (r = −.28 to .20) on the Social Skill Ratings System (SSRS). Their samples consisted of 20 children aged 4–10 years with Autistic Disorder and 17 children aged 5–10 years with Asperger's Disorder. The Assertion subscale in the Asperger's Disorder sample (r = .60) was an exception, demonstrating good inter-rater agreement between parents and teachers. In contrast, Kalyva (2010) reported moderate agreement between parent and teacher (r = .51–.59) ratings on the Matson Evaluation of Social Skills with Youngsters in 21 children and adolescents with Asperger's Disorder, aged 9–16 years.
Further inconsistency is evident when the self-ratings of adolescents with Autism Spectrum Disorders are examined. Kalyva (2010) reported strong agreement between self and teacher ratings (r = .79), whereas self-father agreement (r = .67) was moderate and self-mother agreement (r = .28) was not significant. However, Vickerstaff, Heriot, Wong, Lopes, and Dossetor (2007) found no significant correlation between the total SSRS scores of children, parents and teachers when examining 22 young people with Autistic Disorder, Asperger's Disorder and Atypical Autism, aged 7–13 years. Children rated their social functioning as better than did their parents and teachers, whereas parent and teacher ratings did not significantly differ. Similar results were found by Koning and Magill-Evans (2001) who compared self, parent and teacher SSRS report in 21 adolescent males aged 12–15 years with Asperger's Disorder. Adolescents also rated themselves as more socially skilled than did their parents or teachers and correlations between all informants were weak (r = −.01 to −.10). Thus, some evidence suggests poor agreement occurs in ratings of social functioning in young people with ASD, with a level of correspondence below that reported in non-ASD populations.
Together, these findings demonstrate that agreement between multi-informant ratings of behaviour and emotional problems and social functioning in young people from general community, intellectually disabled, clinical and ASD populations are often low. However, some instances of moderate agreement have also been recorded. Few studies have examined agreement between multiple informants in young people with ASD populations. Hurtig et al. (2009) is the only published study in which self, parent and teacher ratings on broad measures of behaviour and emotional problems have been examined in adolescents with ASD. Furthermore, only three studies have compared self, parent and teacher ratings of social functioning in young people simultaneously (Kalyva, 2010, Koning and Magill-Evans, 2001, Vickerstaff et al., 2007). However, the low mean age and age range of their sample limits the ability to generalise findings to older adolescents with ASD. In view of these limitations and the paucity of available evidence, this study aimed to examine ratings (by parents, teachers and adolescents) of behaviour and emotional problems and social skills in a sample of adolescents with ASD.
Section snippets
Participants
Participants were 45 adolescents who were diagnosed with either Autistic or Asperger's Disorder. Demographic details are presented in Table 1. Participants were recruited from schools, support services, and support groups from Melbourne and greater Victoria. Families who had participated in previous research at the Monash University Centre for Developmental Psychiatry and Psychology and had indicated that they were interested in being contacted about future research projects, were mailed
Results
The mean score and standard deviations of YSR, TRF and CBCL ratings are presented in Table 2 whilst correlations are reported in Table 3. Moderate to strong correlations were found between all broad and subscale YSR and CBCL scores. No significant correlations were found between YSR and TRF broad or subscales scores, with the exception of ratings on the Rule-breaking Behaviour scale. Low to moderate but significant correlation was found between the majority of CBCL and TRF scale scores, with
Discussion
The results from the current study suggest that self, parent and teacher ratings of behaviour and emotional problems and social functioning, in adolescents with Autistic and Asperger's Disorder, are frequently discrepant. However, different patterns of agreement are apparent between the various dyads. Adolescents and parents generally had moderate–strong levels of agreement when reporting upon behaviour and emotional problems and social functioning. Such agreement in the current study suggests
References (58)
- et al.
The Nisonger CBRF: A child behavior rating form for children with developmental disabilities
Research in Developmental Disabilities
(1996) - et al.
Aggregating data from multiple informants in child psychiatry epidemiological research
Journal of the American Academy of Child & Adolescent Psychiatry
(1992) - et al.
The Social Communication Questionnaire (SCQ) as a screener for Autism Spectrum Disorders: Additional evidence and cross-cultural validty
Journal of the American Academy of Child & Adolescent Psychiatry
(2008) - et al.
Validation of the Social Communication Questionnaire in a population cohort of children with Autism Spectrum Disorders
Journal of the American Academy of Child & Adolescent Psychiatry
(2007) - et al.
Multirater congruence on the Social Skills Rating System: Mother, father and teacher assessments of urban head start children's social competencies
Early Childhood Research Quarterly
(1999) - et al.
Rating problem behaviors in outpatients with mental retardation: Use of the Aberrant Behavior Checklist
Research in Developmental Disabilities
(1991) Interpreting change on the WAIS-III/WMS-III in clinical samples
Archives of Clinical Neuropsychology
(2001)- et al.
Comorbid psychopathology with Autism Spectrum Disorder in children: An overview
Research in Developmental Disabilities
(2007) - et al.
Cross-informant ratings of social competence in children and adolescents
Clinical Psychology Review
(2004) - et al.
Patterns of agreement among parent, teacher and youth ratings in a referred sample
European Psychiatry
(2009)