Autism Knowledge Assessments: A Closer Examination of Validity by Autism Experts

 Several autism experts identified questions wherein the research literature was mixed, and the answer indicated to be “correct” for the autism knowledge question oversimplified or did not account for new or variable results in the research literature. Some autism experts cited specific studies that refuted the “correct” answer provided on an autism knowledge questionnaire. For instance: (1) a systematic review showing that equine therapy has beneficial effects for individuals with autism suggests that it can be considered an evidence-based intervention (Srinivasan et al., 2018; #10 on McClain et al., 2020); (2) a meta-analysis examining variables with low risk of bias in randomized controlled trials suggests that early interventions for autistic children may have limited efficacy (Sandbank et al., 2020; #22 on Harrison et al., 2017a³); and (3) another meta-analysis indicating that the male-to-female ratio in autism is closer to 3:1 due to under-diagnosis of females suggests that the often-cited 4:1 ratio may not be accurate (Loomes et al., 2017; #2 on McClain et al., 2020). Autism experts also identified several autism knowledge questions for which people might hold different perspectives, such as whether autism should be classified as a communication disorder or a neurodevelopmental disability with communication difficulties (#35 on Harrison et al., 2017a), whether behavioral therapy should be categorized as effective given that many autistic individuals do not report positive experiences with behavioral therapy (#38 on Harrison et al., 2017a), and whether children who received an autism diagnosis will necessarily maintain that diagnosis in adulthood (#10 on Gillespie-Lynch et al., 2015).

Autism experts also identified several questions that were true of some autistic individuals, but not true of every individual with autism. For instance, while some autistic individuals will attend college and/or marry, many individuals with autism will not (#4 on Gillespie-Lynch et al., 2015); while some autistic individuals may need special education services at school, other individuals with autism will not (#9 on Harrison et al., 2017a); and while some children can be diagnosed with autism at two years of age, other children have more subtle autistic characteristics and may not fully meet the diagnostic criteria for autism until a later age (#22 on McClain et al., 2020). In addition, several autism experts expressed concern about questions seemingly aimed to curtail stereotypes or myths about autism, such as those indicating that individuals with autism show affection (#8 on Gillespie-Lynch et al., 2015), are not violent (#11 on Gillespie-Lynch et al., 2015), and are interested in making friends (#12 on Gillespie-Lynch et al., 2015). While these questions can combat autism-related stereotypes, these questions may also portray autistic individuals as a monolithic and homogeneous group. Both individuals with and without autism can have difficulty showing affection, can be violent, and can be disinterested in friendship. As such, autism experts were concerned that the heterogeneity of the autism spectrum was not fully represented by some autism knowledge questions.

Autism experts identified several ways that autism knowledge questions lacked clarity, potentially causing confusion about how to understand or respond to the question. For instance, several autism knowledge questions used imprecise terminology when referring to proportions of autistic individuals, such as “a lot of children with autism” (#41 on Harrison et al., 2017a) and “many children with autism” (#34 on Harrison et al., 2017a). Terms such as “everyday language” (#44 on Harrison et al., 2017a) and “proper treatment” (#7 on Gillespie-Lynch et al., 2015) are vague and could be interpreted differently across different members of the autism community. Also, terminology taken directly from the DSM-5 (American Psychiatric Association, 2013, 2022), such as “insistence on sameness” (#9 on McMahon et al., 2020), “inflexible adherence to routines” (#22 on McMahon et al., 2020), and “poorly integrated verbal and nonverbal communication” (#4 on McMahon et al., 2020), may have been too technical for a non-professional audience. In addition, although such terminology is taken from the DSM-5, phrases such as “abnormalities in eye contact and body language” (#19 on McMahon et al., 2020) and “failure of normal back-and-forth conversation” (#15 on McMahon et al., 2020) can be perceived as stigmatizing and ableist. Other questions presented ambiguity in content, such as asking whether a child with autism can follow a person’s point, without specifying the age of the child (#28 on Harrison et al., 2017a) and asking about autism prevalence rates internationally, without acknowledge that diagnosis rates may differ from true prevalence rates (#24 on McClain et al., 2020). A few questions raised concern about cultural and religious bias, including a question about special education in the United States, which may not be appropriate for international respondents (#8 on McClain et al., 2020), and a question about God causing autism, which may assess religious beliefs rather than scientific knowledge (#51 on Harrison et al., 2017a). Finally, the general format for two of the questionnaires led to some confusion across participants. For the Gillespie-Lynch et al. (2015) assessment, the 5-point Likert scale was perceived to be an unusual response format for a knowledge questionnaire. For the McMahon et al. (2020) assessment, greater clarity was needed to show that the autistic symptom in the question prompt was one of several symptoms that could be used to diagnose autism, not the singular symptom that should be used to diagnose autism.⁴