The Brief Observation of Symptoms of Autism (BOSA): Development of a New Adapted Assessment Measure for Remote Telehealth Administration Through COVID-19 and Beyond

The COVID-19 pandemic sparked heightened interest in flexible screening and assessment measures for autism spectrum disorder (ASD) that can be conducted remotely via telehealth without requiring face-to-face administration by a clinician. While some attention toward telehealth or technology-based evaluation had begun prior to the pandemic (Dahiya et al., 2020; Duda et al., 2016; Tariq et al., 2018), research on the efficacy of these measures was still in its early stages. Additionally, logistical barriers, such as lack of insurance coverage in the United States for clinical telemedicine services, made offering these services in medical settings largely impractical. However, when clinical services and research protocols were halted due to the pandemic, the field was forced to quickly seek and adopt innovative approaches that were feasible through telehealth or in-person visits with necessary safety protocols, leading to rapid expansion of remote services and approval of insurance-based telehealth coverage. Even as pandemic safety protocols have started to loosen, there is continued interest in the use of flexible, remote approaches to ASD assessment, which likely will continue to shape how services are delivered going forward. Not only would expansion of services improve feasibility and convenience for families, but it could also improve access to care by allowing those in more remote areas to be seen without traveling out of their local area. Bolstering empirical evidence for newly developed tools and methods of assessment is imperative as the field moves to embrace these new approaches.

Several measures that were developed pre-pandemic have been offered as potential remote assessment options. Some of the most widely used (Jang et al., 2021) include the TELE-ASD-PEDS (Corona et al., 2020), Systematic Observation of Red Flags of ASD (SORF; Dow et al., 2020), Naturalistic Observation Diagnostic Assessment (NODA; Smith et al., 2017), and the Childhood Autism Rating Scale (CARS-II; Schopler et al., 2010). While the TELE-ASD-PEDS and SORF offer promising options for young children under the age of three, the TELE-ASD-PEDS had only previously been studied in a laboratory setting and psychometric properties are not yet available for use in the home (Wagner et al., 2021), and research on the SORF has only been conducted on the full one-hour home observation by videographers (Dow et al., 2020) with no data yet substantiating use of a shorter, more feasible timeframe. The NODA is also a promising option for children, though research assessing its specificity in differentiating ASD from non-ASD developmental delays has not yet been conducted. The CARS-2 offers an option that can be used through adulthood but has not yet been validated for use without live, in-person observation.

Despite having some options for remote assessment, many groups expressed interest in continuing to administer the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2; Lord, Rutter, et al., 2012; Lord, Luyster, et al., 2012) with modifications, likely due to its widespread use and number of clinicians trained to administer it. The ADOS-2 also boasts a strong empirical base (Lebersfeld et al., 2021), the ability to assess individuals across the age span, and had previously been required by many insurance providers to ensure coverage of medical services. However, standard administration of the ADOS-2 was not possible during the pandemic, and the adaptations needed altered the social environment when administered in-person while wearing face masks or using barrier shields. Anecdotally, concerns about the impact of masks, both on the client and provider, have been supported by reports from clinicians and parents. Significant adaptations are required for telehealth administration as well, such as only asking social and emotional questions and engaging in conversations for verbally fluent individuals, or having caregivers administer prompts for less verbal individuals and/or young children. While there has been some support for the utility of Module 4 remote administrations in terms of sensitivity with an all ASD sample (Schutte et al., 2015), the specificity of the instrument using standardized scoring would likely be compromised given that the full span of activities cannot be administered to elicit certain behaviors of interest, and nonverbal communication is more difficult (or arguably impossible, in the case of eye contact) to read when directed to a screen. For younger children, requiring caregivers to prompt their children through ADOS-2 tasks changes the standardized way the activities are carried out. While a parent-mediated context may provide rich qualitative information, the standardized scoring from the ADOS-2 should not be applied. Without the ADOS-2 during the pandemic, however, there was a gap where the gold-standard tool to assess autism used to lie, which left many clinicians and researchers searching for a convenient and effective replacement when other available options did not seem like a good fit for their clinic population or set-up.

One benefit of the ADOS-2 is that it provides a relatively natural and consistent context for clinicians to observe social communicative behaviors and to place these observations within a framework of standardized codes. This makes it an important counterpart to caregiver-report or self-report of symptoms and behavior, which can provide an incomplete picture of an individual’s needs. With this in mind, the Brief Observation of Symptoms of Autism (BOSA; Lord et al., 2020) was developed to provide a similarly naturalistic social context with standardized materials and activities, adapted from the Brief Observation of Social Communication Change (BOSCC; Grzadzinski et al., 2016) and ADOS-2 (Lord, Luyster, et al., 2012; Lord, Rutter, et al., 2012). The BOSA consists of a 12–14 min interaction between an individual and a caregiver or clinician. New materials and/or conversational contexts are presented every 2–4 min while the dyad interacts naturally together using the materials and/or prompts.

The administration of the four versions of the BOSA is based primarily on tasks employed in previously developed BOSCC versions (i.e., MV or “minimally verbal”, PSYF or “phrase speech-young fluent”, F1 and F2 for “fluent” 6 through 10 years olds (F1) and “fluent” 11-year-olds through adults (F2)), chosen according to the individual’s age, language, and developmental level. Materials were modified and selected to create a standardized social context consisting of ADOS-2 toys and materials, interactive games that elicit shared affect, and question cards for older children and adults that include conversation prompts and ADOS-2 questions related to emotional experiences, social relationships, and responsibility. There are two sets of toys or materials used in each administration – one for the first approximately 6 min and one for the second 6 min (with slight differences in timing for the different versions).

The BOSA-MV is for individuals of any age who are nonverbal, have single words or only rote phrases, and consists of two sets of ADOS-2 toys and bubbles. The BOSA-PSYF is intended for individuals of any age who use flexible phrase speech or for individuals who are verbally fluent under ages 6 to 8 years old and includes two sets of ADOS-2 toys, bubbles or a rocket launcher, and a dollhouse or toy mailbox to help elicit conversation about the materials. Toy sets include action figures, dolls, furniture, purse with accessories, a Poppin’ Pals, ball, plates, a pinball game and other materials taken from the ADOS-2 kit. Between the ages of 6 and 8, clinical judgment should be used based on the child’s verbal ability and attention span, as the PSYF is more structured and play-based, versus the F1 which is more conversational.

The BOSA-F1 is for verbally fluent children as young as 6 and up through age 10 and involves turn-taking games, answering socioemotional and conversation-starter questions, and having two unstructured conversations without materials present. The BOSA-F2 is for children from age 11 through adults and involves similar activities as the F1 with more advanced, age-appropriate games as well as questions and conversations. A short game of Slap Jack or tabletop basketball is used as a “warm-up,” then games are played while asking and answering questions using question cards (many taken from the socio-emotional questions in the ADOS-2) that correspond to the color of the game piece. Recommended games include Pop the Pig for younger fluent individuals and Jenga for older fluent individuals, as well as other interactive games.

After administration, a clinician trained in the ADOS-2 who has observed the BOSA live or through videorecording, scores the appropriate ADOS-2 protocol based on the participant’s age and language level, as well as any additional codes provided in the BOSA manual for that module. Clinicians should not score items if they do not have enough information to make an accurate judgment regarding the presence or absence of a particular symptom. ADOS-2 scores are then transferred onto a DSM-5 Checklist broken down into ASD symptom categories and converted to binary BOSA scores to indicate presence or absence for each symptom. Additional columns are also provided to note evidence collected from parent report or other observations to aid in determining whether the individual meets diagnostic criteria across domains.

The current project aimed to examine the psychometric properties and optimal use of the BOSA, as supported by empirical evidence and data-driven scoring procedures. Converted ADOS-2 scores were examined by module for best-performing BOSA algorithm items, including sensitivity and specificity. Because of COVID-19 related restrictions, we were aware from the start that our immediate focus would be on sensitivity of the BOSA, because our ability to recruit comparison groups into a new study was very limited; we provide some data on specificity whenever we were able to obtain it. Cutoff scores for each algorithm and suggested ranges of concern were developed to aid in clinical utility, and convergent validity with the ADOS-2 was examined for modules with large enough samples. As with various versions of the ADOS and ADI, we consider this work preliminary and hope it will provide a basis for replications and likely revisions with larger and more representative samples.

The BOSA offers a new option for a remote assessment measure that can be used both in-person and through telehealth. It can be administered by someone without expertise in ASD and scored based on video-recorded observation. Its use of ADOS-2 coding and many of the ADOS-2 materials make it a convenient option for clinicians already trained in the ADOS-2. It could be used as a more accessible and efficient measure to either determine initial risk for ASD, as a level-2 screener to see if a full evaluation with an ADOS-2 or other standardized diagnostic observation is warranted, or in conjunction with other methods (e.g., parent interview and developmental history, behavioral observations) to make a diagnosis. Our intention in creating this measure was to support the clinical and research field to access and deploy the BOSA easily when face-to-face interactions were no longer feasible or safe; therefore, training and materials have been available free-of-charge since conception, with over 20,000 training video views and translation of administration into 5 languages to further facilitate even greater accessibility internationally.