Autism Detection in Children by Combined Use of Gaze Preference and the M-CHAT-R in a Resource-Scarce Setting

Children 36–99 months old were recruited from educational centers throughout Lima, Peru. Children with ASD were identified through three private child development centers in Lima, including the Instituto Medico Lenguaje y Aprendazaje, (IMLA), where they were receiving therapy for a previously established diagnosis of ASD. The diagnosis of ASD for these children was consistent with criteria from the Diagnostic and Statistical Manual of Mental Disorders-5^th Edition (American Psychiatric Association 2013), based on a combination of clinical evaluation and previous exams including evoked auditory potentials, language and sensorial processing evaluations, and psychological evaluations. Participants were excluded if they had a primary developmental disability diagnosis other than ASD, such as a language disorder, general developmental disorder, or learning disability, in addition to autism. The control group consisted of typically developing (TD) children who were identified through public schools and daycare centers in Lima, Peru. Control-group children were eligible if they had no previous diagnosis of ASD, specific language disorder, general developmental disorder, learning disability, intellectual disability, or neurologic illness, and no illness at the time of enrollment.

The MCHAT-R was selected for use as a component of the screening protocol because of its broad availability in a number of languages including Spanish and its ease and accessibility of administration (Canal-Bedia et al. 2011; M-CHAT™—MCHAT R/F Translations 2020). While the MCHAT-R is widely used as a screener for possible ASD (Coury et al. 2017) it is not without its limitations. In addition, traditionally the MCHAT-R is considered most valid as a screening tool for children through 48 months of age. However, given that children in Peru do not typically present for, and are not usually identified with, ASD until 36 months or later (Observatorio Nacional de la Discapacidad | CONADIS Peru—OBSERVATORIO DE LA DISCAPACIDAD 2020), we elected to expand the study group age range to 99 months in order to access a larger pool of individuals for this exploratory study.

All children were screened with the MCHAT-R- and ADOS-II at the time of recruitment.

Control-group children who screened positive with the ADOS-II evaluation at the time of enrollment were excluded from analysis, as they lacked a clinical diagnosis of ASD.

We used a standard laptop with a 17-inch screen with two videos, one of social scenes and the other of abstract scenes playing, separated by 8 inches (Vargas-Cuentas et al. 2017). The original video used for the testing, was saved with a resolution of 1920 × 1080 pixels, at an aspect ratio of 16:9, and displaying at 60 frames per second. The social and the abstract images displayed were projected in rectangular areas on the left and on the right of the screen. The size of the image was 37.5% of the width and 37.04% of the height of the screen. The original video was recorded at a temporal resolution of 60 frames per second and this is the way it is “projected” in the screen. However, the camera that recorded the video of the face of the child, captured a video with a temporal resolution of 30 frames per second.

For the gaze preference screening, each child watched a minute-long video on a computer with a front-mounted camera that recorded their face movements. The video consisted of a 10-s introductory scene to attract the child’s attention, followed by a split screen showing the abstract scene on the right side of the screen and the social scene shown on the left. The social scene consisted of a video of young children playing and interacting with each other, and the abstract scene consisted of moving shapes of various colors. These visits took place at the respective recruitment sites. Each child was seated in a chair 30 cm from the computer screen. If the child was unable to sit alone, they were placed in their guardian’s lap 30 cm from the computer screen. This distance allowed for clear visibility of the eyes by coders without being uncomfortable for the child. If the child’s attention turned away from the video, they were redirected either with auditory or visual cues to the screen by their guardian or supervising study personnel. The procedure was repeated approximately one month after the initial recording. No adverse effects were observed associated with this procedure.

The gaze-preference device was based on the study by Vargas-Cuentas et al. 2017. The current study was done to provide the proof-of-concept for the validity of our approach and correctness of our hypotheses. We used manual coding to bypass any coding software issues. We are in the process of improving our coding software to enable fully automated screening process. Study personnel reviewed each video at a rate of 2 frames per second, one for every 15 frames filmed (101 frames per child). At each time point the child’s gaze was labeled as either left, right, center or distracted. Left indicated focus on the social scene, right indicated focus on the abstract scene. Distraction was defined as any time that the child was not looking within the frame of the video or their eyes were not visible. Center was when the child was looking at the center of the screen between the social and abstract video. Two separate, independent groups of trained coders who were blinded to the diagnostic status of the subjects coded the videos taken at visit one and visit two. In cases of discrepancy, the two coders reanalyzed the frame in question and decided the gaze direction with consensus. Given the constraints of the study conditions, a more traditional method of determining interrater reliability was not undertaken. Gaze preference was determined by calculating the proportion of the stimulus display time—or equivalently, the proportion of all frames—the child spent looking at the social scene, abstract scene, center of the screen, or not looking at the screen (distraction time).

Stata 14.2, StataCorp, College Station, Texas, USA was used for statistical analysis. To demonstrate the gaze-preference patterns among ASD and typically developing (TD) children over the course of the video, we used LOWESS smoothing (locally weighted scatterplot smoothing) (Cleveland and Devlin 1988) with a bandwidth of 0.8. A Mann–Whitney Test was used to compare proportions of social and abstract scenes and distraction times between TD children and children with ASD. We verified the consistency of these results between two separate gaze-preference visits by intra-class correlation ICC) between visits. We employed Wilcoxon signed-rank test to compare gaze preference of the same group of children at different segments of the video. To develop a two-step approach to classifying ASD and control children, we estimated a logistic regression model and used the Hosmer–Lemeshow test with group sizes of 5 and 10 in separate tests to examine the model goodness of fit. We also report the chi-sq test of goodness of fit between the predicted and actual ASD status and have included a plot of their predicted and actual values and a plot of predicted probability vs the number of social scenes in the supplement. This model used the results of the MCHAT-R in addition to gaze preference as predictors of ASD status. We used AUC to compare models based on the first 15 s of gaze-preference data against one based on the entire 50 s length of the video and subsequently we tested the model in one half of the sample and used each to calculate the AUC for the other half of the sample. Additionally, we tested the coefficient of the interaction term between the dummy variable indicating each half of the sample and each of the main model predictors (MCHAT-R and gaze preference results) to test whether the models from the two halves of the sample were different. We used a similar approach to test whether a model based on the age groups shared by cases and controls could predict the ASD status of children outside those age ranges. We also computed sensitivity of the model at specificity levels of approximately 70%, 80% and 90% to indicate how model sensitivity would change at different levels of specificity. To study the association between children’s ADOS-II scores and their gaze preference, the Mann–Whitney test was used. Participating children were excluded from the analysis if they were controls with a positive ADOS-II score (n = 7) in which case their true ASD status could not be ascertained or if their MCHAT-R or ADOS-II scores were missing (n = 16). Additionally, there were two controls and one case excluded due to acute illness.

We administered the gaze-preference test at 2 separate occasions, on average 1 month apart. The analyzable sample size in visits one and two were 101 (73 TD controls and 28 children with ASD) and 100 (73 controls and 27 children with ASD), respectively. Cases were predominantly male (89%) and older (median = 60 months) compared to controls who were 51% male and younger (median = 42 months) (Table 1). Please also see tables S.1 in the Supplement.

Gaze preference classification detected differences between the group of children with ASD and the TD control group (Table 2 depicts these differences in visit 1 while visit 2 results are presented in table S.3 in the Supplement). As seen in Table 2, over the full 50 s, children with ASD spent significantly less time than the TD control group focused on the social scene at both visit 1 (21.5% vs 33.5% of the length of the video, p < 0.0001) and visit 2 (24% vs. 36%, respectively, p < 0.001). Furthermore, although children with ASD were not significantly more distracted than TD children in the first visit (26.7% vs 18.3%, p = 0.170) (Table 2; Fig. 1a, b) this difference was significant in the second visit (25.0% vs 12.2%, p = 0.013). By contrast, attention to the abstract scene during the 50 s, although higher, was not significantly so, among ASD-diagnosed children in visit 1 (50.5% vs 46.2%, p = 0.358) or visit 2 (50.8% vs 50.9%, p = 0.920). Gazing at the center of the screen was rare and it was inconsistent between the two visits among TD children. (Table 2 and Fig. 1a and b, for more detailed graphs and LOWESS with different bandwidth see Supplement Figures S.4-S.6).

Graphs for visit 1 and visit 2 (Fig. 1a–d) suggest a very high degree of consistency in gaze patterns between the two visits (based on the full length of the video, the between visit ICC for social scene = 0.54, 95%CI: 0.40–0.68, for abstract scene = 0.44, 95%CI: 0.29–0.60, for distraction = 0.41, 95%CI: 0.24–0.57, for center = 0, 95%CI: 0–0.20).

Visual inspection of graphs of smoothed trajectories of gaze preference behavior (Fig. 1c and d) suggests that the first 15 s maximally discriminate between children with ASD and controls in both visits: as most TD children started the video by gazing at the social scene an increasingly fewer proportion of them did so until the trajectory for social scene viewing crossed that for abstract scene at around the 600^th frame (17^th second). By contrast, children with ASD tended to view the abstract scene throughout this segment and paid increasingly less attention to the social scene. Based on these observations, we hypothesized that gaze patterns during the first 15 s of the video discriminated between children with ASD and TD controls as well as gaze patterns over the entire length (50 s) of the video (Fig. 1a and b and Table 2).

Attention to the social scene diminished among both TD children and those with ASD as time elapsed (Wilcoxon signed rank test of difference between 15 and 50 s: 50.1% vs 33.5%, p < 0.001 for control group, 30.1% vs 21.5%, p < 0.001 for ASD group) while distraction increased in both groups (Wilcoxon signed rank test of difference between 15 and 50 s: 12.9% vs 18.3%, p < 0.001 for control group, 18.5% vs 26.7%, p < 0.041 for ASD group). Children with ASD spent about 50% of their time viewing the abstract scene, and this tendency did not change significantly throughout the video (Wilcoxon signed rank test of difference between 15 and 50 s: 50.3% vs 50.5%, p = 0.793). By contrast, typically developing children paid significantly less attention to the abstract scene during the first 15 s of the video but paid more attention as time elapsed (Wilcoxon signed rank test: 33.5% vs 46.2%, p < 0.001). These patterns and the significant differences reported in Table 2 suggest that the data from the first 15 s of the video are as discriminating between the ASD and control groups as the data from the entire length of the video for social focus and distraction, while abstract scene viewing is a significant discriminator only in the first 15 s.

Figure 2, corresponding with the first and the second visit, demonstrates that ADOS-II classification was inversely associated with the proportion of social scene frames during the first 15 s; i.e., ADOS-II-positive children viewed the social frame less frequently (Fig. 2). By contrast, ADOS-II classification was directly associated with the proportion of abstract scene frames suggesting that ADOS-II positive children preferred the abstract scene. We also observed that distraction during the first 15 s of the video was higher in ADOS-II-positive children. These associations were statistically significant.

Gaze preference over the full length of the video could discriminate between TD children and those with ASD, but we also observed that the initial 15 s of the video was at least as discriminating which prompted us to hypothesize that a shorter video could represent gaze preference before distraction sets in. This was also suggested by Fig. 1, by the steadily rising distraction lines. We hypothesized that the first few seconds before the lines for abstract and social scene cross and when the difference between the proportion of abstract and social gazes are almost opposite (on average) between cases and controls (prior to frame 600) and while distraction is at its lowest, we could get the most discriminating information. A predictive logistic model with one or more gaze-preference variables also suggested this (Table 3ai). In our sample, MCHAT-R with critical questions demonstrated a sensitivity of 93% (95% CI: 76–99%) and a specificity of 63% (95% CI: 51–74%), with an AUC of 0.78. We used a 2-step screening including gaze preference and MCHAT-R to improve on the MCHAT-R’s low specificity while incorporating its high sensitivity. Of the gaze-preference variables, social scene focus was the most predictive option, and we used it in combination with MCHAT-R results to construct a parsimonious, unweighted logistic regression model, the GP-MCHAT-R model (Table 3ai, ii). Adding other gaze preference variables within the same model did not improve the fit or model prediction. We also examined a model with both abstract scene and distraction – but without the social scene – which could predict ASD similarly well, but we used the model with social scene since it was more parsimonious. The Hosmer–Lemeshow test for the model based on social scene indicated good fit throughout the response range (p = 0.397 based on quintiles of data and p = 0.577 based on deciles). In the Supplement, we have presented the details of the Hosmer–Lemeshow test as well as two graphs presenting a histogram of model predicted probabilities in the two groups of children and the predicted probabilities plotted against social15 values (tables S.4 and S.5 and graphs S.1 and S.2). In another step for model validation, we initially estimated the model in each of two random halves of the sample (seed = 99, n1 = 55 and n2 = 46) and produced the ROC curve for the estimation half (AUC1 = 0.89 and AUC2 = 0.87) as well as the validation half (AUC3 = 0.88 and AUC4 = 0.89). To ensure the model applicability to different random subsamples of the data, we conducted an analysis to test the coefficient of the interaction terms between the dummy variable representing the two random subsamples separately with social15 and MCHAT-R in the model. The p-values for these interactions (0.681 and 0.622, respectively) suggest no difference in these coefficients when estimated in each subsample. Results are presented in the Supplement (Table S.6 and Figure S.3). Table 3a and Fig. 3a demonstrate the 15-s model and the 50-s model. To verify this, we conducted a likelihood-ratio test to examine the effect of the presence or absence of each of social15 and social50 in a model with MCHAT-R. The full model consisted of MCHAT-R, social15 and social50 as predictors. We tested this against a reduced model with social15 but without social50 and one with social50 but without social15 (likelihood-ratio test comparing each reduced model against the full model: p = 0.464 and 0.009, respectively) and concluded that the absence of social15 in the model leads to a significantly worse model whereas absence of social50 is not a significant detriment.

Because the age distribution of the children with ASD and TD controls were different (see Tables S.1 and S.2 in the Supplement) we fitted the model with social15 and MCHAT-R but included a dummy variable identifying ages that were common to both children with ASD and TD children (45–49 months). We then fitted a logistic model with social15, MCHAT-R, the dummy variable and the interaction terms, similar to what was described above for the random subsample analysis (this is also discussed in the Supplement: Comparing the GP-MCHAT-R Model in the Age Groups Including Both Cases and Controls Against Other Ages). Results suggested that the coefficients for social15 and MCHAT-R were applicable to both age groups (p-value for social15 = 0.781, p-value for MCHAT-R = 0.332).

Figure 3a depicts the ROC curves and associated AUC’s for both models in visit 1 and visit 2 data (AUC values are 0.89 and 0.86 for the 15- and the 50-s models). As Fig. 3b indicates, although the AUC for gaze preference (social scene focus in the first 15 s) is slightly less than that for MCHAT-R alone (0.76 vs 0.78, p = 0.796), the combined AUC for the 2-step model, which includes both MCHAT-R and social scene as predictors, presents a significant improvement over MCHAT-R alone (0.89 vs 0.78, p < 0.001), particularly on model specificity.

Considering the importance of both sensitivity and specificity for an effective screening tool and the tendency of MCHAT-R to give false alarms, we also assessed the GP-MCHAT-R model’s sensitivity and specificity in addition to its AUC. Table 3bi demonstrates the sensitivity of the GP-MCHAT-R model associated with specificities of approximately 90%, 80% or 70% in each of visit 1 and visit 2. The GP-MCHAT-R model AUC in visit 1 is 0.89 and in visit 2 it is 0.88 (Fig. 3a). This model has a sensitivity of 64% at a specificity level of exactly 92% when applied to visit 1 data, and a sensitivity of 63% given a specificity of 90% when applied to visit 2 data. Thus, the model can provide an alternative to the low specificity of MCHAT-R when a lower sensitivity is acceptable.