Supporting Caregivers Remotely During a Pandemic: Comparison of WHO Caregiver Skills Training Delivered Online Versus in Person in Public Health Settings in Italy

Intervening early with evidence-based psychological therapies for children with autism spectrum disorder (ASD) is key to optimize development and wellbeing trajectories (Pierce et al., 2016). Despite this, access to early interventions is inadequate in low-and-middle-income countries (LMIC) (Reichow et al., 2013) as well as in high-income countries (HIC), including countries in the European region (Bejarano-Martín et al., 2020; Salomone et al., 2016) and the United States (Smith et al., 2020). This is linked to specific barriers that limit access to services, including cost of treatment for families (Elder et al., 2016) or insufficient number of trained health professionals (Elder et al., 2016; Esposito et al., 2020), large waiting lists, unavailability of specific services in non-metropolitan areas (Monz et al., 2019) and cost and time for travel to clinics (Elder et al., 2016; Murphy et al., 2021). To respond to this treatment gap, interventions adapted for delivery via telehealth have been implemented for a few decades (Reed et al., 2000), however not as mainstream options within public health systems. More recently, the COVID-19 emergency accelerated innovation and attention towards the use of telehealth, which often represented the only viable option to provide psychological, behavioural or psycho-educational care services in place of in-person interventions, which were severely reduced or completely ceased at the height of the early phases of the pandemic (Colizzi et al., 2020; White et al., 2021). Telehealth interventions indeed present several benefits, including affordability and potential to reduce barriers to access. Compared with the in-person model and the in-home model, the telehealth model may provide timely and evidence-based support to families who cannot otherwise access traditional services (Ellison et al., 2021; Sutherland et al., 2018) and at the same time reduce costs for both families and the service delivery system (Little et al., 2018). For some users, remotely delivered services may even provide a welcome opportunity to avoid sensory stressors related to travel or waiting time while retaining the benefits of the interventions (Harris et al., 2021). While certain features of interventions directed to young children or adults with more severe communication impairments may be limited in their potential for adaptation to a telehealth format, parent training, psychoeducation, and parent-mediated interventions lend themselves to be adapted for remote delivery with relatively few modifications.

Telehealth has the potential to break several of the existing barriers to service access and has been recommended as a means to connect people to knowledge and training (Lord et al., 2022). However, it may also, by its nature, have additional, novel barriers that may impact its feasibility and acceptability, i.e., respectively, the extent to which the intervention can be carried out as planned across sites and its appropriateness or relevance to users and providers in real-world settings (Smith et al., 2007). Indeed, despite a high need for remotely delivered interventions at the beginning of the COVID-19 pandemic, in Italy, a country hit hard and early on, only under a third of caregivers reported receiving remote support from public healthcare services, schools or private therapists during the first wave of the pandemic, with only a minimal proportion finding such support useful (Colizzi et al., 2020).

A number of studies have examined the acceptability, feasibility and effectiveness of remotely delivered parent training interventions for caregivers of children with ASD. For feasibility, the primary concern when implementing a telehealth service is technological issues. Although prevalence of owning a device (pc, tablet, smartphone) and an internet connection is growing worldwide, the World Bank (2016) has warned against the rising digital disparities between HICs and LMICs, which have been exacerbated even more by the COVID-19 pandemic (Amaral & de Vries, 2020). Across regions and countries with different levels of income there is still a great percentage of disadvantaged families who have no Internet connection (Little et al., 2018; Salomone & Maurizio Arduino, 2017). For telehealth services to be effective, it is recommended that users have a highspeed, wired-cable Internet connection, a laptop with webcam (Lee et al., 2015) and device that can support necessary software. However, this is not always possible as participants may use cellular services that do not ensure a good and stable connection (Gerow et al., 2021) or there may be multiple people in the same home connected to the internet slowing down upload speeds (Pierson et al., 2021). Socioeconomic status is another aspect that can impact the attendance and feasibility of remote intervention, as caregivers may feel overwhelmed by competing responsibilities (Dai et al., 2021). Also, the presence of disruptions and/or other family members who may require attention, interrupt, and interfere with the assessment or intervention (Lerman et al., 2020) may have an impact on feasibility, to the point that families may need to restrict scheduling of appointments to times when siblings are busy or when another parent is present (Gerow et al., 2021). Lower internet skills and general satisfaction with the service providers may also negatively affect participation (Salomone & Maurizio Arduino, 2017).

Conversely, despite such barriers to feasibility, the acceptability to caregivers is usually high (Baharav & Reiser, 2010; Bearss et al., 2018; Dai et al., 2021; Ingersoll & Berger, 2015; Lau et al., 2022; Montiel-Nava et al., 2022; Pickard et al., 2016; Sengupta et al., 2021a; Tsami et al., 2019; Vismara et al., 2018; Wainer et al., 2021) and, generally, caregivers report high levels of satisfaction (Boisvert & Hall, 2014; Pi et al., 2021; Sutherland et al., 2018). Levels of caregiver involvement are satisfactory and greater in therapist-assisted than in self-directed online programs (Pickard et al., 2016). This is relevant as high levels of acceptability and caregiver involvement are associated with greater likelihood of completing the intervention and higher fidelity of strategies implementation (Ingersoll & Berger, 2015; Little et al., 2018). High levels of parent fidelity to intervention and adherence to home practice can be reached (Bearss et al., 2018; Sengupta et al., 2021a).

Finally, some evidence is available regarding the effectiveness of remotely delivered parent training interventions, i.e., their ability to have an effect in a defined and real clinical setting. Systematic reviews show that telehealth services for children with ASD, including parent-training and parent-mediated intervention, are possibly equivalent and comparable to in person services (Boisvert & Hall, 2014; Ellison et al., 2021; Sutherland et al., 2018) or show better outcome than control groups (Ellison et al., 2021); although a metanalysis did not find consistent improvements in the social communication domains (Pi et al., 2021).

Given the mixed picture of advantages and challenges of telehealth, the field would benefit from evidence obtained from comparisons between in-person and online parent training for parents with children with ASD; however, these studies are rare and limited in design. Ashburner et al. (2016) qualitatively explored through interviews with four caregivers and nine service providers their experience of participating in remote and in-person interventions, identifying the following benefits: cost savings, time and travel, flexibility, and convenience; and the following disadvantages: technical issues and difficulties establishing effective communication. Bearss et al. (2018) compared an online delivery of the RUBI parent training for maladaptive behaviour with the pre-specified benchmarks of the original trial (Bearss et al., 2018), reporting comparable feasibility and preliminary efficacy across modalities. Hao et al. (2021) used a non-randomised design to compare an adaptation of Project ImPACT delivered in-person and remotely showing equivalent and comparable outcomes between groups, but acceptability or feasibility were not measured. Lau et al. (2022) examined acceptability and feasibility of the “Caregiver skills training for families of children with developmental delays or disabilities” (CST; Salomone et al., 2019; WHO, 2022) across asynchronous e-learning, fully virtual synchronous and hybrid in-person and virtual delivery modes showing comparable acceptability and satisfaction rates across the three modes, but the study was limited by the reduced sample size and high attrition rates.

There is therefore scant evidence of equivalence or superiority between remotely-delivered caregiver-mediated interventions and in-person delivery modes. This study aims to contribute to address this research gap, adapting an existing intervention to telehealth format and comparing it to existing data from face-to-face delivery. We employed a mixed-method approach to report on the process of adapting and piloting a remote-delivery version of the CST. The CST, which is informed by principles of naturalistic developmental behavioural interventions (NDBI: Schreibman et al., 2015), has shown good acceptability and feasibility in both high- (Lau et al., 2022; Salomone et al., 2021a; Seng et al., 2022) and low-resource contexts (Montiel-Nava et al., 2022; Sengupta et al., 2021b; Tekola et al., 2020). The study aims to examine the implementation of CST delivered remotely in public health services in Italy. Specifically, we aimed to: (1) compare the overall feasibility, acceptability and clinical outcomes data between the ‘virtual’ (i.e., remotely delivered) and in-person delivery modes, with a secondary analysis of data derived from a pilot randomised controlled trial (RCT) of CST delivered in person in public health settings (Salomone et al., 2021b); (2) assess the feasibility and acceptability of novel and adapted components specific to the virtual CST; (3) explore experiences of parents’ and clinicians’ participating in the virtual CST with a qualitative analysis. Such mixed-method approach with multiple informants (caregivers, interventionists and observers) allowed us to utilize the strengths of both the quantitative and qualitative methods, exploiting the qualitative data to validate or refine quantitative findings and quantitative data to explain findings from the qualitative data (Fetters et al., 2013).

The pilot implementation of CST delivered remotely (hereafter: ‘virtual CST’) had a pre-test/post-test design; data were collected at baseline and immediately post-intervention. Three editions of the virtual CST were delivered on a first-come, first-served basis. Recommended group size in a CST group is 8 caregivers; in our study, group sizes for the three editions were: n = 8; n = 8; n = 9, for a total sample size of 25 participants. The three groups attended the virtual CST between October 2020 and September 2021, when the restrictions due the COVID-19 pandemic resulted in the closure of schools, working from home arrangements and limitation or cessation of most face-to-face services.

Additionally, we used existing data from a pilot RCT of CST delivered in person (n = 43) against enhanced treatment as usual (TAU; n = 43) previously conducted in the same geographical area in Northern Italy (Salomone et al., 2021a; Salomone et al., 2021b) to compare acceptability, feasibility and preliminary clinical outcomes of the two delivery methods. All participants were caregivers of children with a clinical diagnosis of ASD, recruited through Child Neuropsychiatry units of the National Health Service in the Piedmont region of Italy (Table 2). Both in the pilot of virtual CST and in the original trial, caregivers and facilitators completed feasibility and acceptability measures after each session, and at endpoint took part in focus groups or interviews. Observers completed fidelity ratings. For additional details of the procedure, measures used and description of the intervention procedure in the in-person implementation of CST, see Salomone et al. (2021a). See participants’ flow in Fig. 1.

For virtual CST, teleconferencing platforms (Zoom and Webex) were used to deliver the nine weekly group sessions and the second and third home visits. The first home visit was delivered in person, as explained above. The online intervention was delivered as per the adapted manual by three clinical psychologists and one therapist with experience of CST and previous clinical work with families with children with ASD. Three of the four interventionists (‘facilitators’) had previous experience with the use of videocalls in clinical work and had previously delivered CST in-person having reached minimum fidelity criteria. Participation in the intervention was open to two caregivers per family; however, data were collected on one designated target caregiver. Caregivers used personal devices (smartphone, tablet, laptop, desktop computer) to connect to the sessions: 10 (41.7%) used a smartphone for at least 50% of the group sessions, 14 (58.3%) a laptop or tablet.

The in-person CST was delivered per manual (Salomone et al., 2021b).

Immediately after the end of the online delivery, focus groups and telephone interviews were held with all available participants (n = 16) and with all the interventionists (n = 4).

In the virtual CST group, basic internet skills were satisfactory (M = 4.47, SD = .79). 19 (79.2%) participants had previous experience with videocalls with healthcare professionals. Regarding background characteristics, while the TAU and the in-person CST groups did not differ on any of the baseline measures, the virtual CST group differed on child age and caregivers’ nationality (Table 2). Baseline levels of outcome measures are reported in Table 5.

We compared in-person and virtual adaptation of CST on feasibility and acceptability (respectively Tables 3, 4).

Attendance and drop-out rates were not significantly different across the two delivery methods. In the online delivery group three caregivers dropped out due to religious commitments, conflicts with work schedule and issues with childcare during lockdown, respectively after the first home visit, after Session 2 and after Session 5. In the in-person delivery group, two caregivers dropped out after the first home visit for substance abuse, chaotic life patterns and high levels of parental conflict, as reported by them. The two groups did not differ for intensity of adherence to the home practice nor for reported levels of enactment difficulties in the practice, but caregivers in the in-person group endorsed significantly more frequently unexpected circumstances and lack of time as contextual barriers to the practice. Contents were rated by facilitators as significantly more complex to deliver in the online delivery but there was no difference in the evaluation of the amount of content (Table 3).

Regarding acceptability, caregivers in the online group rated the contents significantly less comprehensible, less applicable and less aligned with values, compared to caregivers who received in-person delivery. In the virtual CST, facilitator ratings of caregiver participation were significantly lower compared to those in the in-person group. No other significant differences emerged. The two groups did not differ in therapeutic alliance, comfort and confidence levels (Table 4).

Table 5 shows the results of the ANCOVA analysis examining the effect of group membership (virtual CST, in-person CST, treatment as usual) on pre-post intervention change scores of clinical outcomes, after controlling for baseline scores. There was a large and significant effect of group membership on caregiver competence, measured with the KSQ (η² = .131). Post hoc analyses performed with a Bonferroni adjustment showed that the in-person and the virtual CST groups had both significantly improved more than the TAU group but did not differ from each other. No statistically significant effects emerged for parental stress, measured with the APSI nor for parental self-efficacy, measured with the CSQ.

Facilitator-rated feasibility and acceptability of group sessions were above satisfactory levels (M = 4.33, SD = .516; M = 5.00, SD = .00), although issues with technology were reported. 50% of facilitators reported difficulties with screen sharing and video streaming, and 13.7% of caregiver reported technological difficulties. Distractions and interruptions from other family members were reported in 23.4% of ratings across the nine group sessions. Most caregiver and facilitator acceptability ratings for adapted components were in the ‘good’ range, however ‘unsatisfactory’ for perceived realism of the videorecorded modelling of strategies by caregivers (M = 3.32, SD = 1.18) (Table 6).

Facilitator-rated feasibility of home visits were in the ‘satisfactory’ range (M = 3.83, SD = .40), but the facilitators were able to deliver video-feedback to only 72.1% of caregivers due to technical difficulties. Facilitator-rated acceptability of home visits were above satisfactory levels (M = 4.50, SD = .54).

Most caregiver-acceptability ratings for home visits delivered online were in the ‘good’ range, but in the ‘unsatisfactory’ range for representativeness of child behavior during the home visits (M = 3.76, SD = 1.12). (Table 7).

Five main themes emerged from the thematic analysis, as detailed below (Fig. 2 and Supplementary Materials).

The combination of quantitative and qualitative methods to examine in detail acceptability, feasibility and preliminary indicators of effectiveness of a community-based intervention delivered online and in-person can be considered methodological strengths of this study. However, our findings should be considered in the context of some design limitations. We relied on a secondary analysis of the Salomone et al. (2021b) trial to compare online and in-person delivery modes as it was deemed unethical to randomly assign participants to a ‘treatment as usual’ condition considering that the study was conducted at the height of COVID-19 restrictions, when most services were unavailable or substantially reduced in Italy (Bentenuto et al., 2021). Therefore, the lack of randomization, the striking differences of the historical periods in which the data for the in-person and virtual intervention were collected (during the pandemic and ‘normal time’), differences in some of the baseline characteristics and uneven across groups sample sizes warrant caution in the interpretation of the findings. Additionally, restrictions in place to face-to-face contacts due to the pandemic and ethical considerations to limit the burden of assessments of families prevented the collection of high-quality direct observational data and clinician direct assessments.

Future studies should build on the promising findings with respect to feasibility, acceptability and preliminary outcomes to formally examine the effectiveness of virtual CST against in-person delivery with a randomised controlled design, at a time when there are no restrictions in place to face-to-face contacts and employing blind rated outcomes from direct observational measures and assessments.