Introduction
Several research-based intervention methods for children with autism spectrum disorders (ASD) have been developed and tested in controlled laboratory settings. There is increasing empirical support, for example, for child skills training and parent-mediated interventions based on principles of applied behavior analysis (see National Research Council 2001; Rogers and Vismara 2008) and cognitive behavioral therapy methods (e.g., Chalfant et al. 2007; Wood et al. 2009).
Despite increasing support for specific intervention models, there is limited knowledge about how these strategies may be implemented in “usual care” community-based service settings (National Advisory Mental Health Council 2001). The limited research that has been conducted on community-based services for children with ASD has focused on early intervention/educational services and suggests that there are discrepancies between research-based practices and community-based care (e.g., McLennan et al. 2008; Stahmer 2007; Stahmer et al. 2005). Further, research suggests that outcomes in community-based mental health clinics for children with other psychiatric problems are not as positive as those observed in research trials (Weisz et al. 2006). Detailed research on usual care service context and treatment for childhood disorders is critical to bridge the gap between what is known about research-based practices and community-based practice (Bickman 2000; Hoagwood and Kolko 2009; McLennan et al. 2008; Westfall et al. 2007).
The limited research on community services for children with ASD suggests that services are provided in a number of different sectors including the mental retardation/developmental disability (MR/DD), special education/early intervention, medical, and mental health service systems. A child may be concurrently served in multiple sectors for different needs (Brookman-Frazee et al. 2009a). For example, a child may receive services through the school system for social and academic issues, and the mental health system for behavioral issues. The current study focuses on services provided to children with ASD in the mental health system where co-occurring psychiatric and behavior symptoms may be targeted. Mental health services are designed to address psychiatric problems through psychosocial and pharmacological interventions. The current study focuses specifically on outpatient psychotherapy services provided to children with ASD in community-based mental health clinics.
The mental health system plays a particularly important role in caring for children with ASD for treatment of overlapping and co-morbid psychiatric problems that can occur with ASD. Although there is significant variability in the estimated prevalence rates of co-morbid psychiatric disorders among children with ASD (Lainhart 1999), recent research suggests that approximately 70% of children with ASD meet criteria for at least one psychiatric disorder (Leyfer et al. 2006a; Simonoff et al. 2008). These high rates of a variety of psychiatric problems for children with ASD underscore the importance of examining mental health services.
The limited research conducted on community-based mental health services for children with ASD has examined the costs of these services (Mandell et al. 2006), the characteristics of children receiving specific types of mental health services (e.g., Bryson et al. 2008; Mandell 2008; Mandell and Palmer 2005; Mandell et al. 2005, 2008), and general descriptions of the types of services provided (Kohler 1999; Ruble et al. 2005). While the data on patterns of community mental health service use are growing, there is limited information on what treatment in these settings actually entails (i.e., what types of treatment strategies are used in service sessions and what types of outcomes are obtained) and for whom these treatment strategies are being applied. The limited descriptive data on community-based practice has primarily been focused on very young children receiving early intervention services (e.g., Stahmer 2007) and has not examined school-aged children or those served in mental health settings.
Descriptive data on the nature of mental health services and treatment outcomes is important for a number of reasons. First, while there is a growing body of research on interventions for children with ASD, including those for children with co-occurring mental health problems, it is not known how clinicians providing community care tailor mental health intervention and treatment for the unique needs and clinical presentations of children with ASD. It is likely, however, that community care may not be as effective as treatment provided in research studies since many community mental health systems were not designed to serve children with developmental disabilities, and it is likely that those providing treatment to children with ASD lack specialized training in this spectrum of disorders (Brookman-Frazee et al. 2009a). Therapists providing outpatient psychotherapy in our community indicate that they perceive children with ASD to be particularly challenging to treat (Brookman-Frazee 2009). Further, research on community providers’ knowledge about ASD indicates that providers (including mental health professionals) hold a number of inaccurate beliefs about ASD (Heidgerken et al. 2005). Understanding how consistent (or inconsistent) routine community care is with interventions provided in highly structured university research settings can provide direction for efforts to improve community care (Weersing and Weisz 2002; Weisz et al. 2004, 2006). Further, descriptive data on usual care practice can provide baseline data for efforts to implement research-based practices in the community.
Detailed description of community services needs to include information on the context in which these services are being provided (e.g., client and provider characteristics). Understanding the characteristics of those children receiving specific services is particularly important as research suggests that there may be variability in ASD treatment response (e.g., Sherer and Schreibman 2005). Research in the area of other childhood psychiatric disorders indicates that child sociodemographic and clinical characteristics may be associated with treatment process variables (e.g., Armbruster and Fallon 1994; Brookman-Frazee et al. 2008) and response to treatment variables (Beauchaine et al. 2005; Reyno and McGrath 2006).
The current study aims to address the dearth of knowledge about community-based mental health services by studying outpatient psychotherapy services for children with ASD provided in community mental health clinics. Data from this study are drawn from a large-scale longitudinal, observational study characterizing outpatient psychotherapy services for children presenting with a broad range of disruptive behavior problems, which represent the most prevalent presenting problems to outpatient mental health services (Garland et al. 2001). The main objectives of this sub-study are to (a) describe the characteristics of children with ASD served in outpatient psychotherapy for disruptive behavior problems and (b) characterize treatment process and outcome for this group of children. In order to provide contextual information about services typically provided in this setting, a secondary aim of the study is to examine differences in sociodemographic and service entry characteristics, treatment process and outcomes for children with ASD relative to a comparison group of children without ASD treated by similar therapists.
Methods
Participants
The sample for the current study was drawn from the “Practice and Research: Advancing Collaboration (PRAC)” study which is a longitudinal, observational study of psychotherapeutic treatment processes for children with disruptive behavior problems. It includes a representative sample of psychotherapists, children and their caregivers in San Diego, CA. (Garland et al. 2006a). The PRAC study participants include 218 children and 100 therapists practicing in six clinics in San Diego County. The six participating clinics were selected because they represent the largest contractors for publicly funded, clinic-based outpatient care for children in San Diego County. The clinics are geographically dispersed to maximize representativeness of urban, suburban, and semi-rural areas, as well as racial/ethnic diversity. All clinics serve patients with a wide range of diagnoses and presenting problems (i.e., they do not specialize in ASD).
PRAC Study Participants
Therapists were randomly selected at the start of the study in late 2003 for recruitment from lists of active therapists. Recruitment proceeded until cells were filled to reflect the distribution of psychotherapists in outpatient clinics in the county by mental health discipline and proportional to the size of the clinic. In subsequent years of recruitment (2004–2006), new therapists were recruited sequentially as they joined the clinics. Of the 163 therapists recruited, 131 (80%) agreed to participate, but only 100 had a child patient participating in the study. Therapists who declined to participate did not differ significantly from participants on age, gender, or race/ethnic distribution, but licensed staff had a slightly lower participation rate (72%) compared to unlicensed staff/trainees (86%). Therapists received an honorarium ($100) for agreeing to participate in the study, regardless of the number of patients who entered the study.
Inclusion criteria for child participants were (a) presenting problems included a disruptive behavior problem (including aggression, defiance, delinquency, oppositional behavior), (b) age between 4 and 13 years at the time of recruitment, (c) primary language for child and parent was English or Spanish, and (d) child was entering a new episode of psychotherapy (defined as no therapy for previous 3 months) with a participating therapist. Clinic administrative staff screened all eligible new patients during the initial call to the clinic for services and obtained permission to share names and contact information with the research team for recruitment. Ten percent of parents declined to be contacted by research staff. Of the 550 who agreed to be contacted and met the inclusion criteria listed above, 55% (n = 292) did not engage in treatment at the clinics, leaving 258 potential participants who were actively recruited into this study. Eighty-five percent (n = 218) agreed to participate in the study. Due to HIPAA restrictions we could not collect data on non-participants, so no information about how non-participants may have differed from participants is available. Informed written consent was provided by the parent and assent was provided by children ages 8 and older. Family participants were given financial incentives to participate in the study ($20 to the parent and $10 to the child at the baseline interview), and families were assured that their decision regarding participation would not impact treatment. All protocols were approved by affiliated university, hospital, and county research review committees.
ASD Group
The primary analyses for the current study were conducted with only those children from the PRAC sample who were identified as having an autism spectrum disorder. Additional analyses compared characteristics of this ASD group to a Non-ASD comparison group (see description below). For this study, all children in the PRAC sample with an existing ASD diagnosis on record (based on therapist report) were identified. Of the 218 PRAC participants, 20 had an ASD. Since one therapist participant had two children with an ASD in the study, one child from this therapist’s caseload was randomly selected to be excluded from analyses. Therefore, the final ASD group for this study included 19 children.
The 19 children with ASD included in the current study were ages 4–12 years (M = 8.2; SD = 2.4), 78.9% were male, and were 73.7% Caucasian, 10.5% Hispanic/Latino, 5.3% African-American, 5.3% Asian/Pacific Islander, and 5.3% Native American. English was the primary language for 94.7% of the child participants (one participant’s primarily language was Spanish). The parent/caregiver informants (hereafter referred to as parents) were primarily biological mothers (84.2%), but also included grandmothers (5.3%), biological fathers (5.3%), and stepmothers (5.3%).
Non-ASD Comparison Group
A comparison group of children without ASD was identified to examine how treatment process and outcome differ between children with ASD and those without ASD served in the same service setting. In order to control for therapist characteristics which may account for variability in treatment process and outcome influence, the comparison group was selected from the caseloads of the 19 therapists serving the participants with ASD. Since 5 of the ASD therapists did not have a non-ASD participant enrolled in the study, alternative therapists were selected and were matched on gender, highest degree, mental health discipline, theoretical orientation, months practice (above vs. below average therapist months of practice) and age (above vs. below average therapists age at baseline).
The final 19 children in the Non-ASD comparison group were ages 5–13 years (M = 9.9, SD = 3.0), 63.2% were male, and were 57.9% Caucasian, 31.6% Hispanic/Latino, 5.3% African-American, and 5.3% Biracial. English was the primarily language for 100% of the child participants. The parent informants were primarily biological mothers (68.4%), but also included grandmothers (5.3%), biological fathers (10.5%), foster parents (5.3%) step-parents (5.3%), and other relatives (5.3%). Table 1 provides additional characteristics of the ASD and Non-ASD comparison groups.
Table 1
Child and family characteristics of ASD and comparison groups
Variable | ASD (N = 19) | Non-ASD (N = 19) |
|---|---|---|
% or M (SD) | % or M (SD) | |
Socio-demographic characteristics | ||
Child age | 8.2 (2.4) | 9.9 (3.0) |
Gender (% male) | 78.9 | 63.2 |
Racial/ethnic minority | 26.3 | 42.1 |
Caregiver age | 39.1 (7.8) | 43.0 (11.8) |
Annual income | $50,811 (53,482) | $39,994 (31,076) |
Primary caregiver’s marital status | ||
Married/Living with partner | 52.6 | 36.8 |
Education level | ||
Some high school | 10.5 | 15.8 |
Some college | 52.6 | 63.2 |
College/grad school | 36.8 | 21.1 |
Service entry characteristics | ||
Primary referral agent | ||
Self | 47.1 | 57.9 |
School | 21.1 | 21.1 |
Other | 31.6 | 21.1 |
Funding source | ||
Medi-cal/unfunded | 42.1 | 73.7 |
AB2726 (school district funding) | 57.9 | 26.3 |
Clinical characteristics | ||
More than 1 diagnosis on record | 71.1 | 73.9 |
Diagnosis (Not mutually exclusive) | ||
Asperger’s disorder/PDD-NOS | 89.5 | NA |
Autistic disorder | 10.5 | NA |
ADHD | 47.4 | 73.7 |
Anxiety disorder | 26.3 | 21.1 |
Disruptive behavior disorder (CD, ODD) | 15.8 | 36.8 |
Mood disorder | 10.5 | 42.1 |
Therapist Participants
The 24 therapists included in the current analyses (19 original ASD therapists + five alternative matched therapists) were 83.3% female, and 79.2% were Caucasian. They ranged in age from 24 to 56 years (M = 31.92; SD = 9.11). The range of years of experience was 0–25 years (M = 2.93; SD = 5.12). A majority of the therapists were master’s level (50.0%), with 4.2% doctoral level and 45.8% Bachelors level (enrolled in a Master’s program). Therapists came from different mental health disciplines: Marriage and Family Therapy (50.0%), Psychology (20.8%), and Social Work (29.2%). Multiple theoretical orientations were also represented; 37.5% identified with Family Systems, 33.3% Cognitive Behavioral, 16.7% Eclectic, 4.2% Psychodynamic, 4.2% Behavioral, and 4.2% Other. Therapists were 66.7% trainees and 33.3% staff.
Procedures and Data Collection
Descriptive data on children and parents were collected during a baseline in-person interview. Child psychiatric diagnoses, number of treatment sessions attended, and funding source were collected from billing records. Therapists were allowed to list a maximum of four diagnoses. Funding source was dichotomized into two categories: “Medi-Cal/unfunded” refers to children funded by California’s state Medicaid system and “AB2726” refers to state funding for students in California whose mental health problems are deemed to interfere with academic functioning (provided through California Assembly Bill 2726). Medication use was collected during phone interviews with parents at 4 and 8 month follow-up. Psychotherapeutic treatment process data were collected by video-taping and coding sessions as described below.
Observational Measure
PRAC Therapeutic Process Observational Coding System for Child Psychotherapy: Strategies Scale
The Therapy Process Observational Coding System for Child Psychotherapy (TPOCS-S) (McLeod 2001) was adapted for the PRAC study to characterize treatment strategies observed during psychotherapy sessions. Specifically, the TPOCS-Strategies section (TPOCS-S) was utilized to assess a wide variety of intervention strategies that are theoretically (e.g., cognitive-behavioral, psychodynamic, client-centered) and non-theoretically, or cross-theoretically, driven (e.g., assessing problems). The TPOCS-S content is primarily based on the Therapy Procedures Checklist (TPC) (Weersing et al. 2002) and the format is based on the Therapist Behavior Rating Scale (TBRS) (Hogue et al. 1996) treatment adherence measure.
The TPOCS-S was adapted for this study in collaboration with the PRAC Therapist Advisory Group (TAG: includes one therapist representative from each of the six participating clinics (see Garland et al. 2006b for full description)). The final revised PRAC TPOCS-S (Garland et al. 2008) includes 27 therapeutic strategies, 15 of which reflect therapeutic techniques (e.g., modeling, addressing client-therapist relationship) and 12 of which reflect therapeutic content (e.g., affect management, principles of positive reinforcement). The goal of the PRAC TPOCS-S is to describe the use of a wide array of potential strategies, not to measure fidelity to a narrowly specified treatment model (Garland et al. 2009). At the end of each session, the observer rates whether the individual strategies were observed to occur. If so, they also rate how intensively the strategy was delivered within the session. Use of each strategy was coded separately for strategies directed to children versus caregivers. Occurrence indicates whether the strategy was observed during a session. Intensity measures how actively and thoroughly a strategy was pursued throughout the session. Specifically, it reflects a combination of time spent on the strategy and the thoroughness with which it was pursued. For example, a low intensity rating on the strategy “problem-solving/social skills” would reflect a therapist addressing one aspect of problem-solving skills in a limited way, such as generating alternative solutions, but only for one particular experience the child or caregiver faced, and in a somewhat fleeting, or cursory manner. A high intensity rating would be assigned when the therapist thoroughly addresses the multiple steps in problem solving and generalization to multiple problems. “Occurrence/intensity” was rated at the end of each session for each strategy on a Likert scale of 0–6 (0 = did not occur; 1–2 = low intensity, 3–4 = medium intensity, 5–6 = high intensity).
While the PRAC TPOCS-S includes 27 individual psychotherapeutic strategies, this paper focuses exclusively on 9 strategies delivered to children and 6 delivered to caregivers. The 9 strategies directed to children and 6 of the strategies directed to caregivers were selected because they are conceptually consistent with strategies delivered in research-based behavioral and cognitive behavioral interventions for children across the autism spectrum. Two additional codes targeted to parents (Addressing child’s external care and addressing family issues/care) were selected because they reflect extra-therapeutic care that may be particularly relevant to this population and have been observed frequently in these settings (Zoffness et al. 2009). See Tables 2 and 3 for a list of PRAC TPOCS-S codes examined in the current study and “Appendix” for brief descriptions of the codes.
Table 2
Occurrence and intensity of psychotherapeutic strategies observed with children
Psychotherapeutic strategies with children | ASD group (n = 99 sessions) | Comparison group (n = 103 sessions) | ||
|---|---|---|---|---|
% Of sessions strategy observed | Intensity when used M (SD) | % Of sessions strategy observed | Intensity when used M (SD) | |
Behavioral techniques | ||||
Using positive reinforcement | 86.9 | 2.7 (1.4) | 82.5 | 2.7 (1.4) |
Using punishment/limit setting | 62.6 | 1.9 (1.1) | 54.5 | 2.5 (1.4) |
Cog. Behav. Content | ||||
Affect education | 79.8 | 2.5 (1.5) | 86.4 | 2.7 (1.4) |
Problem-solving/social skills | 57.6 | 2.7 (1.5) | 66.0 | 2.4 (1.4) |
Affect/anger management | 34.3 | 2.9 (1.5) | 42.7 | 2.8 (1.5) |
Teaching techniques | ||||
Psychoeducation | 77.8 | 2.9 (1.4) | 90.3 | 3.0 (1.4) |
Modeling | 47.5 | 2.3 (1.2) | 46.6 | 2.7 (1.6) |
Role-play/practice | 34.3 | 3.2 (1.6) | 37.9 | 3.4 (1.7) |
Assigning/reviewing homework | 18.2 | 3.1 (1.4) | 14.6 | 2.3 (1.2) |
Table 3
Occurrence and intensity of psychotherapeutic strategies observed with parents
Psychotherapeutic strategies with parents | ASD (n = 76 sessions) | Non-ASD (n = 72 sessions) | ||
|---|---|---|---|---|
% Of sessions strategy observed | Intensity when observed M (SD) | % Of sessions strategy observed | Intensity when observed M (SD) | |
Behavioral principles | ||||
Princ. of punishment/limit setting | 31.6 | 2.5 (1.4) | 36.1 | 2.4 (1.2) |
Princ. of positive reinforcement | 21.1 | 3.1 (1.2) | 26.4 | 2.5 (1.4) |
Teaching techniques | ||||
Psychoeducation | 82.9 | 3.5 (1.6) | 87.5 | 3.4 (1.6) |
Modeling | 25.0 | 2.8 (1.4) | 25.0 | 2.7 (1.4) |
Assigning/reviewing homework | 17.1 | 2.4 (1.0) | 8.3 | 1.8 (1.0) |
Role-play/practice | 11.8 | 2.2 (1.1) | 11.1 | 3.0 (1.8) |
Extra therapeutic care | ||||
Addressing child’s external care | 72.4 | 3.0 (1.5) | 73.6 | 3.7 (1.5) |
Addressing parent/family issues, care | 26.3 | 2.0 (1.3) | 52.8 | 2.4 (1.3) |
Inter-Rater Reliability of PRAC TPOCS-S
Of the 1,215 total coded sessions in the PRAC study, 379 (31%) were randomly selected for double-coding to test inter-rater reliability. The mean ICCs and kappa for the individual codes included in current analyses were .71 and .58. Reflecting acceptable reliability (Cichetti 1994) and similar to reliabilities reported in adult psychotherapy process observational research (Malik et al. 2003).
Child and Family Outcome Measures
The following measures were selected because they have established psychometrics and represent a variety of outcome domains (Brookman-Frazee et al. 2006).
Child Symptoms
Eyberg Child Behavior Problems Checklist (Eyberg and Pincus 1999)
The ECBI was used in this study to measure child symptom severity. It is a parent report measure which has been used in many treatment outcome studies for youths with behavior problems, ranging in age from 2 to 16. The ECBI includes 36 items, which are rated on a dichotomous Problem scale as well as a seven-point Intensity scale. Only the intensity scale is used in the current study. The psychometric characteristics of the EBCI are strong. The intensity scale has demonstrated a 3-week test–retest reliability coefficient of .86 (Robinson et al. 1980), internal consistency coefficients of .98 (Eyberg and Robinson 1983; Robinson et al. 1980) and convergent validity demonstrated with significant correlation coefficients of .75 with the Child Behavior Checklist Externalizing problem score among clinic-referred children (Boggs et al. 1990). The established clinical cutoff for the intensity score is 132 (Eyberg and Pincus 1999).
Parenting Practices
Consistent Discipline (Shelton et al. 1996)
This measure is the inconsistent discipline subscale of the Alabama Parenting Questionnaire. The scale includes 6 items which provide parent report on the frequency that inconsistent discipline is used in the home. All items are rated on a five-point scale, ranging from 1 (never) to 5 (always). The internal consistency of the inconsistent discipline subscale has been demonstrated with alphas of .69 for children aged 6–8, .55 for children aged 9–12, and .70 for adolescents aged 13–17 (Frick et al. 1999).
Family Functioning
Family Empowerment Scale (Koren et al. 1992)
The FES is a 34-item instrument that measures the broad construct of parent empowerment and provides three subscales of empowerment: Family, Service System, and Community/Political. The specific dimensions of empowerment measured in this scale include parents’ attitudes, knowledge, and behaviors regarding their children. Only the Family subscale was examined in the current study. It measures level of empowerment in the immediate situation at home. The FES has alpha coefficients ranging from .87 to .88, test–retest Pearson correlations from .77 to .85, and an overall kappa coefficient of .77 (Koren et al. 1992).
Caregiver Strain Questionnaire (Brannan et al. 1997)
The CGSQ is a 21-item scale that measures the impact of caring for a child with emotional and behavioral problems in six areas: economic burden, impact on family relations, disruption of family activities, impact on psychological adjustment of family members, stigma, anger and worry/guilt. The CGSQ yields 3 domains: objective strain, subjective externalized strain, and subjective internalized strain. A CGSQ Global score, representing the mean of all CGSQ items, was used to assess caregiver strain in the current study. In a community sample of children with mental health problems, the Cronbach’s alpha for the CGSQ was .94 (McCabe et al. 2003). Based on data collected at our center on 1,715 youths receive care in five community service sectors, CGSQ scores have correlated with parent impairment (.70 for CIS) and depression (.40 for CES-D), as well as youth impairment (.27 for CIS) and DISC-IV diagnosis (.50 for behavior disorders to .16 for mood and anxiety disorders).
Analysis Plan
Descriptive statistics were used to examine characteristics of children with ASD and to characterize treatment process and outcomes. To examine specificity of treatment delivered to children with ASD, comparisons between the ASD and non-ASD groups were run using t-tests and chi square analyses. Further, analyses of covariance (ANCOVA) were used to examine mean differences for 8-month follow-up scores on the three measures of outcome between the ASD group and the non-ASD comparison group, controlling for baseline outcome measures. Effect sizes were calculated to indicate the magnitude of the effects, given the relatively small sample size.
Results
Throughout this section, analyses of the ASD group only are presented first, followed by analyses comparing the ASD and Non-ASD groups.
Child/Family Socio-Demographic, Service Entry, and Clinical Characteristics
Table 1 presents sociodemographic, service entry, and clinical characteristics for children with ASD and the non-ASD comparison group. Independent samples t-tests and chi square analyses were used to examine differences on sociodemographic and service entry characteristics between the two groups (Differences between groups on diagnosis were not examined since they were selected based on diagnosis). There was a trend for group differences on child age. Specifically, children in the ASD group were younger than children in the non-ASD comparison group (t = 1.95, p = 0.06; Cohen’s d = 0.65). There was also a trend for group differences in funding source such that children in the ASD group were more likely to receive funding through the AB2726 (educational) program, whereas children without ASD were more likely to receive funding through Medi-Cal (χ2 = 3.89, p = .05; Phi = 0.32). There were no significant differences between the two groups on child racial/ethnic minority status, caregiver age, annual household income, referral source, and effects sizes were small (Cohen’s d ranged from −0.25 to 0.44; Phi ranged from 0.13 to 0.23).
Treatment Process
Service Use
Children with ASD attended an average of 16.3 sessions (SD = 9.5; range = 1–29) during the 8-month study period, with 73.7% (n = 14) still attending at 8 months. During the study period, 66.7% (n = 12/181) of the children with ASD received medication treatment for emotional/behavioral symptoms. The most common classes of drugs were stimulants (50.0%, n = 9), antidepressants (16.7%, n = 3), antipsychotic medications (11.1%, n = 2), and mood stabilizers (11.1%, n = 2). There were no significant differences between ASD and the non-ASD comparison group on number of sessions attended, the proportion who remained active in treatment at 8-month follow-up, or proportion who received medications and effect sizes were small (Cohen’s d = 0.23; Phi ranged from 0.02 to 0.20).
Session Participants
Children with ASD were present in almost all of the 103 coded psychotherapy sessions (96.1%) and caregivers were present in at least part of 73.8% of all coded sessions. There were no significant differences between the ASD and Non-ASD groups in the proportions of sessions in which the child or parent were present.
Treatment Strategies Observed (Directed to Children)
Table 2 shows the occurrence and average intensities (when the strategy was used) for PRAC TPOCS-S strategies directed to children for all coded tapes. For the ASD group, the rate of occurrence of each psychotherapeutic strategy directed to children varied. Strategies observed most frequently in sessions with children (>75% of sessions) included: using positive reinforcement (87%); affect education (80%); and psychoeducation (78%). The strategy that was observed to occur least frequently (<25% of sessions) was assigning or reviewing homework (18%). Average intensity ratings of the individual strategies ranged from Low to Moderate (M = 2.6). The strategy with the highest average intensity rating was role-play/practice (M = 3.2; “Moderate”) and the strategy with the lowest intensity was using punishment/limit setting (M = 1.9; “Low”).
Independent samples t-tests were then conducted to examine differences in observed strategy delivery between the ASD and non-ASD comparison group. For these analyses, in order to obtain one score for each child, session data were aggregated for each child by using the maximum combined occurrence/intensity score (possible range: 0–6) across all coded sessions for each PRAC TPOCS-S strategy (not shown in Table 2). Note: there were no significant differences in the average number of sessions coded per child between the ASD (M = 5.0) and non-ASD (M = 5.0) groups. No statistically significant differences between the groups were observed at the .05 level on any of the strategies delivered to children and effect sizes were generally small (Cohen’s d ranged from .08 to .47).
Treatment Strategies Observed (Directed to Parents)
Table 3 shows the occurrence and average intensities for strategies directed to parents. For the ASD group, the rate of occurrence of each psychotherapeutic strategy directed to parents also varied. The strategy observed most frequently in sessions with parents (>75% of sessions) was psychoeducation (83%). Strategies that were observed to occur infrequently (<25% of sessions) included: principles of positive reinforcement (21%); role-play/practice (12%) and assigning and reviewing homework (17%). Average intensity ratings of the individual strategies ranged from Low to Moderate (M = 2.7). The strategy with the highest average intensity rating was psychoeducation (M = 3.5; “Moderate”) and the strategy with the lowest intensity was addressing parent/family issues, care (M = 2.0; “Low”).
Similar to the analyses with strategies directed to children, independent samples t-tests were conducted to examine differences in strategy delivery to parents between the ASD and non-ASD comparison group, aggregated to the child level (not shown in Table 3). Of the eight parent strategies examined, the only significant group difference at the .05 level was on assigning and reviewing homework such that parents of children in the ASD group (M = 1.47; SD = 1.55) received more intensive delivery of this strategy than parents of children in the non-ASD comparison group (M = 0.44; SD = 0.89) (t = −2.37, p = 0.03; Cohen’s d = 0.93). Effect sizes for the other strategies delivered to parents were small (Cohen’s d ranged from .03 to .16) with the exception of addressing parent issues and care (Cohen’s d = .52) such that parents of children in the ASD group (M = 1.35; SD = 1.73) received less intensive delivery of this strategy than parents of children in the non-ASD group (M = 2.19; SD = 1.56).
Outcomes
Paired sample t-tests were used to compare scores on the four outcome measures at the 8-month follow-up interview to baseline scores for the ASD sample. See Table 4 for means and standard deviations for each outcome measure. Results indicate significant decreases on parent reported child problem severity (ECBI) from baseline to 8-month follow-up (t = 3.68; p < .01; Cohen’s d using original standard deviations = 0.65). It is important to note that 78% of the ASD group scored in the clinical range on the ECBI at the baseline interview, compared to 44% at 8-month follow-up. Parents of children with ASD also reported significantly lower caregiver strain on the CGSQ at 8-month follow-up compared to the baseline assessment (t = 3.12; p < .01; Cohen’s d = 0.72). There was a trend for increased family-related empowerment on the FES at follow-up (t = −2.08; p = .05; Cohen’s d = 0.60). There was no significant change on the measure of consistent discipline from baseline to follow up (t = −0.19; p > .05; Cohen’s d = 0.05).
Table 4
Means and standard deviations of baseline and 8-month follow-up on the Eyberg Child Behavior Inventory, Caregiver Strain Questionnaire, Family Empowerment Scale (Family subscale), and Consistent Discipline measures
Measure | ASD (n = 18) | Non-ASD (n = 15) | ||
|---|---|---|---|---|
Baseline | FU | Baseline | FU | |
Eyberg child behavior inventory (ECBI) intensity scale | ||||
M
| 153.5 | 131.9 | 153.5 | 130.4 |
SD | (32.2) | (34.4) | (45.1) | (39.5) |
Caregiver strain questionnaire (CGSQ) | ||||
M
| 2.7 | 2.2 | 2.9 | 2.3 |
SD | (0.8) | (0.6) | (0.9) | (0.8) |
Family empowerment scale (family)a
| ||||
M
| 44.2 | 47.4 | 46.1 | 47.9 |
SD | (6.0) | (4.7) | (7.4) | (6.4) |
Consistent discipline | ||||
M
| 14.0 | 13.8 | 14.8 | 13.4 |
SD | (4.5) | (5.0) | (4.9) | (3.7) |
A series of one-way between groups analyses of co-variance (ANCOVA) were conducted to examine differences in mean 8-month follow-up scores as a function of group (ASD and non-ASD comparison) while controlling for baseline score as a covariate. The assumptions of homogeneity of regression and variance were met for analyses of all outcomes measures. After controlling for baseline scores, no significant group differences were observed on the ECBI (F(1,30) = 0.03, p > .05, η2 = 0.001); the CGSQ (F(1,30) = 0.02, p > .05, η2 = 0.001); the FES-Family (F(1,28) = 0.06, p > .05, partial η2 = 0.002), or on the Consistent Discipline measure (F(1,28) = 0.36, p > .05, partial η2 = 0.013).
Discussion
This study examined treatment processes and outcome for a subgroup of children with ASD and disruptive behavior problems receiving care in community mental health clinics. It represents the first research reporting detailed observational data on community mental health services for children with ASD. Further, it provides important descriptive information about the characteristics of children with ASD and rich contextual data about a matched comparison group without ASD receiving care in this setting.
The children with ASD were an average of 8 years old and primarily male, with almost all having a high functioning (i.e., Asperger’s Disorder) or non-specific (i.e., PDD-NOS) diagnosis on record (i.e., DSM-IV 299.80). This high rate of Asperger’s/PDD-NOS diagnoses may be related the eligibility requirements of publicly-funded community mental health services in our county. That is, a primary diagnosis of autistic disorder is excluded for reimbursement in the publicly-funded mental health service system, as the MR/DD system (not the mental health system) is responsible for funding services for children with this diagnosis. Asperger’s Disorder and PDD-NOS, however, are not excluded diagnoses. Children in the ASD sample were also diagnostically complex, with over 70% labeled as having at least one co-morbid additional psychiatric diagnosis (most frequently ADHD). This rate is consistent with studies examining the prevalence of psychiatric problems in children with ASD (Leyfer et al. 2006b; Simonoff et al. 2008). Lastly, funding for a majority of the children with ASD was provided through the school district. This finding is consistent with national trends indicating that the education system plays a significant role in providing mental health services for children with a broad range of mental health problems (Roanes and Hoagwood 2000).
The observational data on treatment strategies delivered to children with ASD indicate that therapists do frequently deliver a number of strategies conceptually consistent with research-based behavioral methods (delivering positive reinforcement and punishment/limit setting) and certain cognitive behavioral strategies (problem-solving/social skills, affect education, affect management) (e.g., Rogers and Vismara 2008; Wood et al. 2009). However, these strategies were delivered, on average, with low to moderate intensity (i.e. thoroughness). These findings indicate that while therapists in community mental health clinics are targeting many of the same general areas as research-based child skill-building interventions, the interventions are not being delivered in the same manner. The observed occurrence of behavioral parent training strategies (e.g., covering operant conditioning principles) and active teaching strategies typically employed in research-based parent training programs (e.g., role-play/practice, homework) was low. Likewise, when these strategies were observed to be delivered with parents, average intensity was low. While the observational measure used in this study is not a measure of fidelity to a particular treatment model, the low intensity suggests that the strategies are not being delivered as thoroughly as would likely be present in a research-based treatment model. Taken together, these findings replicate results from the larger PRAC study which indicate that certain strategies conceptually consistent with evidence-based practices for children with disruptive behavior problems are delivered with some frequency in community mental health clinics, but they are not delivered very intensively (Garland et al. in press). Further, the findings are consistent with research on community early intervention services that indicate that there are gaps between research-based practices and those that are provided in the community (Stahmer 2007; Stahmer et al. 2005).
It is important to highlight that observed treatment process was very similar for children with and without ASD who are receiving care from the same (or very similar) providers. This finding suggests that therapists may not tailor treatment to the unique characteristics of children with ASD, which may be explained by the limited specialized ASD training reported by therapists in these settings (Brookman-Frazee 2009). It also supports the general lack of specificity of treatment strategies based on child primary diagnosis reported in analyses of the full PRAC study which indicate that variability in practice patterns is not associated with child diagnosis (Brookman-Frazee et al. 2009b).
Outcomes across a number of different domains were measured. Statistically significant improvements in child behavior problems, caregiver strain, and family-related empowerment were observed at 8 month follow-up and medium effect sizes were observed. Significant improvements in parent discipline practices were not seen. These findings provide preliminary evidence of the positive impacts of community services, particularly related to child behavior problems and parent functioning, however, the effect sizes are smaller than those reported in intervention trials (e.g., Wood et al. 2009). It is also important to interpret these findings with caution given that there was no control group. Thus, no conclusions can be drawn about the impact of treatment on these outcomes. More research is needed on the potential impact of treatment provided in this context on social and adaptive functioning. Further, examination of change in internalizing psychiatric symptoms (particularly anxiety) is warranted given that over one quarter of the ASD sample was diagnosed with an anxiety disorder.
This study adds to the literature in a number of different ways. First, it expands the literature on ASD services research. While much of the existing work in this area relies on large administrative datasets to describe the broad types of services that child with ASD receive (e.g., Mandell et al. 2005, 2006; Mandell and Palmer 2005; Ruble et al. 2005) this study provides detailed observational data on the nature of outpatient treatment actually delivered in treatment sessions in community practice. These data have important implications for bridging the research-practice gap. For example, they can be used as baseline data for future studies that attempt to implement research-based ASD interventions in community settings. Further, the findings provide a “roadmap” for specific treatment strategies that should potentially be the focus of therapist training. For example, the low intensity/thoroughness of all strategies delivered suggest that increasing the intensity with which strategies are delivered may be a primary focus. Further, the low frequencies of behavioral parent training and active teaching strategies (role-play/practice and homework) need to be explicitly targeted. This is particularly important given that many of the children also had ADHD and DBD diagnoses for which there is also strong empirical support for behavioral parent training. It is important to note that certain research-based strategies were observed relatively frequently, suggesting that there is some overlap between community services and research-based care and that therapists may be open to learning more about research-based strategies. Overall, the findings highlight the heterogeneity of usual care psychotherapy. While some aspects of usual care treatment resemble research-based models, it is not entirely consistent. The findings also suggest that using treatment as usual is an important control group for future interventions studies.
This study also adds to the research on mental health services more generally. To date, there has been limited attention to ASD in mental health services research (see Brookman-Frazee et al. 2009a). This study confirms that children with ASD are being served in community-based mental settings. That is, they represent almost 10% of children presenting with disruptive behavior problems sequentially recruited into the PRAC study. Since therapists in our community indicate that they have limited training on treatment for ASD, it suggests that efforts to improve mental health services overall should include some targeted training efforts for this population. This study also provides information about the characteristics of children with ASD served in general community mental health settings. This information is important because it facilitates our understanding of who is receiving these services. Lastly, this study provides information on the delivery of psychotherapeutic strategies for all children receiving care in these settings. Consistent with other analyses of the full PRAC study sample (e.g., Brookman-Frazee et al. 2009b), the data for this subgroup indicate that overall intensity of observed strategy delivery was relatively low for strategies directed towards both children and parents. Since evidence-based practices for most childhood disorders (including ASD) typically include thoroughly pursuing specific goals (i.e., intensity), this finding suggests the need for training interventions to strengthen therapists’ use of active and directive techniques to increase the thoroughness of strategies delivered.
Study Limitations
Some study limitations should be noted. First, the broader study from which these data were drawn was not focused exclusively on treatment for children with ASD. Thus, there are no standardized measures to validate ASD diagnoses. However, the sample does reflect children being treated in community mental health settings for ASD, as determined by the treating therapists, thus there is ecological validity. Use of these type of therapist-assigned diagnoses obtained through administrative data is a common method of identification in ASD services research (e.g., Mandell et al. 2002; Ruble et al. 2005). Co-occurring DSM diagnoses were also based on therapist-report and not validated by diagnostic measures. The total number of diagnoses that therapists were able to report was limited to four. Since the ASD diagnosis took up one of the possible diagnoses for the ASD group, the rates of co-occurring diagnoses for the two groups are not directly comparable. Further, although the observational measure used in this study was developed to capture a broad range of potential strategies, it was not specifically designed to be used to characterize treatment for children with ASD.
Related, information about the therapists’ experiences related to ASD is lacking. It is not known how many children on the therapists’ caseloads have an ASD. Likewise, information for ASD training for the participating therapists is lacking. Although this information is not available for those in the current study, we have collected this information from 100 therapists practicing in 9 community-based mental health clinics in San Diego County. Results indicate that only 5% of therapists consider themselves “ASD experts” and an average of 20.7% of therapists’ current caseloads are represented by children diagnosed with an ASD or suspected of having an ASD (Brookman-Frazee 2009). These data suggest that therapists in the current sample have limited ASD experience, but are regularly serving this population.
Another potential limitation relates to the generalizability of the therapist and child/family samples. This study was conducted in one large, geographically diverse county. While the distributions of therapist education level, gender, race/ethnicity, and trainee status is similar to other studies of community mental health providers (Glisson et al. 2008; Hawley and Weisz 2005), the characteristics of children with ASD in this sample may differ from children receiving community mental health services in privately funded mental health settings and/or other locations that may have different fiscal policy factors that impact service referrals for the Mental Health, Special Education, and MR/DD systems. Related, this study utilized a small sample of children ages 4–12 with ASD who were referred to treatment for disruptive behavior problems. Treatment process and outcomes for older children with ASD and those referred for other psychiatric problems (e.g., anxiety, depression) may be different.
This study is also limited by the small sample size. It may limit the power to detect differences between the ASD and comparison group. It is important to note that the lack of significant differences in these small samples is consistent with other analyses of the full PRAC sample which reveal no significant differences on treatment process based on child diagnosis (Brookman-Frazee et al. 2009b). Further, due to the size of the ASD group, we are unable to conduct analyses which examine potential predictors of treatment delivery or outcomes. It is possible that certain therapists are more likely to deliver strategies more intensively; however, analyses of the full sample indicate that there are few therapist characteristics associated with intensity (Brookman-Frazee et al. 2009b). It would also be interesting to determine whether certain strategies, or combinations of strategies, result in more positive changes in child or family outcomes. Based on preliminary analyses of the full PRAC sample, there are few predictors of positive outcomes.
Summary
In summary, these findings represent the first observational data characterizing treatment and outcomes for children with ASD receiving care in community-based mental health clinics. They contribute to our understanding of the current mental health service context for children with ASD, and identify specific discrepancies between research-based treatment approaches and common treatment in usual care. The next step in this line of research is to use this information to build and test therapist training programs to reduce these discrepancies, thus strengthening the integration of research-based treatment into community services and ultimately improving the effectiveness of community care for these children.
Acknowledgments
This work was supported by NIH R01MH66070 (A.G.) and K23MH077584 (L.B.F.). The authors thank Drs. Aubyn Stahmer and Amy Drahota for their feedback on this manuscript and Bill Ganger for assistance with data management.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
