Randomized Clinical Trial of Computerized and Clinician-Delivered CBT in Comparison With Standard Outpatient Treatment for Substance Use Disorders: Primary Within-Treatment and Follow-Up Outcomes

Drug and alcohol use are among the most costly public health problems in the United States (1). Limited availability, uptake, and fidelity of evidence-based treatments have led to increased interest in web-based interventions, which can provide greater accessibility and standardization as well as potential cost savings (2). Meta-analyses suggest a significant but modest effect of these approaches in decreasing substance use in varied populations (3, 4). However, interpretation is complex because of the varied level of rigor in the trials included, with common limitations including weak comparison conditions (waiting list or assessment only), inadequate treatment exposure, and low rates of follow-up (5). Moreover, evaluations of unguided “stand-alone” web-based interventions are often conducted in populations with less severe use disorders (nonclinical populations, risky drinkers), and they rarely conduct well-specified, rigorous comparisons with validated clinician-delivered versions of the same treatment (6).

We previously reported on the efficacy, durability, and cost-effectiveness of computer-based training for cognitive-behavioral therapy (CBT4CBT) as an add-on to standard treatment for substance use in outpatient and methadone maintenance settings (7–10). However, these trials did not address the efficacy of CBT4CBT alone, an important step in establishing its efficacy and utility in the health care system. Here, we describe primary outcomes from a randomized clinical trial evaluating CBT4CBT as a virtual stand-alone treatment as well as clinician-delivered cognitive-behavioral therapy (CBT), each compared with standard outpatient treatment for a heterogeneous group of individuals seeking treatment for substance use disorders. The primary hypothesis was that individuals assigned to receive either form of CBT (clinician-delivered or CBT4CBT) would reduce their substance use relative to those receiving standard treatment. Based on previous work (8, 10, 11), we also hypothesized that the effects of either form of CBT would be durable relative to treatment as usual through a 6-month follow-up.

Method

Participants

Participants were recruited from among individuals seeking treatment at the Substance Abuse Treatment Unit of the Connecticut Mental Health Center in New Haven between January 2012 and October 2016. Participants were English-speaking adults who met DSM-IV-TR criteria for current (past 30 days) cocaine, marijuana, opioid, or alcohol abuse or dependence. Exclusion criteria were minimized to facilitate recruitment of a broad and clinically representative outpatient sample; thus, individuals were excluded only if they had an untreated or unstable psychotic disorder or had current suicidal or homicidal ideation, could not read at a sixth-grade level, or had a legal case pending that resulted in inability to commit to 12 weeks of treatment.

As shown in Figure 1, of 191 individuals screened, 137 were eligible for the study. All participants provided written informed consent, as approved by the Yale University Human Investigations Committee. Participants were then randomly assigned in equal proportion to one of the three treatment conditions described below, using a computerized urn randomization program (12) to balance treatment groups with respect to gender, ethnicity (minority, nonminority), education level (less than high school, high school graduate), primary substance used (cocaine, marijuana, other), self-reported familiarity with computers (yes/no), and referral through the criminal justice system (yes/no).

Treatments

Participants in all three treatment conditions were offered standard ancillary services as needed, which included psychiatric, pharmacologic, and emergency services.

Standard treatment as usual.

Participants in the treatment-as-usual group were offered standard treatment at the clinic, which consisted of weekly group and/or individual therapy, as determined by the clinical team. Treatment as usual was implemented by 22 members of the clinic staff (four had doctoral degrees, 14 had master’s degrees, and four had bachelor’s degrees; 14 were female, and eight were male). Topics discussed at each group or individual session were recorded by the clinicians immediately after each session; the most frequent topics reported were motivational interviewing (N=91), life skills (N=60), relapse prevention (N=22), harm reduction (N=6), mindfulness (N=6), or women and trauma, health and recovery, or Latino recovery (N=3 for each).

Clinician-delivered CBT.

Participants assigned to this condition were offered 12 weekly individual sessions of manual-guided CBT (13), delivered by 15 doctoral-level clinicians or predoctoral-level fellows (six of them male and nine female) who were trained via a didactic seminar and a supervised training case, as described in previous CBT trials (11, 14). All CBT sessions were recorded; 104 of them (52%) were rated using a validated adherence and competence monitoring tool (15), and ongoing feedback was provided to clinicians by an expert supervisor. Ratings indicated high adherence and competence; the mean adherence score (possible ratings ranged from 1, did not occur, to 7, covered extensively and in great depth) for the six core CBT items (functional analysis, coping skills training, reviewing practice exercises, explaining CBT concepts, assigning homework, and agenda setting) was above 3 for all items, and the mean quality score (where possible ratings ranged from 1, very poor, to 7, outstanding) was above 4 for all six items.

CBT4CBT plus monitoring.

In this condition, participants were asked to complete one CBT4CBT module each week as their principal form of treatment, in conjunction with brief (∼10 minutes) in-person weekly clinical monitoring provided by a doctoral-level clinician. Monitoring sessions were manual guided (16) and followed guidelines for low-intensity interventions used in previous placebo-controlled trials (17, 18) and trials of Internet-delivered treatment (19). These were intended to evaluate participants’ current functional status and safety and to review their use of the CBT4CBT program. Three clinicians conducted the monitoring sessions (two had doctoral degrees, and one was a predoctoral fellow; one was male, and two were female). As described previously (7, 10), participants accessed CBT4CBT using a username and password. The program contains seven core CBT skill topics (modules) that include on-screen narration, graphic animation, quizzes, and other interactive exercises to teach and model effective use of skills. Each module presents videos demonstrating use of a targeted CBT skill and concludes with printable take-home practice exercises (homework).

Assessments

Participants were assessed before treatment, weekly during treatment, at the 12-week treatment termination point, and 1, 3, and 6 months after the termination point. The Structured Clinical Interview for DSM-IV-TR Axis I Disorders (SCID) (20) was administered to each participant before randomization to establish substance use and psychiatric diagnoses; the substance use section was readministered at treatment termination to assess changes in rates of meeting diagnostic threshold over time (21). The Substance Use Calendar, which is similar to the timeline follow-back (22, 23), was administered weekly during treatment to collect day-by-day self-reports of drug and alcohol use for the 28-day period before randomization, as well as at each follow-up interview. Self-reports of drug use were verified through urine toxicology screens, for which samples were obtained at every assessment visit. Breath alcohol samples were also collected at each visit. Participants were compensated for each assessment visit with gift cards ranging in value from $10 (weekly assessments) to $75 (final follow-up if all follow-up interviews were completed on time; participants could earn up to $285 in gift cards if all interviews were completed.

Correspondence of self-reports of recent drug use and results from urine toxicology screens was excellent, but varied by drug type. Of 1,378 urine samples collected during treatment (a mean of 10.2 samples per participant), 6.8% (N=94) indicated cocaine use when the participant denied recent use; 1.9% indicated opioid use when the participant denied recent use; 2.8% indicated benzodiazepine use when the participant denied recent use; and 10.5% indicated marijuana use when the participant denied use in the past 7–10 days, an interval reflecting the longer half-life of cannabis and its detectability in urine. This rate is consistent with previous trials of marijuana-using individuals in this setting, where rates of discrepancy have been 13% (24) and 16% (25).

Data Analysis

Power estimations were based on effect sizes of previous studies of CBT4CBT (7, 10) and clinician-delivered CBT (14), resulting in a target of 50 participants per condition (26). The primary outcome measure was change in self-reported frequency of substance use (operationalized as frequency of any drug or alcohol use, by week, from baseline through week 12), evaluated using random-effects regression analyses, in SPSS, version 24 (IBM; Armonk, N.Y.), with a simple linear model and a single random intercept and two contrasts testing the primary hypotheses (clinician-delivered CBT compared with treatment as usual, and CBT4CBT plus monitoring compared with treatment as usual) for the 137 participants assigned to treatment. Primary substance used (cocaine, marijuana, or alcohol) was included as a cluster variable to account for different patterns of use associated with different substance types (e.g., regular daily use of marijuana or alcohol versus binge patterns for cocaine) (27). Time was log-transformed to account for the expectation of greater change early in treatment.

The 6-month follow-up data were analyzed using the same contrasts, with piecewise random regression (28) to evaluate change from baseline through the 6-month follow-up by month and phase (within treatment versus follow-up). Analyses were repeated with the treatment-exposed sample (N=123) as well as those with adequate exposure to treatment (N=81). Results consistently paralleled the intent-to-treat analyses.

Because of the planned heterogeneity in drug and alcohol use in the sample, varying periods of detectability of different substances through urine monitoring (29, 30), and greater sensitivity to missing data (31), results of urine toxicology screens were secondary outcome measures and analyzed via analysis-of-variance models with the same contrasts as above; missing data were not imputed. Indicators of clinical significance (percentage of individuals who submitted urine specimens free of all drugs in the last 2 weeks of treatment, percentage who no longer met diagnostic threshold for abuse or dependence at the 12-week assessment) (21, 31, 32) were analyzed using chi-square models with the same contrasts, as were other secondary outcome measures (CBT knowledge and satisfaction with treatment). The trial was not powered for a direct comparison of CBT4CBT to clinician-delivered CBT (e.g., a noninferiority analysis), as there were no prior direct comparisons of computer-delivered and clinician-delivered CBT on which to base power calculations and estimations of confidence intervals (33, 34).

Results

Table 1 summarizes the baseline demographic and clinical characteristics for the 137 participants who underwent randomized treatment assignment. The sample was predominantly male (75%); 49% identified themselves as African American, 34% as Caucasian, and 8% as Latino or Latina. Most were unemployed, 75% reported that they had completed high school, and 35% were referred by the criminal justice system. The proportions of self-reported primary substance type were 49% for marijuana, 29% for cocaine, 19% for alcohol, 2% for opioids, and 1% for PCP. Most participants (81.8%) used both drugs and alcohol; 55% reported using at least two substances in the past month; 81% of participants submitted at least one urine sample before baseline assessment that was positive for at least one illicit drug.

Treatment Adherence, Retention, and Data Availability by Condition

Of the 137 individuals assigned to treatment, 123 completed at least one session of their assigned treatment (90%). As shown in Table 2, treatment retention was significantly higher in the CBT4CBT condition (a mean of 62 days completed, of 84), lowest in the clinician-delivered CBT condition (43 days), and intermediate in the treatment-as-usual condition (55 days). Number of urine specimens collected also differed significantly by treatment (a mean of 8.0 for the clinician-delivered CBT group, 10.5 for the treatment-as-usual group, and 12.0 for the CBT4CBT group).

Study treatments comprised different components (i.e., group and individual sessions, CBT4CBT modules) and differed across groups, and treatment exposure varied across groups, with a mean of 4.1 individual CBT sessions in the clinician-delivered CBT group, 5.6 individual or group sessions in the treatment-as-usual group, and 6.8 brief individual monitoring sessions in the CBT4CBT group. Participants in the CBT4CBT group also completed a mean of 5.5 modules of the seven modules offered, which is comparable to previous CBT4CBT studies (7, 10, 35, 36). The number of CBT homework assignments completed did not differ by CBT condition.

Rates of serious adverse events are also listed in Table 2. One patient in the clinician-delivered CBT condition died by suicide (institutional review concluded that the suicide did not appear to be related to treatment received), and two patients were withdrawn from the study (one was hospitalized for 5 days for suicidal ideation, the other was referred for a 30-day inpatient treatment stay for substance abuse). Rates of other serious adverse events did not differ by treatment condition, either during the treatment phase or during the 6-month follow-up.

At treatment termination (12-week assessment), data were collected from 120 participants (88% of the intent-to-treat sample and 90% of the treatment-exposed sample). During the follow-up period, 84% of the intent-to-treat sample was reached for at least one follow-up interview, and 79% were reached for the 6-month follow-up interview. Rates of assessment completion at treatment termination significantly differed by treatment condition (χ²=6.44, p=0.04), with contrasts indicating lower rates for clinician-delivered CBT compared with treatment as usual (Wald χ²=3.72, p=0.05), but were not significantly different for the 1-, 3-, or 6-month follow-up interviews. Overall level of data missingness was significantly higher for the clinician-delivered CBT condition than the other two conditions (Wald χ²=6.6, p=0.04).

Effects of Study Treatment on Substance Use Outcomes During Treatment and Follow-Up

Results of random-effects regression analyses for the primary outcome measure (days of any drug or alcohol use by week) are presented in Table 3 and illustrated in Figure 2. For the intent-to-treat sample, analyses of data collected during the treatment phase indicated reduction in frequency of any substance use over time by week for the whole sample during the 12-week treatment period (effect for time, t=−4.61, df=1, 999, p<0.001) and also confirmed the two primary hypotheses: greater reductions in frequency of any drug or alcohol use over time for clinician-delivered CBT compared with treatment as usual (t=–3.41, df=1, 1019, p<0.01) and for CBT4CBT plus monitoring compared with treatment as usual (t=−2.26, df=1, 996, p=0.02). Results were similar regardless of sample examined (all participants assigned to treatment including data after dropout, treatment initiators, treatment exposed, or excluding participants whose primary substance was not marijuana, cocaine, or alcohol) and regardless of how primary substance was modeled (e.g., included as a random factor or ignored).

Follow-up data are also illustrated in Figure 2. Analyses indicate an overall effect of time, as participants as a group reduced their frequency of drug or alcohol use from the start of treatment to the end of follow-up by month (effect for time, t=−4.26, df=1, 1044, p<0.01) but with the effect of phase (during treatment versus during follow-up) falling short of statistical significance (effect for phase, t=1.65, df=1, 1033, p=0.10). The effect for the contrast of CBT4CBT plus monitoring compared with treatment as usual was significant, indicating sustained effects over time of CBT4CBT relative to treatment as usual (t=−2.02, df=1, 1040, p=0.04), but the effect of clinician-delivered CBT compared with treatment as usual was not significant when follow-up data were included.

Secondary Substance Use Outcomes Within Treatment and Follow-Up

Secondary outcomes are presented in Table 2. Among the participants who reported drug use at baseline (N=132), the percentage of drug-free urine specimens was highest in the CBT4CBT plus monitoring group (37%), lowest in the clinician-delivered CBT group (33.1%), and intermediate in the treatment-as-usual group (34.3%), but these differences were not statistically significant. Effects were significant for cocaine-negative urine specimens for both the sample as a whole and for those who reported cocaine as their primary substance, with those assigned to CBT4CBT plus monitoring submitting a significantly higher proportion of cocaine-negative urine specimens than those assigned to treatment as usual. Rates of positive breath-alcohol tests were low and did not differ by treatment condition.

In terms of indicators of clinical significance, the percentage of participants with no urine specimens testing positive for drugs in the last 2 weeks of treatment favored CBT4CBT plus monitoring (34%) over treatment as usual and clinician-delivered CBT (both 18%), a difference that fell short of significance (p=0.09). Rates of individuals no longer meeting DSM-IV-TR diagnostic threshold for current substance dependence at treatment termination also favored CBT4CBT plus monitoring (66.7%) over clinician-delivered CBT (51.6%) and treatment as usual (42.9%), which also fell short of significance (p=0.06).

Results evaluating the self-reported percentage of days abstinent during follow-up were largely consistent with the primary random-effects regression analyses, indicating the highest percentage of days abstinent reported in the CBT4CBT plus monitoring condition, but the difference was not statistically significant. Results of urine toxicology screens collected at each follow-up visit indicated a significantly higher proportion of drug-negative urine samples for participants in the CBT4CBT plus monitoring condition compared with those in the treatment-as-usual condition at the 3-month follow-up (CBT4CBT group, 60.9%; treatment-as-usual group, 33.3%; Wald χ²=4.0, p=0.04); this effect was not significant at the final 6-month follow-up (CBT4CBT group, 50.0%; treatment-as-usual group, 30.9%; Wald χ²=2.6, p=0.11).

Knowledge of CBT Concepts and Treatment Satisfaction

A 40-item true/false test assessing basic knowledge of cognitive and behavioral concepts (“Everyone’s triggers are the same,” “It’s always best to trust your gut when thinking about a problem”) was added after the trial began. Fifty-two participants completed it at baseline and at treatment termination. Participants as a whole increased their scores over time (time, F=8.04, p<0.01); those assigned to CBT4CBT plus monitoring had the largest gain in percent correct over time (mean scores at treatment termination: clinician-delivered CBT group, 65%; treatment-as-usual group=72%; CBT4CBT group, 81%; group-by-time interaction, F=4.32, p=0.02).

A treatment satisfaction form that was validated in previous studies (7, 37) was administered at the treatment termination interview to assess satisfaction with treatment overall and with specific aspects. For the question “Overall, how satisfied are you with the treatment you received?” a larger proportion of participants assigned to CBT4CBT plus monitoring responded with the highest possible level (“very satisfied”) (82.4%) compared with those assigned to clinician-delivered CBT (63.9%) or treatment as usual (60.0%), although the difference fell short of significance (χ²=4.8, p=0.09). Similarly, for the question “Overall, how would you describe your condition at present?” more individuals assigned to CBT4CBT plus monitoring responded with the highest possible level (“excellent”) (44.1%) compared with those assigned to clinician-delivered CBT (19.4%) or treatment as usual (28.9%), with the difference again falling short of significance (χ²=5.1, p=0.08). Satisfaction with amount of treatment received did not differ significantly across treatment groups (“very satisfied” with amount of treatment: clinician-delivered CBT group, 55.6%; treatment-as-usual group, 57.8%; CBT4CBT group, 58.8%; χ²=0.08, p=0.96), and neither did satisfaction with their clinician (“very satisfied with clinician”: clinician-delivered CBT group, 72.2%; treatment-as-usual group, 80.0%; CBT4CBT group, 88.2%; χ²=2.8, p=0.25).

Discussion

This randomized clinical trial evaluating a web-based CBT intervention in a heterogeneous sample of treatment-seeking substance users found that those assigned to either CBT4CBT with minimal clinical monitoring or clinician-delivered CBT had greater reductions in frequency of any drug or alcohol use compared with standard treatment. A 6-month follow-up demonstrated continuing efficacy for CBT4CBT compared with treatment as usual, but not for clinician-delivered CBT compared with treatment as usual. Multiple secondary outcomes favored CBT4CBT plus monitoring, as did indicators of clinical significance, such as a greater percentage of participants no longer meeting DSM-IV-TR criteria for current substance dependence at the end of treatment.

This is, to our knowledge, the first randomized clinical trial to evaluate a web-based intervention delivered with minimal monitoring for individuals with DSM substance use disorders within a treatment-seeking clinical sample. Trials of this type are rare (5) yet essential for validating web-based approaches as well as for realizing the promise of these approaches to reduce the “treatment gap” between the large proportion of individuals in need of evidence-based services and the limited number who actually receive them (38).

The results strongly support the safety, feasibility, and efficacy for CBT4CBT provided with minimal clinical monitoring. Participants assigned to this condition consistently achieved the best outcomes in terms of treatment retention, engagement, and substance use in comparison to an active control condition. Although a direct comparison (i.e., noninferiority) was not tested here, CBT4CBT plus monitoring appeared to outperform clinician-delivered CBT on all outcomes evaluated. There were no indications that CBT4CBT plus monitoring was not “at least as good” as clinician-delivered CBT; in addition to greater reductions in substance use and indicators of clinical significance, those assigned to CBT4CBT plus monitoring showed the greatest increase in knowledge of CBT concepts and were most likely to report the highest levels satisfaction with treatment. This computerized version of CBT thus appears to be an engaging and attractive approach for persons with substance use disorders (39).

While those assigned to clinician-delivered CBT did show greater reductions in substance use compared with those assigned to treatment as usual, this treatment condition had the poorest level of treatment retention and engagement as well as the lowest rates of abstinence during the follow-up period. This was unexpected, given that one of the distinguishing features of CBT is the relative durability of its effects (40, 41). Despite well-trained clinicians with high-quality delivery, participants assigned to clinician-delivered CBT dropped out of treatment sooner, had a greater number of withdrawals from treatment, and had the lowest rates of follow-up data collected. The reasons for this are not clear. It may be that weekly one-on-one CBT was too demanding for patients in this population, many of whom were referred to treatment by the criminal justice system.

Strengths of this trial include rigorous methodological features consistent with those for clinician-delivered therapies (42), including urn randomization, SCID-based diagnosis for inclusion, primary self-report outcome with biological verification, close monitoring of treatment delivery, and rates of follow-up data collection from >80% of the intent-to-treat sample. Inclusion of a broad range of substance use, with most participants (82%) reporting both alcohol and drug use, enhances the generalizability of the findings. However, although this is one of the first trials to include both a virtual stand-alone computerized CBT and clinician-delivered CBT, the study was not powered to directly contrast these two conditions; thus, it cannot be concluded that the effects of CBT4CBT plus monitoring were equivalent or superior to clinician-delivered CBT. Patients in this heterogeneous sample of “all comers” were treated with an array of medications (see Table S1 in the online supplement), but these did not vary by treatment. The differential rate of attrition across treatment conditions limits the inferences that can be drawn regarding the secondary substance use outcomes, as these were evaluated using the intent-to-treat sample regardless of level of treatment exposure. In sum, this study provides strong support for CBT4CBT as an efficacious treatment for substance use, even when offered with limited clinical contact. Web-based CBT4CBT not only may broaden access to an evidence-based treatment, but it also may be a more appealing option for many individuals.