Interpreting treatment effects when cases are institutionalized after treatment☆
Introduction
Drug treatment clients are a population at particularly high risk of institutionalization, defined here as spending a day or more in a controlled environment where the possibility of drug use and criminal activity is substantially diminished (e.g., a jail, prison, hospital, residential treatment or group home setting). This is evident in large samples of drug treatment clients. For instance, in the Drug Abuse Treatment Outcomes Study (Hubbard et al., 1997), 40% of the 2966 clients of U.S. substance abuse treatment programs interviewed 12 months after discharge reported institutionalization for some part of the preceding year (U.S. Department of Health and Human Services, National Institute on Drug Abuse, 2004). Among those with any institutionalization, the average number of days institutionalized out of the past 365 was 115 (U.S. Department of Health and Human Services, National Institute on Drug Abuse, 2004). Similarly, over 2600 cases (about 52% of the sample) from the National Treatment Improvement Evaluation (NTIES; U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment, 2004; Gerstein et al., 1997) were institutionalized during the study's 12-month post-treatment evaluation. Of these cases, more than 630 were incarcerated for the entire evaluation period and excluded from analyses.
Because institutionalization limits people's freedoms, it can cause apparent improvement in many of the most important substance abuse treatment outcomes, such as reductions in drug and alcohol use, drug problems, crime, and even psychological problems (Piquero et al., 2001, Webb et al., 2002). Similarly, it can lead to seeming improvements in outcomes such as participation in educational programs or access to health services. However, since institutionalization is often not a positive outcome, these seemingly positive results can lead to misleading inferences about the benefits of alternative treatments.
Consider, for instance, a hypothetical experiment in which drug users randomly assigned to Treatment A are later found to have higher rates of abstinence, but also higher rates of institutionalization than those assigned to Treatment B. This pattern of findings raises the possibility that Treatment A is reducing drug use only by virtue of its effect on institutionalization. This may be unsatisfactory for at least two reasons. First, some types of institutionalization (e.g., prison) may represent a worsening of the clients’ conditions, not improvement, at a substantial societal cost. In this case, the positive effect on drug use might actually be a side effect of an otherwise costly negative treatment effect. Second, most stakeholders (clients, payers, referrers) seek treatments that will reduce clients likelihood of using substances when free in the community, rather than while institutionalized. As such, rates of drug use during periods of institutionalization actually obscure the effect of interest.
The perspective developed above suggests that for the purpose of understanding most treatment effects (e.g., the differential effects of a single intervention on population subgroups) and providing information needed by stakeholders, estimated rates of drug use (and many other outcomes) should disentangle improvement due to less use among patients when they are free in the community from reductions in use due to institutionalization. For example, we might estimate rates that would be expected if every client was at risk (i.e., not institutionalized) for the entire period of observation. Stated another way, we should try to answer the question, “What would the effects of treatment be if none of its recipients had been institutionalized?”
Fig. 1 demonstrates the problem in a common graphical model of direct and indirect effects (MacKinnon et al., 2000). The figure shows that treatment affects institutionalization which in turn affects substance use or other outcomes and also shows that treatment has a direct effect on outcomes. It is the direct link from treatment to outcomes which is of primary interest to stakeholders and others. For clarity, the figure excludes external factors such as deviance that can influence treatment outcomes and institutionalization. However, it illustrates the important point that treatment affects institutionalization, thereby complicating the evaluation of the effectiveness of treatment.
This paper makes precise the notions of the effect represented by the direct link from treatment to outcomes, and it discusses estimation of this effect. Because institutionalization occurs post-treatment, estimating such an effect can be challenging. The paper also discusses the four most common of these approaches and identifies the often tacit assumptions required for each approach to recover effects of interest. The methods considered are (1) ignoring institutionalization; (2) combining the outcome of interest and institutionalization into a single measure that is used to assess treatment effects; (3) dropping institutionalized cases from the study without any adjustment for the censoring of the population this creates; and (4) controlling for institutionalization with statistical models such as linear regression, path or structural equation models. This paper's focus on providing technical details of statistical models and analytic methods may make it of greatest interest to methodologists; however, the issues surrounding inferences about treatment effectiveness in the presence of institutionalization during the evaluation should be of concern to all treatment researchers.
The next section describes the dataset used to illustrate the effects of different approaches to the institutionalization confound. Section 3 develops a general statistical model for causal effect analyses given post-treatment confounders such as institutionalization, and uses this model to highlight the assumptions and limitations of each of the most common approaches to addressing these confounds. To aid readers unfamiliar with the potential outcomes framework used in the causal models throughout Section 3, verbal descriptions of the statistical model are presented first, followed by formal notation, which is used for precision. The paper closes with a discussion of strategies for investigating treatment effects in the presence of post-treatment confounders.
Section snippets
Empirical example: the effects of treatment modality on adolescent outcomes
To demonstrate the effects of different approaches to addressing institutionalization in outcomes analyses, we use a study of the effects of treatment modality (residential versus outpatient) on the 12-month substance use outcomes for adolescents who participated in the Adolescent Treatment Models (ATM) study, fielded between 1998 and 2002 by the Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment. The ATM study collected treatment admission and
Causal effects of treatment in the presence of institutionalization
To understand the impact of institutionalization on estimators of treatment effects, we need a precise definition of a treatment effect that is consistent with the needs of stakeholders and the inferences they make from common estimators. That is, we need a precise definition of the direct link from treatment to outcomes in Fig. 1. We start by considering the treatment effect in the simple case without any institutionalization. In this case, we are interested in the change in a youth's outcome
Discussion
Institutionalization at follow-up is particularly common in substance abuse treatment studies, and can present challenging confounds for the calculation of unbiased treatment effects in either experimental or observational studies. Although some amount of institutionalization exists in nearly all long-term follow-up studies, the most common approach to its confounding effects on outcomes has been to ignore them. While the resulting treatment effect estimate does provide a valid causal effect
References (29)
Global Appraisal of Individual Needs (GAIN) Manual: Administration, Scoring and Interpretation
(1999)- Dennis, M.L., Babor, T.F., Diamond, G., Donaldson, J., Godley, S.H., Titus, J.C., et al., 2000. The Cannabis Youth...
- et al.
A dual experiment in intensive community supervision: Minnesota's prison diversion and enhanced supervied release programs
Prison J.
(1995) - et al.
An Introduction to the Bootstrap
(1993) - et al.
Principal stratification in causal inference
Biometrics
(2002) - Gerstein, D.R., Datta, A.R., Ingels, J.S., Johnson, R.A., Rasinski, K.A., et al., 1997. NTIES. National Treatment...
- et al.
Adolescents and Young Adults in the National Treatment Improvement Evaluation Study [National Evaluation Data Services]
(1999) - et al.
The national treatment outcome research study (NTORS): 4–5 year follow-up results
Addiction
(2003) Econometric Analysis
(2003)Statistics and causal inference
J. Am. Stat. Assoc.
(1986)
An evaluation of drug treatments for adolescents in 4 US cities
Arch. Gen. Psychiatry
Overview of 1-year follow-up outcomes in the drug abuse treatment outcome study DATOS
Psychol. Addict. Behav.
Equivalence of the mediation, confounding and suppression effect
Prev. Sci.
Cited by (14)
Multidimensional Family Therapy as a community-based alternative to residential treatment for adolescents with substance use and co-occurring mental health disorders
2018, Journal of Substance Abuse TreatmentCitation Excerpt :Youth in this study were at high risk for being placed in a long-term juvenile justice or residential substance use treatment facility at some time during the study assessment window because of the severity of substance use symptoms and delinquency, number of psychiatric diagnoses, and the number of previous substance use treatment placements. McCaffrey, Morral, Ridgeway, and Griffin (2007) caution that behavioral frequency data such as TLFB-assessed substance use and the number of delinquent acts committed as measured by the SRD are subject to selection and suppression effects when placement in a controlled environment such as prison/jail or a residential substance use or mental health treatment facility is not taken into account. Therefore, we treated TLFB and SRD outcomes differently than other outcomes that were less susceptible to such biases (i.e., substance use problem severity as measured by the PEI and externalizing and internalizing symptoms as measured by the YSR).
The effectiveness of community-based delivery of an evidence-based treatment for adolescent substance use
2012, Journal of Substance Abuse TreatmentCitation Excerpt :In our sample, attrition weights are not appropriate because it is highly unlikely that the outcomes among those who responded could be extrapolated to draw inferences about the outcomes among those who did not. Principal stratification, on the other hand, is a model-based approach that works best when one has covariates that are highly predictive of loss to follow-up and that improve both precision and identifiably in the model fit (Griffin, McCaffrey, & Morral, 2008; McCaffrey, Morral, Ridgeway, & Griffin, 2007); in our case, such covariates were not found. We also examined outcome data collected at the 6-month interview to potentially address the differential follow-up concern.
Substance use treatment outcomes in a sample of male serious juvenile offenders
2009, Journal of Substance Abuse TreatmentCitation Excerpt :Second, adolescents in the justice system are differentially housed in controlled environments (such as prison, detention, or residential treatment) for varying periods, and their access to substances (and opportunities for criminal offending) vary as a function of their exposure to these restricted settings (Piquero et al., 2001). Most previous studies have simply ignored the effects of placement in controlled environments, which may overestimate treatment effects (McCaffrey, Morral, Ridgeway, & Griffin, 2007). Although there is no perfect solution to this problem, the current analyses extended prior research by including the proportion of “supervised” time (PST; i.e., residence in a controlled environment) at each time-point as a covariate in the models, as well as testing whether treatment effects varied as a function of time spent in controlled settings.
The longitudinal association between substance use and delinquency among high-risk youth
2008, Drug and Alcohol DependenceAdolescents' Thoughts About Abstinence Curb the Return of Marijuana Use During and After Treatment
2009, Journal of Consulting and Clinical Psychology
- ☆
This research was support by NIDA Grants R01 DA015697, R01 DA016722 and R01 DA017507.