Interpreting treatment effects when cases are institutionalized after treatment

https://doi.org/10.1016/j.drugalcdep.2006.12.032Get rights and content

Abstract

Drug treatment clients are at high risk for institutionalization, i.e., spending a day or more in a controlled environment where their freedom to use drugs, commit crimes, or engage in risky behavior may be circumscribed. For example, in recent large studies of drug treatment outcomes, more than 40% of participants were institutionalized for a portion of the follow-up period. When longitudinal studies ignore institutionalization at follow-up, outcome measures and treatment effect estimates conflate treatment effects on institutionalization with effects on many of the outcomes of interest. In this paper, we develop a causal modeling framework for evaluating the four standard approaches for addressing this institutionalization confound, and illustrate the effects of each approach using a case study comparing drug use outcomes of youths who enter either residential or outpatient treatment modalities. Common methods provide biased estimates of the treatment effect except under improbable assumptions. In the case study, the effect of residential care ranged from beneficial and significant to detrimental and significant depending on the approach used to account for institutionalization. We discuss the implications of our analysis for longitudinal studies of all populations at high risk for institutionalization.

Introduction

Drug treatment clients are a population at particularly high risk of institutionalization, defined here as spending a day or more in a controlled environment where the possibility of drug use and criminal activity is substantially diminished (e.g., a jail, prison, hospital, residential treatment or group home setting). This is evident in large samples of drug treatment clients. For instance, in the Drug Abuse Treatment Outcomes Study (Hubbard et al., 1997), 40% of the 2966 clients of U.S. substance abuse treatment programs interviewed 12 months after discharge reported institutionalization for some part of the preceding year (U.S. Department of Health and Human Services, National Institute on Drug Abuse, 2004). Among those with any institutionalization, the average number of days institutionalized out of the past 365 was 115 (U.S. Department of Health and Human Services, National Institute on Drug Abuse, 2004). Similarly, over 2600 cases (about 52% of the sample) from the National Treatment Improvement Evaluation (NTIES; U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment, 2004; Gerstein et al., 1997) were institutionalized during the study's 12-month post-treatment evaluation. Of these cases, more than 630 were incarcerated for the entire evaluation period and excluded from analyses.

Because institutionalization limits people's freedoms, it can cause apparent improvement in many of the most important substance abuse treatment outcomes, such as reductions in drug and alcohol use, drug problems, crime, and even psychological problems (Piquero et al., 2001, Webb et al., 2002). Similarly, it can lead to seeming improvements in outcomes such as participation in educational programs or access to health services. However, since institutionalization is often not a positive outcome, these seemingly positive results can lead to misleading inferences about the benefits of alternative treatments.

Consider, for instance, a hypothetical experiment in which drug users randomly assigned to Treatment A are later found to have higher rates of abstinence, but also higher rates of institutionalization than those assigned to Treatment B. This pattern of findings raises the possibility that Treatment A is reducing drug use only by virtue of its effect on institutionalization. This may be unsatisfactory for at least two reasons. First, some types of institutionalization (e.g., prison) may represent a worsening of the clients’ conditions, not improvement, at a substantial societal cost. In this case, the positive effect on drug use might actually be a side effect of an otherwise costly negative treatment effect. Second, most stakeholders (clients, payers, referrers) seek treatments that will reduce clients likelihood of using substances when free in the community, rather than while institutionalized. As such, rates of drug use during periods of institutionalization actually obscure the effect of interest.

The perspective developed above suggests that for the purpose of understanding most treatment effects (e.g., the differential effects of a single intervention on population subgroups) and providing information needed by stakeholders, estimated rates of drug use (and many other outcomes) should disentangle improvement due to less use among patients when they are free in the community from reductions in use due to institutionalization. For example, we might estimate rates that would be expected if every client was at risk (i.e., not institutionalized) for the entire period of observation. Stated another way, we should try to answer the question, “What would the effects of treatment be if none of its recipients had been institutionalized?”

Fig. 1 demonstrates the problem in a common graphical model of direct and indirect effects (MacKinnon et al., 2000). The figure shows that treatment affects institutionalization which in turn affects substance use or other outcomes and also shows that treatment has a direct effect on outcomes. It is the direct link from treatment to outcomes which is of primary interest to stakeholders and others. For clarity, the figure excludes external factors such as deviance that can influence treatment outcomes and institutionalization. However, it illustrates the important point that treatment affects institutionalization, thereby complicating the evaluation of the effectiveness of treatment.

This paper makes precise the notions of the effect represented by the direct link from treatment to outcomes, and it discusses estimation of this effect. Because institutionalization occurs post-treatment, estimating such an effect can be challenging. The paper also discusses the four most common of these approaches and identifies the often tacit assumptions required for each approach to recover effects of interest. The methods considered are (1) ignoring institutionalization; (2) combining the outcome of interest and institutionalization into a single measure that is used to assess treatment effects; (3) dropping institutionalized cases from the study without any adjustment for the censoring of the population this creates; and (4) controlling for institutionalization with statistical models such as linear regression, path or structural equation models. This paper's focus on providing technical details of statistical models and analytic methods may make it of greatest interest to methodologists; however, the issues surrounding inferences about treatment effectiveness in the presence of institutionalization during the evaluation should be of concern to all treatment researchers.

The next section describes the dataset used to illustrate the effects of different approaches to the institutionalization confound. Section 3 develops a general statistical model for causal effect analyses given post-treatment confounders such as institutionalization, and uses this model to highlight the assumptions and limitations of each of the most common approaches to addressing these confounds. To aid readers unfamiliar with the potential outcomes framework used in the causal models throughout Section 3, verbal descriptions of the statistical model are presented first, followed by formal notation, which is used for precision. The paper closes with a discussion of strategies for investigating treatment effects in the presence of post-treatment confounders.

Section snippets

Empirical example: the effects of treatment modality on adolescent outcomes

To demonstrate the effects of different approaches to addressing institutionalization in outcomes analyses, we use a study of the effects of treatment modality (residential versus outpatient) on the 12-month substance use outcomes for adolescents who participated in the Adolescent Treatment Models (ATM) study, fielded between 1998 and 2002 by the Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment. The ATM study collected treatment admission and

Causal effects of treatment in the presence of institutionalization

To understand the impact of institutionalization on estimators of treatment effects, we need a precise definition of a treatment effect that is consistent with the needs of stakeholders and the inferences they make from common estimators. That is, we need a precise definition of the direct link from treatment to outcomes in Fig. 1. We start by considering the treatment effect in the simple case without any institutionalization. In this case, we are interested in the change in a youth's outcome

Discussion

Institutionalization at follow-up is particularly common in substance abuse treatment studies, and can present challenging confounds for the calculation of unbiased treatment effects in either experimental or observational studies. Although some amount of institutionalization exists in nearly all long-term follow-up studies, the most common approach to its confounding effects on outcomes has been to ignore them. While the resulting treatment effect estimate does provide a valid causal effect

References (29)

  • M.L. Dennis

    Global Appraisal of Individual Needs (GAIN) Manual: Administration, Scoring and Interpretation

    (1999)
  • Dennis, M.L., Babor, T.F., Diamond, G., Donaldson, J., Godley, S.H., Titus, J.C., et al., 2000. The Cannabis Youth...
  • E. Deschenes et al.

    A dual experiment in intensive community supervision: Minnesota's prison diversion and enhanced supervied release programs

    Prison J.

    (1995)
  • B. Efron et al.

    An Introduction to the Bootstrap

    (1993)
  • C.E. Frangakis et al.

    Principal stratification in causal inference

    Biometrics

    (2002)
  • Gerstein, D.R., Datta, A.R., Ingels, J.S., Johnson, R.A., Rasinski, K.A., et al., 1997. NTIES. National Treatment...
  • D.R. Gerstein et al.

    Adolescents and Young Adults in the National Treatment Improvement Evaluation Study [National Evaluation Data Services]

    (1999)
  • M. Gossop et al.

    The national treatment outcome research study (NTORS): 4–5 year follow-up results

    Addiction

    (2003)
  • W.H. Greene

    Econometric Analysis

    (2003)
  • P.W. Holland

    Statistics and causal inference

    J. Am. Stat. Assoc.

    (1986)
  • Y.I. Hser et al.

    An evaluation of drug treatments for adolescents in 4 US cities

    Arch. Gen. Psychiatry

    (2001)
  • R.L. Hubbard et al.

    Overview of 1-year follow-up outcomes in the drug abuse treatment outcome study DATOS

    Psychol. Addict. Behav.

    (1997)
  • Imbens, G., 2003. Nonparametric estimation of average treatment effects under exogeneity: a review. Technical Working...
  • D.P. MacKinnon et al.

    Equivalence of the mediation, confounding and suppression effect

    Prev. Sci.

    (2000)
  • Cited by (14)

    • Multidimensional Family Therapy as a community-based alternative to residential treatment for adolescents with substance use and co-occurring mental health disorders

      2018, Journal of Substance Abuse Treatment
      Citation Excerpt :

      Youth in this study were at high risk for being placed in a long-term juvenile justice or residential substance use treatment facility at some time during the study assessment window because of the severity of substance use symptoms and delinquency, number of psychiatric diagnoses, and the number of previous substance use treatment placements. McCaffrey, Morral, Ridgeway, and Griffin (2007) caution that behavioral frequency data such as TLFB-assessed substance use and the number of delinquent acts committed as measured by the SRD are subject to selection and suppression effects when placement in a controlled environment such as prison/jail or a residential substance use or mental health treatment facility is not taken into account. Therefore, we treated TLFB and SRD outcomes differently than other outcomes that were less susceptible to such biases (i.e., substance use problem severity as measured by the PEI and externalizing and internalizing symptoms as measured by the YSR).

    • The effectiveness of community-based delivery of an evidence-based treatment for adolescent substance use

      2012, Journal of Substance Abuse Treatment
      Citation Excerpt :

      In our sample, attrition weights are not appropriate because it is highly unlikely that the outcomes among those who responded could be extrapolated to draw inferences about the outcomes among those who did not. Principal stratification, on the other hand, is a model-based approach that works best when one has covariates that are highly predictive of loss to follow-up and that improve both precision and identifiably in the model fit (Griffin, McCaffrey, & Morral, 2008; McCaffrey, Morral, Ridgeway, & Griffin, 2007); in our case, such covariates were not found. We also examined outcome data collected at the 6-month interview to potentially address the differential follow-up concern.

    • Substance use treatment outcomes in a sample of male serious juvenile offenders

      2009, Journal of Substance Abuse Treatment
      Citation Excerpt :

      Second, adolescents in the justice system are differentially housed in controlled environments (such as prison, detention, or residential treatment) for varying periods, and their access to substances (and opportunities for criminal offending) vary as a function of their exposure to these restricted settings (Piquero et al., 2001). Most previous studies have simply ignored the effects of placement in controlled environments, which may overestimate treatment effects (McCaffrey, Morral, Ridgeway, & Griffin, 2007). Although there is no perfect solution to this problem, the current analyses extended prior research by including the proportion of “supervised” time (PST; i.e., residence in a controlled environment) at each time-point as a covariate in the models, as well as testing whether treatment effects varied as a function of time spent in controlled settings.

    View all citing articles on Scopus

    This research was support by NIDA Grants R01 DA015697, R01 DA016722 and R01 DA017507.

    View full text