Elsevier

Aggression and Violent Behavior

Volume 31, November–December 2016, Pages 106-115
Aggression and Violent Behavior

Making ‘what works’ work: A meta-analytic study of the effect of treatment integrity on outcomes of evidence-based interventions for juveniles with antisocial behavior

https://doi.org/10.1016/j.avb.2016.08.003Get rights and content

Highlights

  • The moderating effect of treatment integrity was examined meta-analytically.

  • Studies were included if adequate treatment integrity procedures were applied.

  • Medium-to-large significant effects were found for high treatment integrity

  • Small and non-significant effects were found for low treatment integrity.

  • Delivering interventions with high treatment integrity should be stimulated.

Abstract

This study meta-analytically examined the effect of treatment integrity on client outcomes of evidence-based interventions for juveniles with antisocial behavior. A total of 17 studies, from which 91 effect sizes could be retrieved, were included in the present 3-level meta-analysis. All included studies, to a certain level, adequately implemented procedures to establish, assess, evaluate and report the level of treatment integrity. A moderator analysis revealed that a medium-to-large effect of evidence-based interventions was found when the level of treatment integrity was high (d = 0.633, p < 0.001), whereas no significant effect was found when integrity was low (d = 0.143, ns). Treatment integrity was significantly associated with effect size even when adjusted for other significant moderators, indicating the specific contribution of high levels of treatment integrity to positive client outcomes. This implies that delivering interventions with high treatment integrity to youth with antisocial behavior is vital.

Introduction

It takes about seven years to develop and implement an evidence-based intervention in a community setting, around 17.000 dollar to provide it to a single juvenile, and on average, juveniles in youth care are exposed to an intervention for a 12-month period (Aos et al., 2006, Kalidien et al., 2010). Without assuring the proper delivery of interventions, there is a chance that interventions might not produce the desired effects and leave many youths with significant problems underserved or unserved (Fulda et al., 2009, Kataoka et al., 2002, McLeod et al., 2013, Perepletchikova and Kazdin, 2005), which can have serious negative consequences for both these youngsters and their social environment. The community can be confronted with criminal offenses, which impose substantial psychological costs (e.g., victimization) and financial costs on society (e.g., the expenses of imprisonment are on average 700 dollar a person a day), especially when this behavior turns into persistent delinquent behavior (Algemene Rekenkamer, 2012, Cohen et al., 2010). For that reason, it is important to effectively prevent or decrease juvenile antisocial behavior. This meta-analysis is the first to examine the effect of treatment integrity (i.e., delivery of the intervention as intended) on the effectiveness of evidence-based interventions for juveniles with antisocial behavior, while taking the operationalization of treatment integrity into account.

There is a growing number of intervention studies examining the effect of treatment integrity on client outcomes. These studies have found mixed effects. Several studies showed that higher levels of treatment integrity were associated with greater reduction of adolescent's antisocial behavior, whereas other studies did not find such an association. Interestingly, one study examining the effects of individual drug counseling in adult patients, found support for a curvilinear relation between treatment integrity and outcomes, with both low and high levels of integrity showing worse outcomes, and intermediate levels showing the best outcomes (Barber et al., 2006). Barber et al. (2006) argued that very high levels of treatment integrity might reflect a lack of flexibility on the part of the therapist in responding to the client's needs, whereas very low levels of treatment integrity might reflect an inability to translate a therapeutic model or theory into practice as prescribed, which may lead to unsatisfying outcomes. In addition to this explanation, Weisz, Ugueto, Cheron, and Herren (2013) have pointed out that community clinic youths have high rates of comorbidity, which may require a shift of focus during treatment in order to be able to target the most pressing problems, resulting in intermediate levels of treatment integrity. In their review of research on the influence of implementation on program outcomes in prevention and health promotion programs for children and adolescents, Durlak and DuPre (2008) found the maximum level of treatment integrity in outcome studies to be around 80%, and they estimated that positive client outcomes can be expected when levels of treatment integrity are over 60%.

Given that previous research has revealed somewhat inconsistent findings on the association between treatment integrity and client outcomes, a meta-analysis could provide insight into the overall effect of treatment integrity. Previous meta-analyses on the effects of interventions for juveniles with antisocial behavior have suggested that delivering an intervention with high integrity is associated with positive client outcomes. Based on 548 independent study samples, Lipsey (2009) demonstrated that higher quality implementation of interventions targeting juvenile delinquency, such as surveillance, deterrence, discipline, restorative programs, counseling, and skill building programs, was associated with a reduction in recidivism of offending juveniles. Based on 30 independent study samples, Tennyson (2009) concluded that individual, family, group, or multisystemic therapies, as well as correctional programs, parent training, interventions focusing on peer influences, or restitution programs that were delivered with the highest level of treatment integrity produced the greatest reduction in recidivism of juvenile offenders. Thus, based on this research it can be concluded that higher levels of treatment integrity are related to more positive outcomes, which is specifically true for the reduction of recidivism. However, these previous meta-analyses did not take the quality of treatment integrity procedures of the included studies into account, while the validity of treatment integrity measurement likely has consequences for the interpretation of findings.

Measuring treatment implementation is needed to determine whether an intervention failed due to the failure of the intervention or its components, or due to the insufficient or inadequate application of the intervention (Schoenwald et al., 2011). Treatment integrity encompasses two aspects: 1) therapist adherence and 2) therapist competence (Perepletchikova, Treat, & Kazdin, 2007; see for a thorough discussion, Goense et al., 2016, Goense et al., 2014). Therapist adherence can be described as the degree to which the therapist delivers the prescribed components of a specific intervention (i.e., the delivery of an intervention is consistent with the intervention manual). Therapist competence refers to the level of the therapist's technical skills and judgment (timing and appropriateness) in delivering the components of the intervention (Barber et al., 2007a, Barber et al., 2007b, Barber et al., 2006, Perepletchikova et al., 2007). As for therapist competence, McLeod et al. (2013) divided competence into a) technical competence, pertaining to specific components of the intervention, such as the delivery of behavioral cognitive elements in interventions for youth with aggression problems and b) common competence, pertaining to common (non-specific) elements of treatment (e.g., alliance and creating positive expectancies).

Therapists might also be experienced in delivering particular treatment methods acquired in previous therapeutic work that are not part of the specific intervention under study (McLeod et al., 2013). The degree to which the therapists deliver these other treatment methods and consequently deviate from the planned intervention is referred to as treatment differentiation (Kazdin, 1994). Some researchers have suggested that measuring treatment differentiation is not necessary, because the assessment of treatment adherence is considered to preserve intervention purity (e.g., Perepletchikova et al., 2007, Waltz et al., 1993). However, McLeod et al. (2013) argued that without measuring treatment differentiation, examining additional treatment methods that may decrease or increase treatment effects is not possible.

The meta-analyses from Lipsey (2009) and Tennyson (2009) on the effects of interventions for juveniles with antisocial behavior examined if treatment integrity increased treatment efficacy. Lipsey (2009) considered level of involvement of the researcher in treatment implementation as a proxy for the extent to which attention was given to implementing the intervention as intended. Tennyson (2009) examined whether a specific treatment was manualized, if training was provided to practitioners, therapists received supervision, and/or were engaged in adherence checks. Tennyson (2009) grouped these four measures together as a novel means of assessing treatment integrity. Considering the construct of treatment integrity, it is highly questionable whether the operationalization of treatment integrity used by Tennyson (2009) and Lipsey (2009) was valid and comprehensive enough to assess the delivery of the intervention as intended in the studies that were included. These meta-analyses (Lipsey, 2009, Tennyson, 2009) did not actually measure delivery of the intervention in terms of adherence and/or competence, and therefore the assessment of treatment integrity was compromised because of construct underrepresentation.

It can be argued that meta-analyses on this topic have not operationalized treatment integrity in such a way that delivery as intended can be determined in the primary studies that were included. Therefore, a new meta-analytic study that focuses on studies that have incorporated an adequate (sufficiently comprehensive) operationalization of treatment integrity procedures, is needed. With the upcoming focus on treatment integrity, and the growing resources to measure this construct, the demands on the measurement and reporting of treatment integrity in clinical trials are increasing (Fixsen & Ogden, 2014). This enables to conduct a meta-analytic study that takes an adequate (sufficiently comprehensive) operationalization of treatment integrity procedures in the primary studies into account.

The aim of this study is to determine the impact of treatment integrity on client outcomes of evidence-based interventions for juveniles with antisocial behavior. This study differs from previous meta-analyses of Lipsey (2009) and Tennyson (2009) in how treatment integrity procedures are operationalized. In the present meta-analysis, a more valid and comprehensive operationalization of treatment integrity procedures has been used as a selection criterion for the primary studies that were to be included. This operationalization enables an assessment of the degree to which interventions are delivered as intended in the primary studies. In the present meta-analysis we examined whether treatment integrity is a moderator of the reduction of client antisocial behavior after an intervention. In addition to treatment integrity, other study characteristics possibly moderate the reduction of client antisocial behavior after an intervention, including intervention (e.g., intervention type, intervention duration, intervention modality) and methodological characteristics (e.g., study design and follow up time). We subsequently examined these characteristics as moderators. Finally, we examined the unique contribution of several moderating variables in a multivariate (multiple moderator) model.

Section snippets

Inclusion criteria

To be included in the current meta-analysis, studies had to evaluate the effects of an evidence-based intervention targeting juveniles with antisocial behavior. We included studies on the basis of four criteria. First, studies had to examine the effectiveness of evidence-based interventions. In this meta-analysis, evidence-based interventions refer to interventions that at least are theoretically based, well documented, protocolled and structured, described in a manual, and have gained

Descriptive statistics, central tendency and variability, and assessment of missing data

The present study included 14 articles describing k = 17 studies with a total of 91 effect sizes (see Appendix B). In total seven different intervention types were described in the studies: Multisystem Therapy (MST) (k = 7), Wrap-Around (k = 3), Functional Family Therapy (FFT) (k = 2), and Motivational Interviewing (MI) (k = 2). Further, the remaining interventions category of the moderator intervention type consisted of three different interventions: Tip-Sheet (k = 1), Multidimensional Family Therapy

Discussion

The present meta-analytic study integrated previous findings on the link between evidence-based interventions and client outcomes, and examined the moderating effect of treatment integrity on client outcomes. This meta-analysis differs from previous meta-analyses because only outcome studies that, to a certain level, have adequately operationalized treatment integrity procedures were included. By adding this inclusion criterion, we ensured that the level of treatment integrity was validly

Declaration of conflicting interests

The authors declare no potential conflicts of interest with respect to the research, authorship, or publication of this article.

Funding

No funding was provided for this research.

Acknowledgment

We thank Jose van Laar for coding and helping with analyzing the data.

References1 (63)

  • M. Assink et al.

    Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review

    Clinical Psychology Review

    (2015)
  • M.H. Knotter et al.

    Staff's attitudes and reactions towards aggressive behaviour of clients with intellectual disabilities: A multi-level study

    Research in Developmental Disabilities

    (2013)
  • D.G. *Stewart et al.

    Effectiveness of motivational incentives for adolescent marijuana users in a school-based intervention

    Journal of Substance Abuse Treatment

    (2015)
  • Algemene Rekenkamer

    Werkwijze in jeugdgevangenis verbeterd; Justitie weet (nog) niet wat het effect op jeugdigen is. Behandeling veel duurder, maar budget lager door minder jeugdcriminaliteit

  • S. Aos et al.

    Evidence-based public policy options to reduce future prison construction, criminal justice costs, and crime rates

    (2006)
  • J.J. *Asscher et al.

    Sustainability of the effects of multisystemic therapy for juvenile delinquents in the Netherlands: Effects on delinquency and recidivism

    Journal of Experimental Criminology

    (2014)
  • J.P. Barber et al.

    Assessing intervention competence and its relation to therapy outcome

    Professional Psychology: Research and Practice

    (2007)
  • J. Barber et al.

    Considerations in treatment integrity: Implications and recommendations for PTSD research

    Journal of Traumatic Stress

    (2007)
  • J.P. Barber et al.

    The role of therapist adherence, therapist competence, and alliance in predicting outcome of individual drug counseling: Results from the national institute drug abuse collaborative cocaine treatment study

    Psychotherapy Research

    (2006)
  • E.J. *Bruns et al.

    Adherence to wraparound principles and association with outcomes

    Journal of Child and Family Studies

    (2005)
  • M.W.L. Cheung

    Modeling dependent effect sizes with three-level meta analyses: A structural equation modeling approach

    Psychological Methods

    (2014)
  • J. Cohen

    Statistical power analysis for the behavioral sciences

    (1988)
  • M.A. Cohen et al.

    Crime costs across offender trajectories: Studying the costs of crime across offender trajectories

    Criminology & Public Policy

    (2010)
  • S. De Valk et al.

    Does punishment in secure residential youth care work? An overview of the evidence

    Journal of Children's Services

    (2015)
  • J. Durlak et al.

    Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation

    American Journal of Community Psychology

    (2008)
  • V. *Effland et al.

    Connecting the dots: Stages of implementation, wraparound fidelity and youth outcomes

    Journal of Child and Family Studies

    (2011)
  • M. Egger et al.

    Bias in meta-analysis detected by a simple graphical test

    British Medical Journal

    (1997)
  • D.L. Fixsen et al.

    Editorial. Facing the challenges of implementation

    Zeitschrift für Psychologie

    (2014)
  • D.L. Fixsen et al.

    Implementation research: A synthesis of the literature

    (2005)
  • K.G. Fulda et al.

    Unmet mental health care needs for children with special health care needs stratified by socioeconomic status

    Child and Adolescent Mental Health

    (2009)
  • A.F. Garland et al.

    Use of effective and efficient quality control methods to implement psychosocial interventions

    Clinical Psychology: Science and Practice

    (2013)
  • P.B. Goense et al.

    Support systems for treatment integrity

    Research on Social Work Practice

    (2015)
  • P.B. Goense et al.

    Measuring treatment integrity: Use of and experiences with measurements in routine youth care practice settings

    (2016)
  • P. Goense et al.

    Implementation of treatment integrity procedures: An analysis of outcome studies of youth interventions targeting externalizing behavioral problems

    Zeitschrift für Psychologie

    (2014)
  • C. *Graham et al.

    Evaluation of functional family therapy in an Irish context

    Journal of Family Therapy

    (2014)
  • S.W. *Henggeler et al.

    Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination

    Journal of Consulting and Clinical Psychology

    (1997)
  • S.W. *Henggeler et al.

    Multisystemic treatment of substance-abusing and dependent delinquents: Outcomes, treatment fidelity, and transportability

    Mental Health Services Research

    (1999)
  • M. Houben et al.

    The relation between short-term emotion dynamics and psychological well-being: A meta-analysis

    Psychological Bulletin

    (2015)
  • J. Hox

    Multilevel analysis: Techniques and applications

    (2002)
  • S.J. *Huey et al.

    Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning

    Journal of Consulting and Clinical Psychology

    (2000)
  • S.N. Kalidien et al.

    Criminaliteit en Rechtshandhaving 2010. Ontwikkelingen en samenhangen

    (2010)
  • Cited by (40)

    • Effects of the Big Brothers Big Sisters of America Community-Based Mentoring Program on social-emotional, behavioral, and academic outcomes of participating youth: A randomized controlled trial

      2023, Children and Youth Services Review
      Citation Excerpt :

      Research suggests that these variations may have affected program impacts. A recent meta-analysis (Goense et al., 2016), for example, of 17 studies of evidence-based interventions for youth with antisocial behavior found medium to large effects when treatment integrity was high and no significant effects when integrity was low. Uneven implementation could be explained, in part, by the fact that aspects of these practices were new expectations associated with the initiative in which the agencies were participating.

    • Elucidating care for families with multiple problems in routine practice: Self-registered practice and program elements of practitioners

      2020, Children and Youth Services Review
      Citation Excerpt :

      This may be the case for several reasons. First, the shifting needs and problems of FMP during the intervention may complicate application of the intervention as documented in the manual (Goense et al., 2016). Second, the time and financial pressure faced by care providers and practitioners themselves may influence the content of interventions in routine practice (Weisz, Ng, & Bearman, 2014).

    • Evaluating art therapy processes with children diagnosed with Autism Spectrum Disorders: Development and testing of two observation instruments for evaluating children's and therapists’ behaviour

      2019, Arts in Psychotherapy
      Citation Excerpt :

      An instrument that monitors what art therapists actually do during treatment will help to evaluate the professional skills of the art therapist. This also may shed a light on therapeutic integrity by evaluating if the art therapist is delivering those treatment components that are intended to be provided (Goense, Assink, Stams, Boendermaker, & Hoeve, 2016). Previous steps for the development of rating instruments to monitor AT were described in a Delphi study Schweizer, Knorth, Van Yperen, & Spreen, 2019.

    View all citing articles on Scopus
    1

    References marked with an asterisk indicate studies included in the meta-analysis.

    View full text