Making ‘what works’ work: A meta-analytic study of the effect of treatment integrity on outcomes of evidence-based interventions for juveniles with antisocial behavior
Introduction
It takes about seven years to develop and implement an evidence-based intervention in a community setting, around 17.000 dollar to provide it to a single juvenile, and on average, juveniles in youth care are exposed to an intervention for a 12-month period (Aos et al., 2006, Kalidien et al., 2010). Without assuring the proper delivery of interventions, there is a chance that interventions might not produce the desired effects and leave many youths with significant problems underserved or unserved (Fulda et al., 2009, Kataoka et al., 2002, McLeod et al., 2013, Perepletchikova and Kazdin, 2005), which can have serious negative consequences for both these youngsters and their social environment. The community can be confronted with criminal offenses, which impose substantial psychological costs (e.g., victimization) and financial costs on society (e.g., the expenses of imprisonment are on average 700 dollar a person a day), especially when this behavior turns into persistent delinquent behavior (Algemene Rekenkamer, 2012, Cohen et al., 2010). For that reason, it is important to effectively prevent or decrease juvenile antisocial behavior. This meta-analysis is the first to examine the effect of treatment integrity (i.e., delivery of the intervention as intended) on the effectiveness of evidence-based interventions for juveniles with antisocial behavior, while taking the operationalization of treatment integrity into account.
There is a growing number of intervention studies examining the effect of treatment integrity on client outcomes. These studies have found mixed effects. Several studies showed that higher levels of treatment integrity were associated with greater reduction of adolescent's antisocial behavior, whereas other studies did not find such an association. Interestingly, one study examining the effects of individual drug counseling in adult patients, found support for a curvilinear relation between treatment integrity and outcomes, with both low and high levels of integrity showing worse outcomes, and intermediate levels showing the best outcomes (Barber et al., 2006). Barber et al. (2006) argued that very high levels of treatment integrity might reflect a lack of flexibility on the part of the therapist in responding to the client's needs, whereas very low levels of treatment integrity might reflect an inability to translate a therapeutic model or theory into practice as prescribed, which may lead to unsatisfying outcomes. In addition to this explanation, Weisz, Ugueto, Cheron, and Herren (2013) have pointed out that community clinic youths have high rates of comorbidity, which may require a shift of focus during treatment in order to be able to target the most pressing problems, resulting in intermediate levels of treatment integrity. In their review of research on the influence of implementation on program outcomes in prevention and health promotion programs for children and adolescents, Durlak and DuPre (2008) found the maximum level of treatment integrity in outcome studies to be around 80%, and they estimated that positive client outcomes can be expected when levels of treatment integrity are over 60%.
Given that previous research has revealed somewhat inconsistent findings on the association between treatment integrity and client outcomes, a meta-analysis could provide insight into the overall effect of treatment integrity. Previous meta-analyses on the effects of interventions for juveniles with antisocial behavior have suggested that delivering an intervention with high integrity is associated with positive client outcomes. Based on 548 independent study samples, Lipsey (2009) demonstrated that higher quality implementation of interventions targeting juvenile delinquency, such as surveillance, deterrence, discipline, restorative programs, counseling, and skill building programs, was associated with a reduction in recidivism of offending juveniles. Based on 30 independent study samples, Tennyson (2009) concluded that individual, family, group, or multisystemic therapies, as well as correctional programs, parent training, interventions focusing on peer influences, or restitution programs that were delivered with the highest level of treatment integrity produced the greatest reduction in recidivism of juvenile offenders. Thus, based on this research it can be concluded that higher levels of treatment integrity are related to more positive outcomes, which is specifically true for the reduction of recidivism. However, these previous meta-analyses did not take the quality of treatment integrity procedures of the included studies into account, while the validity of treatment integrity measurement likely has consequences for the interpretation of findings.
Measuring treatment implementation is needed to determine whether an intervention failed due to the failure of the intervention or its components, or due to the insufficient or inadequate application of the intervention (Schoenwald et al., 2011). Treatment integrity encompasses two aspects: 1) therapist adherence and 2) therapist competence (Perepletchikova, Treat, & Kazdin, 2007; see for a thorough discussion, Goense et al., 2016, Goense et al., 2014). Therapist adherence can be described as the degree to which the therapist delivers the prescribed components of a specific intervention (i.e., the delivery of an intervention is consistent with the intervention manual). Therapist competence refers to the level of the therapist's technical skills and judgment (timing and appropriateness) in delivering the components of the intervention (Barber et al., 2007a, Barber et al., 2007b, Barber et al., 2006, Perepletchikova et al., 2007). As for therapist competence, McLeod et al. (2013) divided competence into a) technical competence, pertaining to specific components of the intervention, such as the delivery of behavioral cognitive elements in interventions for youth with aggression problems and b) common competence, pertaining to common (non-specific) elements of treatment (e.g., alliance and creating positive expectancies).
Therapists might also be experienced in delivering particular treatment methods acquired in previous therapeutic work that are not part of the specific intervention under study (McLeod et al., 2013). The degree to which the therapists deliver these other treatment methods and consequently deviate from the planned intervention is referred to as treatment differentiation (Kazdin, 1994). Some researchers have suggested that measuring treatment differentiation is not necessary, because the assessment of treatment adherence is considered to preserve intervention purity (e.g., Perepletchikova et al., 2007, Waltz et al., 1993). However, McLeod et al. (2013) argued that without measuring treatment differentiation, examining additional treatment methods that may decrease or increase treatment effects is not possible.
The meta-analyses from Lipsey (2009) and Tennyson (2009) on the effects of interventions for juveniles with antisocial behavior examined if treatment integrity increased treatment efficacy. Lipsey (2009) considered level of involvement of the researcher in treatment implementation as a proxy for the extent to which attention was given to implementing the intervention as intended. Tennyson (2009) examined whether a specific treatment was manualized, if training was provided to practitioners, therapists received supervision, and/or were engaged in adherence checks. Tennyson (2009) grouped these four measures together as a novel means of assessing treatment integrity. Considering the construct of treatment integrity, it is highly questionable whether the operationalization of treatment integrity used by Tennyson (2009) and Lipsey (2009) was valid and comprehensive enough to assess the delivery of the intervention as intended in the studies that were included. These meta-analyses (Lipsey, 2009, Tennyson, 2009) did not actually measure delivery of the intervention in terms of adherence and/or competence, and therefore the assessment of treatment integrity was compromised because of construct underrepresentation.
It can be argued that meta-analyses on this topic have not operationalized treatment integrity in such a way that delivery as intended can be determined in the primary studies that were included. Therefore, a new meta-analytic study that focuses on studies that have incorporated an adequate (sufficiently comprehensive) operationalization of treatment integrity procedures, is needed. With the upcoming focus on treatment integrity, and the growing resources to measure this construct, the demands on the measurement and reporting of treatment integrity in clinical trials are increasing (Fixsen & Ogden, 2014). This enables to conduct a meta-analytic study that takes an adequate (sufficiently comprehensive) operationalization of treatment integrity procedures in the primary studies into account.
The aim of this study is to determine the impact of treatment integrity on client outcomes of evidence-based interventions for juveniles with antisocial behavior. This study differs from previous meta-analyses of Lipsey (2009) and Tennyson (2009) in how treatment integrity procedures are operationalized. In the present meta-analysis, a more valid and comprehensive operationalization of treatment integrity procedures has been used as a selection criterion for the primary studies that were to be included. This operationalization enables an assessment of the degree to which interventions are delivered as intended in the primary studies. In the present meta-analysis we examined whether treatment integrity is a moderator of the reduction of client antisocial behavior after an intervention. In addition to treatment integrity, other study characteristics possibly moderate the reduction of client antisocial behavior after an intervention, including intervention (e.g., intervention type, intervention duration, intervention modality) and methodological characteristics (e.g., study design and follow up time). We subsequently examined these characteristics as moderators. Finally, we examined the unique contribution of several moderating variables in a multivariate (multiple moderator) model.
Section snippets
Inclusion criteria
To be included in the current meta-analysis, studies had to evaluate the effects of an evidence-based intervention targeting juveniles with antisocial behavior. We included studies on the basis of four criteria. First, studies had to examine the effectiveness of evidence-based interventions. In this meta-analysis, evidence-based interventions refer to interventions that at least are theoretically based, well documented, protocolled and structured, described in a manual, and have gained
Descriptive statistics, central tendency and variability, and assessment of missing data
The present study included 14 articles describing k = 17 studies with a total of 91 effect sizes (see Appendix B). In total seven different intervention types were described in the studies: Multisystem Therapy (MST) (k = 7), Wrap-Around (k = 3), Functional Family Therapy (FFT) (k = 2), and Motivational Interviewing (MI) (k = 2). Further, the remaining interventions category of the moderator intervention type consisted of three different interventions: Tip-Sheet (k = 1), Multidimensional Family Therapy
Discussion
The present meta-analytic study integrated previous findings on the link between evidence-based interventions and client outcomes, and examined the moderating effect of treatment integrity on client outcomes. This meta-analysis differs from previous meta-analyses because only outcome studies that, to a certain level, have adequately operationalized treatment integrity procedures were included. By adding this inclusion criterion, we ensured that the level of treatment integrity was validly
Declaration of conflicting interests
The authors declare no potential conflicts of interest with respect to the research, authorship, or publication of this article.
Funding
No funding was provided for this research.
Acknowledgment
We thank Jose van Laar for coding and helping with analyzing the data.
References1 (63)
- et al.
Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review
Clinical Psychology Review
(2015) - et al.
Staff's attitudes and reactions towards aggressive behaviour of clients with intellectual disabilities: A multi-level study
Research in Developmental Disabilities
(2013) - et al.
Effectiveness of motivational incentives for adolescent marijuana users in a school-based intervention
Journal of Substance Abuse Treatment
(2015) Werkwijze in jeugdgevangenis verbeterd; Justitie weet (nog) niet wat het effect op jeugdigen is. Behandeling veel duurder, maar budget lager door minder jeugdcriminaliteit
- et al.
Evidence-based public policy options to reduce future prison construction, criminal justice costs, and crime rates
(2006) - et al.
Sustainability of the effects of multisystemic therapy for juvenile delinquents in the Netherlands: Effects on delinquency and recidivism
Journal of Experimental Criminology
(2014) - et al.
Assessing intervention competence and its relation to therapy outcome
Professional Psychology: Research and Practice
(2007) - et al.
Considerations in treatment integrity: Implications and recommendations for PTSD research
Journal of Traumatic Stress
(2007) - et al.
The role of therapist adherence, therapist competence, and alliance in predicting outcome of individual drug counseling: Results from the national institute drug abuse collaborative cocaine treatment study
Psychotherapy Research
(2006) - et al.
Adherence to wraparound principles and association with outcomes
Journal of Child and Family Studies
(2005)
Modeling dependent effect sizes with three-level meta analyses: A structural equation modeling approach
Psychological Methods
Statistical power analysis for the behavioral sciences
Crime costs across offender trajectories: Studying the costs of crime across offender trajectories
Criminology & Public Policy
Does punishment in secure residential youth care work? An overview of the evidence
Journal of Children's Services
Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation
American Journal of Community Psychology
Connecting the dots: Stages of implementation, wraparound fidelity and youth outcomes
Journal of Child and Family Studies
Bias in meta-analysis detected by a simple graphical test
British Medical Journal
Editorial. Facing the challenges of implementation
Zeitschrift für Psychologie
Implementation research: A synthesis of the literature
Unmet mental health care needs for children with special health care needs stratified by socioeconomic status
Child and Adolescent Mental Health
Use of effective and efficient quality control methods to implement psychosocial interventions
Clinical Psychology: Science and Practice
Support systems for treatment integrity
Research on Social Work Practice
Measuring treatment integrity: Use of and experiences with measurements in routine youth care practice settings
Implementation of treatment integrity procedures: An analysis of outcome studies of youth interventions targeting externalizing behavioral problems
Zeitschrift für Psychologie
Evaluation of functional family therapy in an Irish context
Journal of Family Therapy
Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination
Journal of Consulting and Clinical Psychology
Multisystemic treatment of substance-abusing and dependent delinquents: Outcomes, treatment fidelity, and transportability
Mental Health Services Research
The relation between short-term emotion dynamics and psychological well-being: A meta-analysis
Psychological Bulletin
Multilevel analysis: Techniques and applications
Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning
Journal of Consulting and Clinical Psychology
Criminaliteit en Rechtshandhaving 2010. Ontwikkelingen en samenhangen
Cited by (40)
The development of adolescents in a non-residential alternative educational facility, including the prevention of secure residential placement
2023, Children and Youth Services ReviewEffects of the Big Brothers Big Sisters of America Community-Based Mentoring Program on social-emotional, behavioral, and academic outcomes of participating youth: A randomized controlled trial
2023, Children and Youth Services ReviewCitation Excerpt :Research suggests that these variations may have affected program impacts. A recent meta-analysis (Goense et al., 2016), for example, of 17 studies of evidence-based interventions for youth with antisocial behavior found medium to large effects when treatment integrity was high and no significant effects when integrity was low. Uneven implementation could be explained, in part, by the fact that aspects of these practices were new expectations associated with the initiative in which the agencies were participating.
The outcome of non-residential youth care compared to residential youth care: A multilevel meta-analysis
2020, Children and Youth Services ReviewElucidating care for families with multiple problems in routine practice: Self-registered practice and program elements of practitioners
2020, Children and Youth Services ReviewCitation Excerpt :This may be the case for several reasons. First, the shifting needs and problems of FMP during the intervention may complicate application of the intervention as documented in the manual (Goense et al., 2016). Second, the time and financial pressure faced by care providers and practitioners themselves may influence the content of interventions in routine practice (Weisz, Ng, & Bearman, 2014).
Evaluating art therapy processes with children diagnosed with Autism Spectrum Disorders: Development and testing of two observation instruments for evaluating children's and therapists’ behaviour
2019, Arts in PsychotherapyCitation Excerpt :An instrument that monitors what art therapists actually do during treatment will help to evaluate the professional skills of the art therapist. This also may shed a light on therapeutic integrity by evaluating if the art therapist is delivering those treatment components that are intended to be provided (Goense, Assink, Stams, Boendermaker, & Hoeve, 2016). Previous steps for the development of rating instruments to monitor AT were described in a Delphi study Schweizer, Knorth, Van Yperen, & Spreen, 2019.
- 1
References marked with an asterisk indicate studies included in the meta-analysis.