Consistency of outcomes of home-based family treatment in The Netherlands as an indicator of effectiveness
Introduction
Youth care interventions are increasingly required to be evidence based. Ideally, studies that include a control group and random assignment would support the efficacy of an intervention. The randomized controlled trial has become the gold standard for establishing evidence, as changes in the functioning of youth and their families could be ascribed to the intervention that was studied (APA Presidential Task Force on Evidence-Based Practice, 2005, Bower, 2003, Shadish et al., 2002). If these experimental designs are not feasible, evidence from quasi-experimental designs utilizing some form of control (e.g., untreated group, placebo group, and wait list group) would be accepted as an alternative. The growing number of internet databases that summarize effective treatments for youth consider the evidence only from these two kinds of studies as sound empirical support (see for instance Blueprints for Healthy Youth Development, www.blueprintsprograms.com; and The California Evidence-based Clearinghouse, www.cebc4cw.org). However, in practice, most youth care interventions do not have such evidence gathered on their behalf (Kazdin, 2003, Thoburn, 2010). It is estimated that not more than 10% of youth prevention and intervention programs meet this requirement (Kumpfer and Alvarado, 2003, Veerman and van Yperen, 2007). Furthermore, in the Dutch database of The Netherlands Youth Institute (NJI), which currently (October 2015) contains 220 interventions for youth and families with psychosocial problems, the same percentage (22 interventions) meets this requirement (www.jeugdinterventies.nl). Obviously, there are significant obstacles to the implementation of evidence-based interventions (Axford and Morpeth, 2013, Weisz et al., 2013). In the meantime, children and their families who need care have the right to receive that care, and it is practically impossible and ethically unacceptable to withhold interventions and wait for the necessary evidence supporting these interventions.
Given this situation, it would be very informative to learn more about the effects of interventions from experiences in practice and to turn around the flow of evidence. Instead of taking the top-down route, in which evidence-based interventions are studied in a controlled situation and then implemented in practice, we might take a bottom-up route. This means we take the current state of youth care as a point of departure and encourage practitioners to conduct their treatment as usual. At the same time, we insist that they formulate an explicit theory that shows how and why an intervention is supposed to work and offer empirical data to show that an intervention does what it is supposed to do — bring beneficial changes to the lives of their clients. Veerman and Van Yperen (2007) argued that explicating the theoretical foundation of an intervention and delivering empirical signs of beneficial change are two important steps on an effect ladder that ranges from knowing nothing about the effects of an intervention to establishing cause–effect relationships. By insisting too much on (quasi-)experiments as the gold standard for evidence and for the publication of studies, we have no idea about the effectiveness or ineffectiveness of most youth care interventions (Kazdin, 1997, Thoburn, 2010).
To establish this kind of knowledge, it is very important to conduct practice-based studies with at least two measurements, one at the start and one at the end of a theoretically well-founded intervention, with one or more standardized instruments aimed at measuring the expected outcomes. Methodologically, this is a pre–post design without a control group (Shadish et al., 2002). A major critique of such a design is that it does not tell us much about the cause–effect relationship. However, not looking at the results of such studies because they do not meet the gold standard, and therefore not publishing them, leaves practitioners and administrators with empty hands in front of their clients, policy makers, and insurance companies. Although the gold standard should not be abandoned, the reality is that in the practice of care, RCTs are difficult to conduct as intended, which undermines the strengths of this design: internal validity and statistical power to detect significant changes (James, Asscher, Dekovic, Van der Laan, & Stams, 2013). Jacobs (2003) maintained that experimental designs are not feasible to use with the vast majority of child and family programs, and therefore we should be satisfied with less rigorous research designs that are ‘good enough’. Moreover, as we discuss later in this introduction, it is possible to strengthen the evidence from this design (Axford et al., 2005, Reynolds, 2004).
The present study focuses on showing empirical indications of the effectiveness of an intervention that possesses a solid theoretical base. The treatment at stake is Intensive Family Treatment (IFT) (known in Dutch as IOG) carried out in The Netherlands. IFT is a home-based family preservation program developed in the 1990s. The general aim is to prevent out-of-home placement of children at risk of such placement. The next section presents the content of this intervention in more detail. The Dutch database of effective interventions in youth care classifies IFT as ‘theoretically well founded’ (www.jeugdinterventies.nl). IFT is also one of 17 Dutch family preservation services in a meta-analysis that showed a medium overall pre–post effect size of 0.52 for externalizing behaviors and 0.55 for parenting stress (Veerman, Janssens, & Delicat, 2005). IFT had an effect size of 0.60 and 0.58, respectively. A later Dutch study on IFT confirmed this result (Veerman, De Meyer, & Roosma, 2007).
In the present study, we will report outcome data of IFT from subsequent years collected in different locations and analyzed in different ways. This fulfills the ‘consistency principle’ Reynolds (2004) introduced. According to Reynolds, consistency of association between treatment exposure and outcome ‘indicates whether the estimated program effect is similar across sample populations, at different times and places, under different types of analyses and model specifications, and for similar intervention theories. The greater the consistency of findings favoring positive effects, the more likely the observed effects are real’ (Reynolds, 2004, p. 20). To conduct different analyses, we used group statistics as well as individual statistics to look for changes in children's behavior problems. Group analyses are used in almost all change studies. Such analyses usually compare the group means of an outcome measure before and after treatment and use a statistical test to decide whether improvement has occurred. An effect size is calculated to determine the magnitude of the observed changes (mostly Cohen's d). However, as Hiller, Schindler, and Lambert (2012) maintained, group analysis has a limitation in that it does not provide information about changes that have taken place in individual clients, and hence it provides no information about the relative proportions of clients who respond to the treatment. Group statistics also provide no information on who improved after the treatment and who did not. Hence, there is growing consensus among treatment researchers that individual treatment outcomes should be evaluated and reported in addition to group analysis (Barkham et al., 2008, Nelson et al., 2013). We used the reliable change index (RCI) to analyze individual treatment outcomes (Jacobson & Truax, 1991). Briefly, the RCI indicates change that is very likely to be greater than chance. Moreover, by including a criterion for normal and problematic functioning, several patterns of change can be identified. This is further outlined in the Analyses section. Based on the earlier Dutch studies, we expected positive outcomes from the group analysis as well as from the individual analyses. Furthermore, we expected these outcomes to be stable across years and locations, thereby fulfilling Reynolds' principle of consistency.
Section snippets
Participants and treatment
Participants were Dutch youths aged between four and 18 years, who had been referred to Intensive Family Treatment (IFT) at three youth care agencies in three provinces in the north and east of The Netherlands and completed their treatment between 1999 and 2008. IFT is a home-based family-preservation service for families with several severe psychosocial problems (often characterized as multi-problem families), such as parenting problems, marital problems, behavior problems of the child(ren),
Group outcomes
A MANOVA was conducted with the Total problem score as the within-subject factor, and age and gender (4–11 and 12–18 years of age) as between-subject factors to decide whether further analyses can be conducted for the total group or whether the data have to be analyzed for age and gender subgroups separately. The MANOVA showed a main effect for the within-subject factor Total problem score (F(1, 891) = 527.76, p = 0.000). The interactions of total score by age and total score by gender were not
Discussion
In the present study, we used group and individual statistics of the outcome of IFT, a home-based service in The Netherlands, to analyze practice-based evidence of effectiveness. Group statistics of IFT showed significant changes in children's emotional and behavioral problems over the years and for each year separately. The pre–post effect sizes were at least in the medium range, with an overall mean of 0.66 (range of 0.51–0.80), which is common for treatments for youth (Garland et al., 2014,
Acknowledgments
We would like to thank the Dutch “Hulp aan Huis” (Help at Home) agencies in Groningen, Drenthe and Overijssel for funding the research in this study, and in particular Dries Roosma for his role in the implementation of the study in these agencies. We also would like to thank the many family workers who participated in the data collection.
References (35)
- et al.
Evidence-based programs in children's services. A critical appraisal
Children and Youth Services Review
(2013) - et al.
Under one roof: A review and selective meta-analysis on the outcomes of residential child and youth care
Children and Youth Services Review
(2008) Research on early childhood interventions in the confirmatory mode
Children and Youth Services Review
(2004)- et al.
Degrees of freedom and degrees of certainty. A developmental model for the establishment of evidence-based youth care
Evaluation and Program Planning
(2007) Manual for the child behavior checklist/4–18 and 1991 profile
(1991)- et al.
Manual for the ASEBA school-age forms & profiles
(2001) Evidence-based practice in psychology
American Psychologist
(2005)- et al.
Evaluating children's services. Recent conceptual and methodological developments
British Journal of Social Work
(2005) - et al.
Effects of psychological therapies in randomized trials and practice-based studies
British Journal of Clinical Psychology
(2008) Efficacy in evidence-based practice
Clinical Psychology and Psychotherapy
(2003)
A power primer
Psychological Bulletin
Basic trust: An attachment-oriented intervention based on mind-mindedness in adoptive families
Research on Social Work Practice
Searching for elements of evidence-based practices in children's usual care and examining their impact
Journal of Clinical Child & Adolescent Psychology
Defining response and remission in psychotherapy research: A comparison of the RCI and the method of percent improvement
Psychotherapy Research
Child and family program evaluation. Learning to enjoy complexity
Applied Developmental Science
Clinical significance. A statistical approach to defining meaningful change in psychotherapy research
Journal of Consulting and Clinical Psychology
Endeavors in an experimental study on the effectiveness of an aftercare program in The Netherlands
Criminal Justice Policy review
Cited by (8)
Improving interventions for families with multiple problems: Identifying elements that predict improvement
2023, Children and Youth Services ReviewParental empowerment as a buffer between parental stress and child behavioral problems after family treatment
2021, Children and Youth Services ReviewCitation Excerpt :Despite these favorable treatment outcomes, on average, substantial child behavioral problems still existed at the end of IFT that, moreover, persisted in the subsequent period. This is consistent with the findings of other studies (Asscher et al., 2013; Van Assen et al., 2020; Veerman & De Meyer, 2015). The behavioral problems of children in vulnerable families could be a stressor for parents after treatment.
Predictors of parent-professional alliance in home-based parenting support
2018, Children and Youth Services ReviewCitation Excerpt :Involved children (70.7% male) were between 3 and 17 years old (M = 11.35 years, SD = 3.98). Home-based parenting support services were part of routine practices of nine Dutch youth care organizations, meaning that services were eclectic, non-protocolized, and grounded in various approaches (e.g., Intensive Family Treatment; Veerman & De Meyer, 2015). Services were provided by 42 professionals (90.5% female, M age = 45.82 years, SD = 9.74; range 29–60 years), serving one to four families each (M = 1.43, SD = 0.70).
The Child Protection System in the Netherlands: Characteristics, Trends, and Evidence
2023, The Oxford Handbook of Child Protection SystemsParental empowerment and child behavioural problems during youth care involvement
2019, Child and Family Social Work