Elsevier

Child Abuse & Neglect

Volume 57, July 2016, Pages 30-40
Child Abuse & Neglect

Research article
A study in contrasts: Supports and barriers to successful implementation of two evidence-based parenting interventions in child welfare

https://doi.org/10.1016/j.chiabu.2016.06.002Get rights and content

Abstract

Although evidence-based interventions (EBIs) are spreading to child welfare, research on real-world dynamics of implementation within this setting is scarce. Using a six-factor implementation framework to examine implementation of two evidence-based parenting interventions, we sought to build greater understanding of key facilitators and barriers by comparing successful versus failed EBI implementation in a child welfare setting. Semi-structured interviews were conducted with a purposive sample of 15 frontline practitioners and state-level managers. Interviews were transcribed verbatim and data analysis used a modified analytic approach. Our results showed the successful EBI was viewed more positively on all six factors; however, implementation was multidimensional, multilevel, and mixed with accomplishments and challenges. An accumulation of strengths across implementation factors proved beneficial. Implementation frameworks may be advantageous in organizing and explaining the numerous factors that may influence successful versus failed implementation. While encountering obstacles is largely inevitable, understanding which factors have shaped the success or failure of EBI implementations in child welfare settings may optimize future implementations in this context.

Introduction

Evidence-based interventions (EBIs) are increasingly common in many fields and changing the way public-sector service settings function. EBIs have been identified as the gold standard by many professionals, and researchers continue to provide evidence on the positive impact that results from EBI utilization (Landsverk, Brown, Reutz, Palinkas, & Horwitz, 2011). To ensure desired outcomes, researchers study factors leading to successful EBI implementation (Aarons et al., 2014). Although scholars have delineated multiple frameworks that identify the general areas of focus needed for successful EBI implementation (e.g., Aarons et al., 2011, Century et al., 2012, Fixsen et al., 2009), more research is needed to understand the fit between these factors identified in the literature and real-world EBI implementation.

While EBIs are common in fields such as medicine and mental health, their implementation in child welfare settings is more recent. Thus, the body of knowledge on the challenges and supports of EBI delivery in child welfare is in the early stages of development. Prior literature on EBIs in child welfare indicate that this client population benefits from provision of evidence-based services (e.g., Barth et al., 2005, Horwitz et al., 2010), and in recent years, the child welfare field has increasingly been expected to use EBIs (Children's Bureau, 2012). Despite the advantages of EBIs, scholars acknowledge challenges arise throughout the implementation process, and underscore the need for additional research related to implementation science (Aarons et al., 2011, Fixsen et al., 2009). Many complicated factors shape EBI implementation in child welfare settings, such as client characteristics, involvement of court and school systems, worker turnover, and budget constraints. Hence, EBI implementation in these settings must consider many factors (Aarons & Palinkas, 2007).

Understanding the fit between conceptualized implementation factors and localized real-world implementation of EBIs is critical. Moreover, understanding the real-world dynamics of supports and challenges within key implementation factors is understudied. To address these gaps, the objectives of this study were to: (1) examine the implementation of two evidence-based parenting interventions in a child welfare setting, one which experienced success and the other which did not; (2) build knowledge of the specific factors that shape EBI implementation in a child welfare setting; and (3) evaluate the fit between current implementation frameworks and EBI implementation experiences in a child welfare setting.

Based on a thorough review of existing EBI implementation frameworks, Akin et al. (2014) identified six key factors that have been associated with successful implementation in past research: process, provider, innovation, client, organizational, and structural. Below is a description of each factor.

Process factors include staff selection, training, coaching, and performance assessment (Akin et al., 2014). Also known as “competency drivers,” process factors impact fidelity and competency in adherence to the EBI (Kaye, DePanfilis, Bright, & Fisher, 2012). Researchers underscore the significance of process factors in child welfare due to the field's emphasis on training and supervision/coaching (Akin, Brook, Byers, & Lloyd, 2015). Further, Kaye et al. (2012) highlighted the importance of a well-trained workforce in a field where turnover is commonplace and also where workers are required to frequently collaborate with other systems (e.g., courts, mental health agencies, and schools). Many scholars have also demonstrated how training has been linked with a plethora of positive outcomes for professionals in child welfare settings, such as increased knowledge and skills, improved recruitment and retention of workers, and decreased burnout (Kaye et al., 2012, Turcotte et al., 2009).

Providers are an essential aspect of EBI implementation, particularly in a field such as child welfare where individual human characteristics can significantly alter the outcome of the EBI being adopted (Aarons & Palinkas, 2007). For example, providers’ perspectives on the adoption and utilization of the EBI can influence how successful an EBI is (Aarons, 2004, Akin et al., 2014). Considering the influence that provider attitudes can have on the implementation process, Aarons (2004) developed the Evidence-Based Practice Attitude Scale (EBPAS) with the aim of examining behavioral health service providers’ attitudes toward adoption of EBPs and its association with various factors (e.g., education level, professional status, and more). Some main findings from study of the EBPAS were that those with higher levels of educational attainment were more open to adopting EBIs, and interns endorsed more positive attitudes toward adopting a new EBI compared to professional providers (Aarons, 2004).

Innovation factors concern the complexity, adaptability, and cost of EBI implementation (Akin et al., 2015). These can include the agency's perspective on the adaptability of an EBI to the population they serve, logistical issues (e.g., availability of transportation, language translators, and more), as well as the congruency of the EBI with the culture, values, and methods of the organization implementing it (Damschroder et al., 2009). Likewise, Simpson (2009) emphasized the importance of not only the adaptability, but also the relevance of a particular EBI to the population that an agency serves.

Client factors have been studied across various disciplines and indicated as undoubtedly influential on implementation, yet research on client factors in child welfare settings is limited (Akin et al., 2015). These factors include the complexity of clients’ presenting concerns, their perceptions (e.g., “buy-in”) of the EBI, resistance toward a program, as well as environmental barriers clients may encounter (Akin et al., 2014). Clients’ environments and the malleability of an EBI to clients’ specific situations can greatly impact the implementation. Further, many child welfare clients are involuntary; thus, their attitudes about and willingness to be involved in the EBI can significantly impact implementation (Aarons et al., 2009c, Akin et al., 2015, Kemp et al., 2009).

Organizational factors range anywhere from the culture, climate, attitudes, and values to the policies and procedures of an agency. Specifically, the organization's leaders’ attitudes about the EBI can influence providers’ attitudes, which can then influence clients’ attitudes (Aarons and Palinkas, 2007, Akin et al., 2014, Akin et al., 2015). In addition, researchers have identified strong leaders who understand the advantages of EBIs and advocate for them as a requirement for successful implementation and sustainability (Willging, Green, Gunderson, Chaffin, & Aarons, 2015). Furthermore, if the EBI does not align with the organization's values and methods, it may be difficult for staff to “buy-in” to the EBI (Aarons & Palinkas, 2007).

Structural factors are those that occur at the system level including the broader environment in which organizations operate. These can include workforce issues, court systems, and interagency collaboration – all of which have the potential to either greatly bolster or hinder EBI implementation (Aarons and Palinkas, 2007, Willging et al., 2015). Many scholars emphasize the importance of structural factors in child welfare settings where there is a heavy influence from numerous stakeholders, revolving contracts, and involvement of multiple systems (Raffel et al., 2013, Willging et al., 2015). Furthermore, parental involvement is often involuntary and parental needs are multifaceted, and come with compliance requirements for each of the systems in which they are involved (Akin et al., 2015).

This implementation study was part of a statewide, federally-funded family drug treatment court (FDTC) evaluation project (“state grant project”). The study was conducted at the end of the second year of a three year evaluation and aimed to enhance the capacity of communities to respond to trauma-related needs of child welfare involved families participating in five FDTC settings across a Midwestern state. This state's initiative involved (1) implementing two evidence-based parenting programs, (2) building community capacity and collaboration to better serve families affected by substance use, and (3) addressing disparities in behavioral and other healthcare service utilization among families involved in the child welfare system with substance-abuse related issues.

Five counties were engaged by the state office of the Supreme Court to implement the Strengthening Families Program (SFP) and Celebrating Families! (CF!). SFP is a 14-week family skills training program aimed at increasing resilience among at-risk parents and children and designed to decrease difficulties in children's behavioral, emotional, academic, and social interactions (Kumpfer & Alvarado, 2003). The program's weekly group sessions comprise three parts: a family meal, break-out sessions where parents and children are separated for role-specific and age-appropriate curriculum, and family time where parents and children practice newly learned skills in the presence of the group facilitators. The SFP curriculum has been used in numerous countries and adapted to meet the needs of culturally diverse populations (Kumpfer and Alvarado, 2003, Kumpfer et al., 2002, Kumpfer and Johnson, 2011, Kumpfer et al., 2010). At the time of the study, SFP was listed on the National Registry of Evidence-based Programs and Practices (NREPP; www.nrepp.samhsa.gov).

CF! was also listed on the NREPP. It follows a 14-week curriculum; yet, unlike SFP, was designed for families with a parent in recovery from substance use disorder(s) (Brook, Akin, Lloyd, & Yan, 2015). Like SFP, it includes a family meal, break-out sessions, and joint parent–child sessions. The CF! curriculum has previously been paired with the SFP curriculum for FDTC-involved families in a different state (Brook et al., 2015).

It is important to note that three key aspects of the project did not occur as originally planned for the implementation. First, the state office responsible for the project established a Memoranda of Understanding (MOU) with foster care agencies, which involved a funding mechanism to supplement the federal grant. However, the MOU fell apart early in the grant period and the project proceeded with the federal grant covering training expenses but not any new or designated funding source for SFP or CF! service provision. Second, project leaders initially intended to pair SFP and CF! in a 28-week sequence; however, CF! was delivered only once at one site and was subsequently dropped across all sites. Third, although project leaders originally intended to fund SFP/CF! delivery in partnership with local FDTCs, it did not occur consistently across sites. All program participants were also FDTC participants, as was required by the funder; however, support from the judge and FDTC team varied considerably across sites.

Below is a brief description of the circumstances surrounding implementation at each site:

  • Site A: The FDTC and one agency signed a Memoranda of Understanding to implement SFP and CF! at the start of the state grant project. Following a two-year delay, SFP was implemented. At the time the interviews for the present study were conducted, the first SFP implementation was underway. Prior to SFP implementation, the state project dropped CF!.

  • Site B: One agency (without FDTC involvement) received a separate housing grant to implement an EBI and chose to implement SFP in partnership with the state grant project. The housing grant did not cover CF!, thus CF! was dropped. SFP was implemented and at the time of interviews, four rounds of SFP had been conducted.

  • Site C: One agency (with FDTC support) implemented both SFP and CF! in sequence as was originally planned. The site was beginning its second round of SFP when interviews were conducted. Around the time of interviews, the state grant project dropped CF!.

  • Site D: One agency (with FDTC support) implemented SFP only, and completed one 14-week SFP group at the time of the study.

  • Site E: Without FDTC support, agencies partnered to implement SFP only, and completed one 14-week SFP group at the time of the study.

The study research questions were designed to investigate the differential implementation of SFP across the five sites and the factors associated with the discontinuation of the CF! program. Using the implementation framework described above, our research questions were:

  • (1)

    What supports and barriers across the hypothesized implementation factors led to the successful implementation of SFP?

  • (2)

    What supports and barriers across the hypothesized implementation factors led to failed implementation of CF!?

Section snippets

Sample

Site coordinators and agency practitioners involved with the delivery and implementation of the SFP program in five counties were invited to participate in the study via email. Fifteen staff members agreed to participate in semi-structured phone interviews. The Institutional Review Board of the [blinded] approved all of the study's procedures.

Participants included the state grant project director, three state-level site coordinators, four county-level site coordinators, two agency CEOs, and

Results

The study's results are organized by the six implementation factors. Each factor is described in terms of the supports and challenges reported across the implementation sites. While each factor is distinguished by its own features and dynamics, a good deal of overlap and interconnectedness also exists among the factors. These results, therefore, reflect these intersections, similarities, and interrelatedness across implementation factors.

Overall, implementation of SFP and CF! were dramatically

Discussion

Through this study, we identified important insights into the experiences of child welfare and community-based professionals implementing two evidence-based parenting interventions with substance-affected families involved with the child welfare system. Our study builds on the prior literature by exploring implementations that were both successful and failed, and discusses these findings within an implementation framework that illuminates facilitators and barriers. This study offers a unique

Conclusion

Although our results revealed that one EBI was viewed more positively than the other, they also showed that successful implementation was multidimensional, multilevel, and involved both celebrating small victories and grappling with challenges. Many challenges were resolved, others were not. As scholars have asserted, real world implementation is nearly always a mix of small successes and unpredictable setbacks (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). This study also helps to

Acknowledgements

This manuscript was part of the Iowa Court Collaborative, which was funded by the Iowa Office of Children's Justice via the Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services. The article's contents are solely the responsibility of the authors and do not necessarily represent the official views of the Iowa Office of Children's Justice or SAMHSA.

References (39)

  • G.A. Aarons et al.

    Evidence-based practice implementation and staff emotional exhaustion in children's services

    Behaviour Research and Therapy

    (2009)
  • B.A. Akin et al.

    Implementation of an evidence-based intervention to reduce long-term foster care: Practitioner perceptions of key challenges and supports

    Children and Youth Services Review

    (2014)
  • T.M. Crea et al.

    Variability in the implementation of Team Decisionmaking (TDM): Scope and compliance with the family to family practice model

    Children and Youth Services Review

    (2008)
  • G.A. Aarons

    Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS)

    Mental Health Services Research

    (2004)
  • G.A. Aarons et al.

    Expanding the domains of attitudes towards evidence-based practice: The evidence based practice attitude scale-50

    Administration and Policy in Mental Health and Mental Health Services Research

    (2012)
  • G.A. Aarons et al.

    Mixed-method study of a conceptual model of evidence-based intervention sustainment across multiple public-sector service settings

    Implementation Science

    (2014)
  • G.A. Aarons et al.

    Advancing a conceptual model of evidence-based practice implementation in public service sectors

    Administration and Policy in Mental Health and Mental Health Services Research

    (2011)
  • G.A. Aarons et al.

    Implementation of evidence-based practice in child welfare: Service provider perspectives

    Administration and Policy in Mental Health and Mental Health Services Research

    (2007)
  • G.A. Aarons et al.

    The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect

    Journal of Consulting and Clinical Psychology

    (2009)
  • G.A. Aarons et al.

    Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis

    American Journal of Public Health

    (2009)
  • B. Akin et al.

    Worker perspectives from the front line: Implementation of evidence-based interventions in child welfare settings

    Journal of Child and Family Studies

    (2015)
  • R.P. Barth et al.

    Parent-training programs in child welfare services: Planning for more evidence-based approach to serving biological parents

    Research on Social Work Practice

    (2005)
  • R.C. Bogdan et al.

    Qualitative research for education: An introduction to theories and methods

    (2003)
  • J. Brook et al.

    Family drug court, targeted parent training, and family reunification: Did this enhanced service strategy make a difference?

    Juvenile & Family Court Journal

    (2015)
  • J. Century et al.

    Measuring enactment of innovations and the factors that affect implementation and sustainability: Moving toward common language and shared conceptual understanding

    Journal of Behavioral Health Services & Research

    (2012)
  • L.J. Damschroder et al.

    Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science

    Implementation Science

    (2009)
  • D.L. Fixsen et al.

    Core implementation components

    Research on Social Work Practice

    (2009)
  • J.F. Gilgun

    Hypothesis generation in social work research

    Journal of Social Service Research

    (1992)
  • M. Gray et al.

    Implementing evidence-based practice a review of the empirical research literature

    Research on Social Work Practice

    (2013)
  • Cited by (19)

    • Barriers and facilitators to the implementation of parenting programs in real-world settings: A qualitative systematic review

      2022, Mental Health and Prevention
      Citation Excerpt :

      In multiple instances, both providers’ self-efficacy and skill proficiency were associated by informants with factors related to the prevention support system theme, with one study identifying that “[c]linicians frequently highlighted the benefit of high-quality supervision, ultimately increasing their sense of competence and confidence with [the program]” (Reay et al., 2019, pp. 252–253) and another identifying that provider supervision sessions and “the structured training they received prior to and during program implementation facilitated their ability to implement the program (Agazzi et al., 2019, p. 2180). Ten studies (Akin et al., 2016; Bekkema et al., 2008; Christian et al., 2014; Côté & Gagné, 2020; Czymoniewicz-Klippel et al., 2017; Falconer et al., 2020; McLennan, 2010; Reay et al., 2019; Shapiro et al., 2015; Westerlund et al., 2017) reported that program compatibility was an important facilitator of program implementation. Conversely, five studies (Akin et al., 2016; Christian et al., 2014; McLennan, 2010; Reay et al., 2019; Stern et al., 2008) reported eight examples of program incompatibility as a barrier to program implementation.

    • Preliminary effectiveness of an LGBTQ+ affirmative parenting intervention with foster parents

      2021, Children and Youth Services Review
      Citation Excerpt :

      At least one of the facilitators at each site was a member of the LGBTQ + community, and at least one of facilitators had expertise working within the foster care system. Consistent with implementation best practices (Akin et al., 2016; Lambert et al., 2014; Armstrong et al., 2014), all facilitators attended a two-day, 16-hour training on AFFIRM Caregiver and participated in weekly coaching calls with the interventionists who developed the model. At both sites, sessions were recorded, and a random sample of these sessions were assessed for fidelity by the developers of AFFIRM Caregiver.

    • Factors that predict child welfare caseworker referrals to an evidence-based parenting program

      2020, Children and Youth Services Review
      Citation Excerpt :

      An additional factor worth further investigation is the lack of caseworker referrals to EBPs (Dorsey, Kerns, Trupin, Conover, & Berliner, 2012; Whitaker et al., 2015), as research has shown that child welfare caseworker referrals facilitate access to services for children and families in the child welfare system (Bunger, Stiffman, Foster, & Shi, 2010; Stiffman, Pescosolido, & Cabassa, 2004). Despite caseworkers being at the forefront of EBP implementation for their role as conduits to services for families, research on EBP implementation has largely focused on intervention, clinician/provider and organizational factors (Aarons, Wells, Zagursky, Fettes, & Palinkas, 2009; Akin et al., 2016; Kerns et al., 2014) necessitating further research on the caseworker referral mechanism. As previously mentioned, there is limited research on child welfare caseworker referrals to EBPs.

    View all citing articles on Scopus
    View full text