Elsevier

Child Abuse & Neglect

Volume 53, March 2016, Pages 40-50
Child Abuse & Neglect

Research article
Filling potholes on the implementation highway: Evaluating the implementation of Parent–Child Interaction Therapy in Los Angeles County

https://doi.org/10.1016/j.chiabu.2015.11.011Get rights and content

Abstract

In October 2012, first 5 LA funded a unique collaboration between Los Angeles County Department of Mental Health (DMH) and UC Davis PCIT Training Center (UCD PCIT) to train county-contracted agencies to provide Parent–Child Interaction Therapy (PCIT). This $20 million dollar, 5-year grant represented the largest implementation effort of an empirically based treatment to date. The purpose of this paper was to describe the first 2 years of the implementation process of this project, beginning with project start up and pre-implementation phases, and to present agency training and client performance outcomes from our first year of training. Results presented in this evaluation suggest that it is possible to train LA County providers in PCIT, and that PCIT is an effective intervention for DMH-contracted providers in LA County. This evaluation also discusses challenges to successful implementation. Barriers to progress included unanticipated delays building county infrastructure, trainee attrition, and insufficient client referrals. We discuss the results of the current implementation with respect to theory, research, and others’ training models, with the aim of evaluating and prioritizing different implementation drivers, noting the ongoing competition between knowing what to do and the need for action.

Introduction

Implementation science scholars have shown us that the process of making an empirically based mental health treatment (EBT) accessible to people requires much more than simply training therapists to provide the treatment. EBTs need to be “planted” in an organization. To survive, EBTs require a system for connecting appropriate clients to providers, for maintaining providers’ fidelity to an EBT model, and for sustaining the practice in a site over time. However, the more complex the organizational context in which the implementation takes place, the more complex the implementation process.

The administrative and operational systems that “drive” an organization's ability to implement an EBT because of their ability to facilitate, challenge, and at times actively compete with implementation efforts, are known as implementation drivers. Successful implementation of an EBT in a community mental health agency depends on identifying the implementation drivers in their administrative and operational systems. Successful implementation of an EBT “to scale” in a county or state requires the additional step of identifying the county legal, administrative, and operational systems and procedures that can influence agencies’ ability to create an implementation-friendly environment. For example, agencies have internal billing policies that set productivity expectations for their therapists. These agencies’ internal policies are in turn influenced by county's claiming guidelines and reimbursement rates. Both sets of policies could drive implementation through their effects on the amount of “release” time trainees were permitted for training. Both agency and county systems influence implementation and should be considered in a large-scale implementation project.

While we have clear guidelines for implementing EBTs, it is not always clear which factors in organizational systems will “drive” our implementation toward success, even if pre-training organizational assessments are conducted. Some factors are hidden, some are glossed over by the organization, and some are identified and addressed but inadequately resolved. In an effort to identify and label specific factors that affect the implementation of evidence-based mental health treatments, this paper describes our attempt to implement Parent–Child Interaction Therapy (PCIT) in the Los Angeles County Department of Mental Health (LACDMH) system of care. By reflecting upon expected implementation drivers described by Fixsen, Naoom, Blase, Friedman, and Wallace (2005), Fixsen, Blase, Naoom, and Wallace (2009), and our experiences implementing PCIT during the first 2 years of a 5-year training project, we hope to identify the potholes along the path of implementation, so that others may avoid them. We begin by describing the context of the training, project start up and pre-implementation phases, followed by a description of the training process and outcomes of our first year of training.

Los Angeles County, with a population of approximately 10 million residents (U.S. Census Bureau, 2014), is more densely populated than most states (U.S. Census Bureau, 2014). Los Angeles County Department of Mental Health (LACDMH) is the largest county public mental health system in the United States, with 80 directly operated programs and contracting with more than 700 non-governmental agencies and individual practitioners to provide mental health services (Los Angeles County Department of Mental Health, n.d.). In 2009, the County implemented a Prevention and Early Intervention plan that created strong financial incentives for their contracted providers to use EBTs, which in turn created strong incentives to have clinical staff trained in those EBTs. Since 2010, LACDMH has facilitated trainings in at least 20 EBTs for their directly operated and contracted providers by providing organization, infrastructure, and additional incentives. As of December 2014, there were 33 interventions on their list of approved EBTs. All this paints a picture of a county with a well-established mental health service bureaucracy familiar with the demands of EBT training and their supporting role.

In October 2012, First 5 LA funded Los Angeles County (DMH) and UC Davis PCIT Training Center (UCD PCIT) to collaborate in training therapists in at least 58 county-contracted agencies to provide PCIT over a 5-year period. First 5 LA, a non-profit child advocacy organization funded by a voter-approved tax on tobacco products, contributed $20 million dollars toward a 5-year effort to implement PCIT in LA County. At that time, PCIT was one of the few EBTs for young children that had not been implemented widely in LA County; and the proposal was strongly supported by the chair of the LA County Board of Supervisors. Funding covered the cost of agencies’ capital expenditures to outfit a PCIT treatment room, encrypted video conferencing equipment to support the telehealth training modality, the cost of lost productivity to the agency during training, a 5% match for funding of the cost of client services during training, and associated costs for LACDMH to support and administer the training project. LACDMH was responsible for selecting agencies, coordinating reimbursement for capital expenditures and training time, and supporting training via monthly support groups, consultation, and outreach. UCD PCIT was responsible for the content and coordination of “basic” training (i.e., how to provide PCIT) and for workshops in advanced PCIT topic areas.

Fixsen et al. (2009) identified seven core implementation drivers: staff selection, pre-service and in-service training, ongoing coaching and consultation, performance assessment, decision support data systems, facilitative administrative support, and systems interventions. These are basic mechanisms they found to be critical to the success of the implementation effort. They are conceptualized as falling into one of three categories: leadership, organization, and competency drivers. Some of these mechanisms driving the implementation are engaged before the implementation takes place, and some become active during the training process. Good responsive leadership and effective management anticipate the effects of training on an agency's internal systems, on other programs, and agency staff, and the needs of training participants. Qualities of the organization that facilitate successful implementation of EBTs are the ability to acquire good information about training effectiveness, good communication among staff, and the flexibility to change systems to better support the new EBT. The quality and effectiveness of the training itself are competency drivers, including the selection of appropriate trainees, having content necessary and sufficient to teach core skills, and a mode of teaching and coaching that facilitates learning in trainees. We will define and discuss these terms as they relate to our countywide training efforts.

The context in which a mental health intervention is implemented strongly affects the number of factors that need to be considered in the pre-implementation phase. EBTs implemented in an organization or a large mental health system must pay attention to how well their administrations understand the human factors involved in training and sustaining an EBT. Decreasing trainees’ caseload, adjusting productivity expectations, and planning to accommodate the competing needs of other EBTs that compete for similar types of clients or trainees all show sensitivity to staff needs. Administrations that understand the importance of fidelity in EBTs’ effectiveness are likely to effectively sustain their EBTs over time.

Some EBT trainers assess an organization's readiness to adopt an EBT as part of their practice before beginning training. Before training begins, the trainer will meet with administrators and training coordinators to determine whether the agency's leadership will support implementation of an EBT. The National Child Traumatic Stress Network (NCTSN) developed an “Assessment of Organizational Readiness” tool (Allred et al., 2005) to address these issues. Using this tool or one tailored to the needs of their EBT, trainers assess whether the administration sets policies and procedures that support the level of organization and oversight required by an EBT. They also attend to whether the potential trainees are open to learning different practices and welcome the new intervention, the numbers of clients who would benefit from the intervention, and the compatibility of the new intervention with existing programs. These areas of focus are intended to tease out potential barriers to implementation success.

Often, EBT trainers conduct Pre-service or In-service Trainings for staff and local stakeholders, to provide information about the EBT—its theoretical foundation, appropriate referral information, its effectiveness, and the intervention's potential to improve the lives of the clients. Some trainers use web-based fundamental training as pre-service training (e.g., Trauma Focused-Cognitive Behavioral Therapy training at http://tfcbt.musc.edu/; PCIT training at http://pcit.ucdavis.edu; Dialectical Behavior Therapy and other therapies at http://behavioraltech.org/ol), a strategy found to be significantly more effective than manuals and workshops (Dimeff et al., 2009).

The primary purpose of these fundamental pre-service and in-service trainings, whether given by an expert in person or web-based, is to prepare staff for the new culture of the EBT and build commitment to help sustain its rigors. Staff enthusiasm and commitment to the EBT, and stakeholders who are cheerleaders for its adoption are important drivers of successful implementation (Fixsen et al., 2005). Pre-service or in-service workshops that include community stakeholders can help the community understand the new intervention (at the agency) and clarify funding and referral streams. While research has not supported using only pre-service and in-service models for training (e.g., Herschell, Kolko, Baumann, & Davis, 2010), they are noted as being expedient methods for communicating information (Fixsen et al., 2009). For example, as part of their training models, Multidimensional Treatment Foster Care (Gilliam & Fisher, 2014) conducts community presentations at agency training sites, inviting all agency staff, teachers, social workers, pediatricians, and any community stakeholder who might refer a client to the new program.

As part of the pre-implementation discussion, EBT trainers also discuss with administrators the external systems needed to support the organization's adoption of the EBT: funding and referrals. This may include clarifying funding streams, identifying fundable diagnostic and insurance codes, and delineating grant and service contract inclusion criteria. Additionally, developing a steady stream of appropriate referrals supports the effective use of the EBT, which often involves educating both community members and agency staff. Further, agencies with multiple EBTs may need to reconfigure their internal referral systems (i.e., engage in Systems Intervention) so that they can identify which children should be referred to the new EBT, thus ensuring that the training will prosper and the new program will thrive. Trainers may even postpone training until referral and funding problems have been resolved. To give an example, according to Swenson and Schaeffer (Swenson & Schaeffer, 2014), agencies seeking training in Multi-Systemic Therapy for Child Abuse and Neglect (MST-CAN) must complete a site assessment with an MST-CAN program developer. To be a licensed program, they must complete goals and guidelines, and a feasibility checklist and agree to the terms of MST-CAN implementation such as collaborative relationships, established referral criteria, established program clinical goals, a team structure, and an agreement to implement the program fidelity requirements. There must also be evidence of an effective working relationship with key stakeholders such as Child Protective Services, and evidence of these stakeholders’ commitment to the training. If these criteria are not met, the organization may not be eligible to receive MST-CAN training. Most EBT trainers use this “pre-implementation” time to help agencies make enough changes to allow them to begin the training process, anticipating the need for ongoing consultation to support EBT growth and fidelity to the original (and empirically supported) model.

Staff selection is the process by which decisions are made concerning which staff members are best suited to learn an intervention (Fixsen et al., 2005). Treatment developers and EBT trainers often set guidelines for academic qualifications, experience, and/or licensure; and these guidelines are generally based on the degree of clinical responsibility a job function requires. Apart from these basic guidelines, there is limited understanding regarding which therapist characteristics are best suited to learn to deliver an EBT, which therapists will be more likely to be more successful at acquiring skills in an EBT training program, or who will adhere to protocol and provide higher quality services in the long run. While the notion that some individuals within an organization may be more open to innovation (i.e., “early adopters” as discussed by Rogers, 2003), it is not clear to what degree this thinking applies to implementation of new EBTs. Furthermore, staff selection is more often a choice among a limited selection of treatment providers; and while treatment developers and trainers can give advice on staff selection and may require a clinical supervisor to receive some training in the EBT, the organization often makes the final decision.

Section snippets

Project start-up

Members of the collaborative partnership (i.e., UCD PCIT, LACDMH, and First 5 LA) began meeting by telephone and face-to-face shortly after UC Davis was given official notification of the grant award, but before either of the grantees (i.e., LACDMH and UCD PCIT) had an executed contract in hand. The project requirements described in the Request for Qualification (i.e., call for proposals) made it clear that First 5 LA expected the trainers to “hit the ground running,” and no startup time was

Implementation drivers – supporting the training process

In addition to continued influences of facilitative administration, other implementation drivers begin to influence the process of training. These drivers supported and defined the training process, addressing the quality of training, how progress in training is measured, and competence is defined. Ongoing coaching and consultation are the mechanisms that Fixsen et al. (2005) identified as the way practitioners learn a new intervention. Interestingly, educational researchers have long

Conclusions

The purpose of this paper was to describe from an implementation science point of view, the progress of the first 2 years of a large-scale collaborative project to train therapists in at least 58 agencies to provide PCIT in Los Angeles County. Results document the viability of training a large number of LA County therapists to effectively provide a complex, empirically based treatment. This evaluation also brings to light challenges to the implementation process, and understanding about the

References (29)

  • S.M. Eyberg et al.

    Eyberg child behavior inventory and Sutter-Eyberg student behavior inventory-revised: Professional manual

    (1999)
  • S.M. Eyberg et al.

    Conduct problem behavior: Standardization of a behavioral rating scale with adolescents

    Journal of Clinical Child Psychology

    (1983)
  • D. Fixsen et al.

    Core implementation components

    Research on Social Work Practice

    (2009)
  • D.L. Fixsen et al.

    Implementation research: A synthesis of the literature (FMHI #231)

    (2005)
  • Cited by (0)

    The authors are grateful for the funding and support for this project by First 5 LA, Los Angeles, CA.

    View full text