Skip to main content

Advertisement

SpringerLink
Log in

Validity of Therapist Self-Report Ratings of Fidelity to Evidence-Based Practices for Adolescent Behavior Problems: Correspondence between Therapists and Observers

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Developing therapist-report fidelity tools to support quality delivery of evidence-based practices in usual care is a top priority for implementation science. This study tested the reliability and accuracy of two groups of community therapists who reported on their use of family therapy (FT) and motivational interviewing/cognitive-behavioral therapy (MI/CBT) interventions during routine treatment of inner-city adolescents with conduct and substance use problems. Study cases (n = 45) were randomized into two conditions: (a) Routine Family Therapy (RFT), consisting of a single site that featured family therapy as its standard of care for behavioral treatment; or (b) Treatment As Usual (TAU), consisting of five sites that featured non-family approaches. Therapists and trained observational raters provided FT and MI/CBT adherence ratings on 157 sessions (104 RFT, 53 TAU). Overall therapist reliability was adequate for averaged FT ratings (ICC = .66) but almost non-existent for MI/CBT (ICC = .06); moreover, both RFT and TAU therapists were more reliable in reporting on FT than on MI/CBT. Both groups of therapists overestimated the extent to which they implemented FT and MI/CBT interventions. Results offer support for the feasibility of using existing therapist-report methods to anchor quality assurance procedures for FT interventions in real-world settings, though not for MI/CBT.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Core element interventions, which refer to specific treatment techniques, should not be confused with common elements/factors in therapy, which refer to non-specific relational processes germane to virtually every psychotherapy approach and thought to promote client gains (e.g., therapeutic alliance, positive regard).

References

  • Aarons, G. A., Hurlburt, M., & Horowitz, S. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23.

    Article  PubMed Central  PubMed  Google Scholar 

  • American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, DC: APA.

  • Baldwin, S. A., Christian, S., Berkeljon, A., Shadish, W., & Bean, R. (2012). The effects of family therapies for adolescent delinquency and substance abuse: A meta-analysis. Journal of Marital and Family Therapy, 38, 281–304.

    Article  PubMed  Google Scholar 

  • Ball, S. A., Martino, S., Nich, C., Frankforter, T. L., Van Horn, D., Crits-Christoph, P., et al. (2007). Site matters: Multisite randomized trial of motivational enhancement therapy in community drug abuse clinics. Journal of Consulting and Clinical Psychology, 75, 556–567.

    Article  PubMed Central  PubMed  Google Scholar 

  • Barth, R. P., Kolivoski, K. M., Lindsey, M. A., Lee, B. R., & Collins, K. S. (2014). Translating the common elements approach: Social work’s experiences in education, practice, and research. Journal of Clinical Child and Adolescent Psychology, 43(2), 301–311.

    Article  PubMed  Google Scholar 

  • Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward, A., Ugueto, A. M., et al. (2013). More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health and Mental Health Services Research, 40, 518–529.

    Article  PubMed Central  PubMed  Google Scholar 

  • Bearsley-Smith, C., Sellick, K., Chesters, J., & Francis, K. (2008). Treatment content in child and adolescent mental health services: Development of the Treatment Recording Sheet. Administration and Policy in Mental Health and Mental Health Services Research, 35, 423–435.

    Article  PubMed  Google Scholar 

  • Becker, S., & Curry, J. (2008). Outpatient interventions for adolescent substance abuse: A quality of evidence review. Journal of Consulting and Clinical Psychology, 76, 531–543.

    Article  PubMed  Google Scholar 

  • Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of cognitive therapy performance: Do therapists know how competent they are? Behavioural and Cognitive Psychotherapy, 36, 581–587.

    Article  Google Scholar 

  • Carroll, K. M., Ball, S. A., Nich, C., Martino, S., Frankforter, T. L., Farentinos, C., et al. (2006). Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: A multisite effectiveness study. Drug and Alcohol Dependence, 81, 301–312.

    Article  PubMed Central  PubMed  Google Scholar 

  • Carroll, K. M., Nich, C., & Rounsaville, B. J. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8, 307–320.

    Article  Google Scholar 

  • Carroll, K. M., Nich, C., Sifry, R., Nuro, K. F., Frankforter, T. L., Ball, S. A., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug & Alcohol Dependence, 57, 225–238.

    Article  CAS  Google Scholar 

  • Chapman, J. E., McCart, M. R., Letourneau, E. J., & Sheidow, A. J. (2013). Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting and Clinical Psychology, 81, 674–680.

    Article  PubMed Central  PubMed  Google Scholar 

  • Chorpita, B. F. (2007). Modular cognitive-behavioral therapy for childhood anxiety disorders. New York: Guilford.

    Google Scholar 

  • Chorpita, B. F., Becker, K. D., & Daleiden, E. L. (2007). Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 647–652.

    Article  PubMed  Google Scholar 

  • Chorpita, B., Daleiden, E., Ebesutani, C., Young, J., Becker, K., Nakamura, B., et al. (2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice, 18, 154–172.

    Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Modularity in the design and application of therapeutic interventions. Applied & Preventive Psychology, 11, 141–156.

    Article  Google Scholar 

  • Chorpita, B. F., Weisz, J. R., Daleiden, E. L., Schoenwald, S. K., Palinkas, L. A., Miranda, J., et al. (2013). Long-term outcomes for the Child STEPs randomized effectiveness trial: A comparison of modular and standard treatment designs with usual care. Journal of Consulting and Clinical Psychology, 81, 999–1009.

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

    Article  Google Scholar 

  • Cook, J. M., Biyanova, T., Elhai, J., Schnurr, P. P., & Coyne, J. C. (2010). What do psychotherapists really do in practice? An internet study of over 2,000 practitioners. Psychotherapy: Theory Research, Practice, Training, 47, 260–267.

    Article  Google Scholar 

  • Deming, W. E. (1986). Out of the crisis. Cambridge, MA: MIT Press.

    Google Scholar 

  • Dennis, M. L., Godley, S. H., Diamond, G., Tims, F. M., Babor, T., Donaldson, J., et al. (2004). The Cannabis Youth Treatment (CYT) study: Main findings from two randomized trials. Journal of Substance Abuse Treatment, 27, 197–213.

    Article  PubMed  Google Scholar 

  • Diamond, G. M., Diamond, G. S., & Hogue, A. (2007). Attachment-based family therapy: Adherence and differentiation. Journal of Marital and Family Therapy, 33, 177–191.

    Article  PubMed  Google Scholar 

  • Embry, D. D., & Biglan, A. (2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychology Review, 11, 75–113.

    Article  PubMed Central  PubMed  Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

  • Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105–112.

    Article  Google Scholar 

  • Gallo, K. P., & Barlow, D. H. (2012). Factors involved in clinician adoption and nonadoption of evidence-based interventions in mental health. Clinical Psychology: Science and Practice, 19, 93–106.

    Google Scholar 

  • Garland, A. F., Bickman, L., & Chorpita, B. F. (2010a). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 37, 15–26.

    Article  PubMed Central  PubMed  Google Scholar 

  • Garland, A. F., Haine-Schlagel, R., Brookman-Frazee, L., Baker-Ericzen, M., Trask, E., & Fawley-King, K. (2013). Improving community-based mental health care for children: Translating knowledge into action. Administration and Policy in Mental Health and Mental Health Services Research, 40, 6–22.

    Article  PubMed Central  PubMed  Google Scholar 

  • Garland, A. F., Hawley, K. M., Brookman-Frazee, L. I., & Hurlburt, M. (2008). Identifying common, core elements of evidence-based practice for children with disruptive behavior disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 505–514.

    Article  PubMed  Google Scholar 

  • Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M., & Accurso, E. C. (2010b). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research, 37, 208–220.

    Article  PubMed Central  PubMed  Google Scholar 

  • Gifford, E. V., Tavakoli, S., Weingardt, K. R., Finney, J. W., Pierson, H. M., et al. (2012). How do components of evidence-based psychological treatment cluster in practice? A survey and cluster analysis. Journal of Substance Abuse Treatment, 42, 45–55.

    Article  PubMed  Google Scholar 

  • Hawkins, E. H. (2009). A tale of two systems: Co-occurring mental health and substance abuse disorders treatment for adolescents. Annual Review of Psychology, 60, 197–227.

    Article  PubMed  Google Scholar 

  • Hepner, K. A., Azocar, F., Greenwood, G. L., Miranda, J., & Burnam, M. A. (2010). Development of a clinician report measure to assess psychotherapy for depression in usual care settings. Administration and Policy in Mental Health and Mental Health Services Research, 37, 221–229.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hoagwood, K., Atkins, M., & Ialongo, N. (2013). Unpacking the black box of implementation: The next generation of policy, research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 40, 451–455.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., & Dauber, S. (2013). Assessing fidelity to evidence-based practices in usual care: The example of family therapy for adolescent behavior problems. Evaluation and Program Planning, 37, 21–30.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C. E., Inclan, J., et al. (2008a). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137–147.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Dauber, S., & Henderson, C. E. (2012). Therapist self-report of evidence-based practices in usual care for adolescent behavior problems: Factor and construct validity. Administration and Policy in Mental Health and Mental Health Services Research, 41(1), 126–139. doi:10.1007/s10488-012-0442-8.

    Article  Google Scholar 

  • Hogue, A., Dauber, S., Samuolis, J., & Liddle, H. A. (2006). Treatment techniques and outcomes in multidimensional family therapy for adolescent behavior problems. Journal of Family Psychology, 20, 535–543.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Henderson, C. E., Dauber, S., Barajas, P., Fried, A., & Liddle, H. A. (2008b). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., & Liddle, H. A. (2009). Family-based treatment for adolescent substance abuse: Controlled trials and new horizons in services research. Journal of Family Therapy, 31, 126–154.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Liddle, H., Rowe, C., Turner, R., Dakof, G. A., & LaPann, K. (1998). Treatment adherence and differentiation in individual versus family therapy for adolescent drug abuse. Journal of Counseling Psychology, 45, 104–114.

    Article  Google Scholar 

  • Hogue, A., Ozechowski, T. J., Robbins, M. R., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology: Science and Practice, 20, 60–77.

    Google Scholar 

  • Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244.

    Article  PubMed Central  PubMed  Google Scholar 

  • Kazak, A., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T., Vargas, L., et al. (2010). A meta-systems approach for evidence-based practice for children and adolescents. American Psychologist, 65, 85–97.

    Article  PubMed  Google Scholar 

  • Korathu-Larson, P., Park, A., Fierstein, M., & Chorpita, B. F. (2011). Therapist satisfaction with multiple designs of evidence-based treatments: Did therapists match with MATCH?. Toronto, Canada: The Association of Behavioral and Cognitive Therapies (ABCT).

    Google Scholar 

  • Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193.

    Article  PubMed Central  PubMed  Google Scholar 

  • McHugh, R., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatment: A review of current efforts. American Psychologist, 65, 73–84.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20, 14–32.

    Google Scholar 

  • Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology, 72, 1050–1062.

    Article  PubMed  Google Scholar 

  • O’Neill, S. M., Hempel, S., Lim, Y. W., Danz, M. S., Foy, R., Suttorp, M. J., et al. (2011). Identifying continuous quality improvement publications: What makes an improvement intervention ‘CQI’? British Medical Journal Quality & Safety, 20, 1011–1019.

    Article  Google Scholar 

  • Perepletchikova, F. (2011). On the topic of treatment integrity. Clinical Psychology: Science and Practice, 18, 148–153.

    Google Scholar 

  • Regan, J., Daleiden, E. L., & Chorpita, B. F. (2013). Integrity in mental health systems: An expanded framework for managing uncertainty in clinical care. Clinical Psychology: Science and Practice, 20, 78–98.

    Google Scholar 

  • Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s…fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18, 142–147.

    Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43.

    Article  PubMed Central  PubMed  Google Scholar 

  • Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child and Adolescent Psychology, 34(4), 658–670.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Sheidow, A. J., & Chapman, J. E. (2009). Clinical supervision in treatment transport: Effects on adherence and outcomes. Journal of Consulting and Clinical Psychology, 77, 410–421.

    Article  PubMed Central  PubMed  Google Scholar 

  • Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E., et al. (1998). The Mini International Neuropsychiatric Interview (MINI): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry, 59, 22–33.

    PubMed  Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

    Article  CAS  PubMed  Google Scholar 

  • Thorndike, R. (2007). Fisher’s Z transformation. In N. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 361–365). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., et al. (2013). Tracking evidence based practice with youth: Validity of MATCH and standard manual consultation records. Journal of Clinical Child and Adolescent Psychology, 42, 44–55.

    Article  PubMed  Google Scholar 

  • Weersing, R. V., Weisz, J. R., & Donenberg, G. R. (2002). Development of the Therapy Procedures Checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.

    Article  Google Scholar 

  • Weisz, J. R., Chorpita, B. F., Palinkas, L. A., Schoenwald, S. K., Miranda, J., Bearman, S. K., et al. (2012). Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: A randomized effectiveness trial. Archives of General Psychiatry, 69, 274–282.

    Article  PubMed  Google Scholar 

  • Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2005). EBP does not equal EST: Reply to Crits-Christoph et al. (2005) and Weisz et al. (2005). Psychological Bulletin, 131, 427–433.

    Article  Google Scholar 

  • Winters, K. C., Stinchfield, R., Latimer, W. W., & Lee, S. (2007). Long-term outcome of substance-dependent youth following 12-step treatment. Journal of Substance Abuse Treatment, 33, 61–69.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Preparation of this article was supported by grant R01DA023945 from the National Institute on Drug Abuse. The authors would like acknowledge the dedicated work of the observational coders for this project (in addition to authors EL and MB): Daniela Caraballo, Benjamin Goldman, Jacqueline Horan, Candace Johnson, Akiyo Kodera, Emily McSpadden, Annie Niermann, and Gabi Spiewak. We are also extremely grateful to the community therapists working at six study sites who generously agreed to participate in a research study and submit recordings of their sessions in the hopes of advancing the clinical science in their field.

Author information

Authors and Affiliations

  1. Treatment Research Division, The National Center on Addiction and Substance Abuse at Columbia University, 633 Third Avenue, 19th floor, New York, NY, 10017, USA

    Aaron Hogue, Sarah Dauber, Emily Lichvar & Molly Bobek

  2. Department of Psychology, Sam Houston State University, Huntsville, TX, 77340, USA

    Craig E. Henderson

Authors
  1. Aaron Hogue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah Dauber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emily Lichvar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Molly Bobek
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Craig E. Henderson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aaron Hogue.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hogue, A., Dauber, S., Lichvar, E. et al. Validity of Therapist Self-Report Ratings of Fidelity to Evidence-Based Practices for Adolescent Behavior Problems: Correspondence between Therapists and Observers. Adm Policy Ment Health 42, 229–243 (2015). https://doi.org/10.1007/s10488-014-0548-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-014-0548-2

Keywords

Search

Navigation