Skip to main content

Advertisement

SpringerLink
Log in

Reliability of Therapist Self-Report on Treatment Targets and Focus in Family-Based Intervention

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Reliable therapist-report methods appear to be an essential component of quality assurance procedures to support adoption of evidence-based practices in usual care, but studies have found weak correspondence between therapist and observer ratings of treatment techniques. This study examined therapist reliability and accuracy in rating intervention target (i.e., session participants) and focus (i.e., session content) in a manual-guided, family-based preventive intervention implemented with 50 inner-city adolescents at risk for substance use. A total of 106 sessions selected from three phases of treatment were rated via post-session self-report by the participating therapist and also via videotape by nonparticipant coders. Both groups estimated the amount of session time devoted to model-prescribed treatment targets (adolescent, parent, conjoint) and foci (family, school, peer, prosocial, drugs). Therapists demonstrated excellent reliability with coders for treatment targets and moderate to high reliability for treatment foci across the sample and within each phase. Also, therapists did not consistently overestimate their degree of activity with targets or foci. Implications of study findings for fidelity assessment in routine settings are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Barber, J. P., Sharpless, B., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology, 38, 493–500.

    Article  Google Scholar 

  • Barnoski, R. (2003). Outcome evaluation of Washington State’s research-based programs for juvenile offenders. Washington State Institute for Public Policy (wsipp.wa.gov).

  • Bearsley-Smith, C., Sellick, K., Chesters, J., & Francis, K. (2008). Treatment content in child and adolescent mental health services: Development of the treatment recording sheet. Administration and Policy in Mental Health and mental Health Services Research, 35, 423–435.

    Article  PubMed  Google Scholar 

  • Becker, S. J., & Curry, J. F. (2008). Outpatient interventions for adolescent substance abuse: A quality of evidence review. Journal of Consulting and Clinical Psychology, 76, 531–543.

    Article  PubMed  Google Scholar 

  • Carroll, K. M., Nich, C., & Rounsaville, B. J. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8, 307–320.

    Article  Google Scholar 

  • Cicchetti, D. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

    Article  Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Diamond, G. S., & Liddle, H. A. (1996). Resolving a therapeutic impasse between parents and adolescents in multidimensional family therapy. Journal of Consulting and Clinical Psychology, 64, 481–488.

    Article  CAS  PubMed  Google Scholar 

  • Diamond, G. S., & Liddle, H. A. (1999). Transforming negative parent-adolescent interactions: From impasse to dialogue. Family Process, 38, 5–26.

    Article  CAS  PubMed  Google Scholar 

  • Dobson, K. S., & Shaw, B. F. (1993). The training of cognitive therapists: What have we learned from treatment manuals? Psychotherapy, 30, 573–577.

    Article  Google Scholar 

  • Fixsen, D.L., Naoom, S.F., Blasé, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la parte Florida Mental Health Institute, The National Implementation Research Network. (Report No.: FMHI Publication #231).

  • Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105–112.

    Article  Google Scholar 

  • Garland, A. F., Bickman, L., & Chorpita, B. F. (2010a). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 37, 15–26.

    Article  PubMed Central  PubMed  Google Scholar 

  • Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M., & Accurso, E. C. (2010b). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research, 37, 208–220.

    Article  PubMed Central  PubMed  Google Scholar 

  • Glisson, C., Schoenwald, S. K., Hemmelgarn, A., Green, P., Dukes, D., Armstrong, K. S., et al. (2010). Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology, 78, 537–550.

    Article  PubMed Central  PubMed  Google Scholar 

  • Henggeler, S. W., & Sheidow, A. J. (2012). Empirically supported family-based treatments for conduct disorder and delinquency in adolescents. Journal of Marital and Family Therapy, 38, 30–58.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hoagwood, K. E. (2005). Family-based services in children’s mental health: A research review and synthesis. Journal of Child Psychology and Psychiatry, 46, 690–713.

    Article  PubMed  Google Scholar 

  • Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C. E., Inclan, J., et al. (2008a). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137–147.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Dauber, S., Samuolis, J., & Liddle, H. A. (2006). Treatment techniques and outcomes in multidimensional family therapy for adolescent behavior problems. Journal of Family Psychology, 20, 535–543.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008b). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Johnson-Leckrone, J., & Liddle, H. A. (1999). Recruiting high-risk families into family-based prevention and prevention research. Journal of Mental Health Counseling, 21, 337–351.

    Google Scholar 

  • Hogue, A., & Liddle, H. A. (1999). Family-based preventive intervention: An approach to preventing substance use and antisocial behavior. American Journal of Orthopsychiatry, 69, 278–293.

    Article  CAS  PubMed  Google Scholar 

  • Hogue, A., Liddle, H. A., & Becker, D. (2002a). Multidimensional family prevention for at-risk adolescents. In T. Patterson (Ed.), Comprehensive handbook of psychotherapy, Vol. II Cognitive-behavioral approaches (pp. 141–166). New York: Wiley.

    Google Scholar 

  • Hogue, A., Liddle, H. A., Becker, D., & Johnson-Leckrone, J. (2002b). Family-based prevention counseling for high-risk young adolescents: Immediate outcomes. Journal of Community Psychology, 30, 1–22.

    Article  Google Scholar 

  • Hogue, A., Liddle, H. A., Dauber, S., & Samuolis, J. (2004). Linking session focus to treatment outcome in evidence-based treatments for adolescent substance abuse. Psychotherapy, 41, 83–96.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33, 332–345.

    Article  Google Scholar 

  • Hogue, A., Liddle, H. A., Singer, A., & Leckrone, J. (2005). Intervention fidelity in family-based prevention counseling for adolescent problem behaviors. Journal of Community Psychology, 33, 191–211.

    Article  Google Scholar 

  • Hogue, A., Ozechowski, T. J., Robbins, M. S., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology, 20, 60–77.

    Google Scholar 

  • Huey, S., Henggeler, S. W., Brondino, M. J., & Pickrel, S. G. (2000). Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting & Clinical Psychology, 68, 451–467.

    Article  Google Scholar 

  • Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244.

    Article  PubMed Central  PubMed  Google Scholar 

  • Kelley, S. D., Vides de Andrade, A. R., Sheffer, E., & Bickman, L. (2010). Exploring the black box: Measuring youth treatment process and progress in usual care. Administration and Policy in Mental Health and Mental Health Services Research, 37, 287–300.

    Article  PubMed  Google Scholar 

  • Liddle, H. A., Rowe, C. L., Gonzalez, A., Henderson, C. E., Dakof, G. A., & Greenbaum, P. E. (2006). Changing provider practices, program environment, and improving outcomes by transporting multidimensional family therapy to an adolescent drug treatment setting. The American Journal on Addictions, 15, 102–112.

    Article  PubMed  Google Scholar 

  • Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193.

    Article  PubMed Central  PubMed  Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. American Psychologist, 65, 73–84.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology, 20, 14–32.

    PubMed  Google Scholar 

  • Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology, 12, 365–383.

    Google Scholar 

  • Robbins, M. S., Feaster, D. J., Horigian, V. E., Puccinelli, M. J., Henderson, C. E., & Szapocznik, J. (2011). Therapist adherence in brief strategic family therapy for adolescent drug abusers. Journal of Consulting & Clinical Psychology, 79, 43–53.

    Article  Google Scholar 

  • Schmidt, S. E., Liddle, H. A., & Dakof, G. A. (1996). Changes in parenting practices and adolescent drug abuse during multidimensional family therapy. Journal of Family Psychology, 10, 12–27.

    Article  Google Scholar 

  • Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s…fidelity measurement in the real world. Clinical Psychology, 18, 142–147.

    PubMed Central  PubMed  Google Scholar 

  • Schoenwald, S. K., Chapman, J. E., Kelleher, K. E., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35, 84–97.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43.

    Article  PubMed Central  PubMed  Google Scholar 

  • Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child and Adolescent Psychology, 34, 658–670.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Sheidow, A., & Letourneau, E. J. (2004). Toward effective quality assurance in evidence-based practice: Links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology, 33, 94–104.

    Article  PubMed  Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

    Article  CAS  PubMed  Google Scholar 

  • Waldron, H. B., & Turner, C. W. (2008). Evidence-based psychosocial treatments for adolescent substance abuse. Journal of Clinical Child and Adolescent Psychology, 37, 238–261.

    Article  PubMed  Google Scholar 

  • Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211.

    Article  PubMed  Google Scholar 

  • Weersing, R. V., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.

    Article  Google Scholar 

  • Zazzali, J. L., Sherbourne, C., Hoagwood, K. E., Greene, D., Bigley, M. F., & Sexton, T. (2008). The adoption and implementation of an evidence based practice in child and family mental health services organizations: A pilot study of functional family therapy in New York State. Administration and Policy in Mental Health and Mental Health Services Research, 35, 38–49.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Preparation of this article was supported by Grants R01DA023945 from the National Institute on Drug Abuse and HD1SP07054 from the Center for Substance Abuse Prevention. The authors are grateful for the dedicated work of the prevention counselors at Temple Teen Care and the observational coders for the TBRS-2 project.

Author information

Authors and Affiliations

  1. Treatment Research Division, National Center on Addiction and Substance Abuse (CASA) at Columbia University, 633 Third Avenue, 19th floor, New York, NY, 10017, USA

    Aaron Hogue & Sarah Dauber

  2. Sam Houston State University, Huntsville, TX, USA

    Craig E. Henderson

  3. Center for Treatment Research on Adolescent Drug Abuse, University of Miami Miller School of Medicine, Miami, FL, USA

    Howard A. Liddle

Authors
  1. Aaron Hogue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah Dauber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Craig E. Henderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Howard A. Liddle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aaron Hogue.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hogue, A., Dauber, S., Henderson, C.E. et al. Reliability of Therapist Self-Report on Treatment Targets and Focus in Family-Based Intervention. Adm Policy Ment Health 41, 697–705 (2014). https://doi.org/10.1007/s10488-013-0520-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-013-0520-6

Keywords

Search

Navigation