Abstract
Pragmatic measures of therapist delivery of evidence-based practice (EBP) are critical to assessing the impact of large-scale, multiple EBP implementation efforts. As an initial step in the development of pragmatic measurement, the current study examined the concordance between therapist and observer ratings of items assessing delivery of EBP strategies considered essential for common child EBP targets. Possible EBP-, session-, and therapist-levels factors associated with concordance were also explored. Therapists and independent observers rated the extensiveness of therapist (n = 103) EBP strategy delivery in 680 community psychotherapy sessions in which six EBPs were used. Concordance between therapist- and observer-report of the extensiveness of therapist EBP strategy use was at least fair (ICC ≥ .40) for approximately half of the items. Greater therapist-observer concordance was observed in sessions where a structured EBP was delivered and in sessions where therapists reported being able to carry out planned activities. Findings highlighted conditions that may improve or hinder therapists’ ability to report on their own EBP strategy delivery in a way that is consistent with independent observers. These results can help inform the development of pragmatic therapist-report measures of EBP strategy delivery and implementation efforts more broadly.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We use the term “EBPs” throughout, inclusive of MAP, for ease of communication. We acknowledge that MAP differs from the other EBPs as it is an evidence management system, with a direct service component that coordinates multiple evidence sources, employs practices components drawn from over 700 evidence based treatments, and offers structured process management supports to guide clinical care for anxiety, trauma, depression, conduct problems in children 0–21 years old. In the LACDMH system, the MAP direct service model complements a diverse array of EBPs by allowing providers to select, review, adapt, or construct promising treatments as needed to match particular child characteristics based on the latest scientific findings (see Chorpita and Daleiden 2018).
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65.
Barnett, M., Brookman-Frazee, L., Regan, J., Saifan, D., Stadnick, N., & Lau, A. S. (2017). How intervention and implementation characteristics relate to community therapists’ attitudes toward evidence-based practices: A mixed methods study. Administration and Policy in Mental Health and Mental Health Services Research, 44(6), 824–837. https://doi.org/10.1007/s10488-017-0795-0.
Bearsley-Smith, C., Sellick, K., Chesters, J., Francis, K., & Gippsland Adolescent Depression Research Group. (2008). Treatment content in child and adolescent mental health services: Development of the treatment recording sheet. Administration and Policy in Mental Health and Mental Health Services Research, 35(5), 423–435. https://doi.org/10.1007/s10488-008-0184-9.
Beidas, R. S., Aarons, G., Barg, F., Evans, A., Hadley, T., Hoagwood, K., et al. (2013). Policy to implementation: Evidence-based practice in community mental health – study protocol. Implementation Science, 8(1), 38. https://doi.org/10.1186/1748-5908-8-38.
Borntrager, C. F., Chorpita, B. F., Orimoto, T., Love, A., & Mueller, C. W. (2015). Validity of clinician’s self-reported practice elements on the Monthly Treatment and Progress Summary. The Journal of Behavioral Health Services & Research, 42(3), 367–382. https://doi.org/10.1007/s11414-013-9363-x.
Boustani, M. M., Gellatly, R., Westman, J. G., & Chorpita, B. F. (2017). Advances in cognitive behavioral treatment design: Time for a glossary. The Behavior Therapist, 40(6), 199–208.
Borntrager, C., Chorpita, B., Orimoto, T., Love, A., & Mueller, C. (2013). Validity of clinicians' self-reported practices on the Monthy Treatment and Progress Summary. Journal of Behavioral Health Services and Research, 1–15.
Brookman-Frazee, L., Drahota, A., & Stadnick, N. (2012). Training community mental health therapists to deliver a package of evidence-based practice strategies for school-age children with autism spectrum disorders: A pilot study. Journal of Autism and Developmental Disorders, 42(8), 1651–1661. https://doi.org/10.1007/s10803-011-1406-7.
Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of cognitive therapy performance: Do therapists know how competent they are? Behavioural and Cognitive Psychotherapy, 36(5), 581–587. https://doi.org/10.1017/S1352465808004438.
Carroll, K., Nich, C., & Rounsaville, B. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8(3), 307–320. https://doi.org/10.1080/10503309812331332407.
Child and Adolescent Mental Health Division. (2003). Service provider Monthly Treatment and Progress Summary. Hawaii Department of Health Child and Adolescent Mental Health Division.
Cho, E., Wood, P. K., Hausman, E. M., Andrews, J. H., & Hawley, K. M. (2019). Evidence-based treatment strategies in youth mental health services: Results from a national survey of providers. Administration and Policy in Mental Health, 46, 71–81. https://doi.org/10.1007/s10488-018-0896-4.
Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children's mental health services. Journal of Consulting and Clinical Psychology, 79(4), 470–480. https://doi.org/10.1037/a0023982.
Chorpita, B. F., Bernstein, A., Daleiden, E. L., & The Research Network on Youth Mental Health. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 114–123. https://doi.org/10.1007/s10488-007-0151-x.
Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566–579. https://doi.org/10.1037/a0014565.
Chorpita, B. F., & Daleiden, E. L. (2018). Coordinated strategic action: Aspiring to wisdom in mental health service systems. Clinical Psychology: Science and Practice, 25(4), e12264. https://doi.org/10.1111/cpsp.12264.
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284.
Cook, J. M., Thompson, R., & Schnurr, P. P. (2015). Perceived Characteristics of Intervention Scale: Development and psychometric properties. Assessment, 22(6), 704–714. https://doi.org/10.1177/1073191114561254.
Decker, S. E., & Martino, S. (2013). Unintended effects of training on clinicians’ interest, confidence, and commitment in using motivational interviewing. Drug & Alcohol Dependence, 132(3), 681–687. https://doi.org/10.1016/j.drugalcdep.2013.04.022.
Dopp, A. R., Hanson, R. F., Saunders, B. E., Dismuke, C. E., & Moreland, A. D. (2017). Community-based implementation of trauma-focused interventions for youth: Economic impact of the learning collaborative model. Psychological Services, 14(1), 57–65. https://doi.org/10.1037/ser0000131.
Fung, J. J., & Lau, A. S. (2010). Factors associated with parent–child (dis)agreement on child behavior and parenting problems in Chinese immigrant families. Journal of Clinical Child & Adolescent Psychology, 39(3), 314–327. https://doi.org/10.1080/15374411003691693.
Garland, A. F., Bickman, L., & Chorpita, B. F. (2010a). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 37(1–2), 15–26. https://doi.org/10.1007/s10488-010-0279-y.
Garland, A. F., Brookman-Frazee, L., Hurlburt, M. S., Accurso, E. C., Zoffness, R. J., Haine-Schlagel, R., et al. (2010b). Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services, 61(8), 788–795. https://doi.org/10.1176/ps.2010.61.8.788.
Garland, A. F., Brookman-Frazee, L., & McLeod, B. D. (2008a). Scoring manual for the PRAC study therapy process observational coding system for child psychotherapy: Strategies scale. Department of Psychiatry: University of California, San Diego.
Garland, A. F., Hawley, K. M., Brookman-Frazee, L., & Hurlburt, M. S. (2008b). Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior problems. Journal of the American Academy of Child & Adolescent Psychiatry, 47(5), 505–514. https://doi.org/10.1097/CHI.0b013e31816765c2.
Glasgow, R. E., & Riley, W. T. (2013). Pragmatic measures: What they are and why we need them. American Journal of Preventive Medicine, 45(2), 237–243. https://doi.org/10.1016/j.amepre.2013.03.010.
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., Green, P., & The Research Network on Youth Mental Health. (2008). Assessing the Organizational Social Context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 98–113. https://doi.org/10.1007/s10488-007-0148-5.
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023
Herschell, A. D., Quetsch, L. B., & Kolko, D. J. (2019). Measuring adherence to key teaching techniques in an evidence-based treatment: A comparison of caregiver, therapist, and behavior observation ratings. Journal of Emotional and Behavioral Disorders. https://doi.org/10.1177/1063426618821901.
Hogue, A., Dauber, S., Henderson, C. E., & Liddle, H. A. (2014). Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration and Policy in Mental Health and Mental Health Services Research, 41(5), 697–705. https://doi.org/10.1007/s10488-013-0520-6.
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2.
Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 230–244. https://doi.org/10.1007/s10488-009-0251-x.
Kelley, S. D., & Bickman, L. (2009). Beyond outcomes monitoring: Measurement Feedback Systems (MFS) in child and adolescent clinical practice. Current Opinion in Psychiatry, 22(4), 363–368. https://doi.org/10.1097/YCO.0b013e32832c9162.
Kelley, S. D., de Andrade, A. R. V., Sheffer, E., & Bickman, L. (2010). Exploring the black box: Measuring youth treatment process and progress in usual care. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 287–300. https://doi.org/10.1007/s10488-010-0298-8.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012.
Lau, A. S., & Brookman-Frazee, L. (2016). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science, 11(1), 31–38. https://doi.org/10.1186/s13012-016-0388-4.
Lau, A. S., Garland, A. F., Yeh, M., Mccabe, K. M., Wood, P. A., & Hough, R. L. (2004). Race/ethnicity and inter-informant agreement in assessing adolescent psychopathology. Journal of Emotional and Behavioral Disorders, 12(3), 145–156. https://doi.org/10.1177/10634266040120030201.
Loades, M. E., & Myles, P. J. (2016). Does a therapist’s reflective ability predict the accuracy of their self-evaluation of competence in cognitive behavioural therapy? The Cognitive Behaviour Therapist, 9, 1–14. https://doi.org/10.1017/S1754470X16000027.
Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19(2), 181–193. https://doi.org/10.1080/10503300802688460.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30.
McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020.
Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology, 72(6), 1050–1062. https://doi.org/10.1037/0022-006X.72.6.1050.
Nakamura, B. J., Chorpita, B. F., Hirsch, M., Daleiden, E., Slavin, L., Amundson, M. J., et al. (2011). Large-scale implementation of evidence-based treatments for children 10 years later: Hawaii’s evidence-based services initiative in children’s mental health. Clinical Psychology: Science and Practice, 18(1), 24–35. https://doi.org/10.1111/j.1468-2850.2010.01231.x.
National Institute of Mental Health. (2017). RFA-MH-17–500: Pragmatic strategies for assessing psychotherapy quality in practice (R01). https://grants.nih.gov/grants/guide/rfa-files/rfa-mh-17-500.html
Orimoto, T. E., Higa-McMillan, C. K., Mueller, C. W., & Daleiden, E. L. (2012). Assessment of therapy practices in community treatment for children and adolescents. Psychiatric Services, 63(4), 343–350. https://doi.org/10.1176/appi.ps.201100129.
Regan, J. M. (2014). Client report of session content in an effectiveness trial: In search of efficient fidelity measurement [University of California, Los Angeles]. https://www.escholarship.org/uc/item/2np6k11r
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0.
Stanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., et al. (2019). Pragmatic measures for implementation research: Development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Translational Behavioral Medicine. https://doi.org/10.1093/tbm/ibz164.
Ward, A., Regan, J., Chorpita, B., Starace, N., Rodriguez, A., Okamura, K., et al. (2012). Tracking evidence-based practice with youth: Validity of the MATCH and standard manual consultation records. Journal of Clinical Child and Adolescent Psychology. https://doi.org/10.1080/15374416.2012.700505.
Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68(6), 1038–1050. https://doi.org/10.1037//0022-006X.68.6.1038.
Funding
This study was funded by the National Institute of Mental Health (R01MH100134).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
Dr. Ann Garland is a co-author on this manuscript and an Associate Editor of Administration and Policy in Mental Health Services Research. Dr. Garland will not be involved in the review of this submission. The authors declare that they have no other conflict of interest.
Ethical Approval
All procedures performed in this study that involved human participants were in accordance with the ethical standards of the Institutional Review Boards at the participating academic institutions and practice organizations, and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Brookman-Frazee, L., Stadnick, N.A., Lind, T. et al. Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children’s Mental Health Services. Adm Policy Ment Health 48, 155–170 (2021). https://doi.org/10.1007/s10488-020-01054-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-020-01054-x