Top

Gepubliceerd in:

16-11-2019 | Original Article

How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis

Auteurs: Franziska Kühne, Ramona Meister, Ulrike Maaß, Tatjana Paunov, Florian Weck

Gepubliceerd in: Cognitive Therapy and Research | Uitgave 2/2020

Abstract

Assessments of psychotherapeutic competencies play a crucial role in research and training. However, research on the reliability and validity of such assessments is sparse. This study aimed to provide an overview of the current evidence and to provide an average interrater reliability (IRR) of psychotherapeutic competence ratings. A systematic review was conducted, and 20 studies reported in 32 publications were collected. These 20 studies were included in a narrative synthesis, and 20 coefficients were entered into the meta-analysis. Most primary studies referred to cognitive-behavioral therapies and the treatment of depression, used the Cognitive Therapy Scale, based ratings on videos, and trained the raters. Our meta-analysis revealed a pooled ICC of 0.82, but at the same time severe heterogeneity. The evidence map highlighted a variety of variables related to competence assessments. Further aspects influencing the reliability of competence ratings and regarding the considerable heterogeneity are discussed in detail throughout the manuscript.

volgende artikel Identifying the Most Effective Strategies for Improving Autobiographical Memory Specificity and Its Implications for Mental Health Problems: A Meta-analysis

Alleen toegankelijk voor geautoriseerde gebruikers

American Psychiatric Association (APA). (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington VA: American Psychiatric Association.CrossRef

American Psychiatric Association (APA). (2017). What is psychotherapy? Retrieved from https://www.apa.org/ptsd-guideline/patients-and-families/psychotherapy.

* Barber, J. P., & Crits-Christoph, P. (1996). Development of a therapist adherence/competence rating scale for supportive-expressive dynamic psychotherapy: A preliminary report. Psychotherapy Research, 6(2), 81–94. https://doi.org/10.1080/10503309612331331608.

* Barber, J. P., Crits-Christoph, P., & Luborsky, L. (1996). Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology, 64(3), 619–622. https://doi.org/10.1037/0022-006x.64.3.619.

* Barber, J. P., Foltz, C., Crits-Christoph, P., & Chittams, J. (2004). Therapists’ adherence and competence and treatment discrimination in the NIDA Collaborative Cocaine Treatment Study. Journal of Clinical Psychology, 60(1), 29–41. https://doi.org/10.1002/jclp.10186

* Barber, J. P., Krakauer, I., Calvo, N., & Badgio, P. C. (1997). Measuring adherence and competence of dynamic therapists in the treatment of cocaine dependence. Journal of Psychotherapy Practice & Research, 6(1), 12–24.

* Barber, J. P., Liese, B. S., & Abrams, M. J. (2003). Development of the cognitive therapy adherence and competence scale. Psychotherapy Research, 13(2), 205–221.

Barber, J. P., Sharpless, B. A., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice,38, 493–500. https://doi.org/10.1037/0735-7028.38.5.493.CrossRef

Baujat, B., Mahé, C., Pignon, J. P., & Hill, C. (2002). A graphical method for exploring heterogeneity in meta-analyses: Application to a meta-analysis of 65 trials. Statistics in Medicine,21(18), 2641–2652.CrossRef

Beck Institute for Cognitive Behavior Therapy. (2019, October 10). Cognitive Therapy Rating Scale (CTRS). Retrieved from https://beckinstitute.org/wp-content/uploads/2017/03/CTRS-Scale-and-Score-Report-2016.pdf.

* Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): Psychometric properties. Behavioural and cognitive psychotherapy, 29(4), 431–446.

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken: Wiley.CrossRef

* Brueck, R. K., Frick, K., Loessl, B., Kriston, L., Schondelmaier, S., Go, C.,…, Berner, M. (2009). Psychometric properties of the German version of the motivational interviewing treatment integrity code. Journal of Substance Abuse Treatment, 36(1), 44–48. https://doi.org/10.1016/j.jsat.2008.04.004.

* Chevron, E. S., & Rounsaville, B. J. (1983). Evaluating the clinical skills of psychotherapists: A comparison of techniques. Archives of General Psychiatry, 40(10), 1129–1132. https://doi.org/10.1001/archpsyc.1983.01790090091014.

* Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012a). Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(6), 720–730. https://doi.org/10.1080/10503307.2012.716528.

* Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012b). How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(4), 475–488. https://doi.org/10.1080/10503307.2012.674790.

* Dittmann, C., Müller‐Engelmann, M., Stangier, U., Priebe, K., Fydrich, T., Görg, N., …, Steil, R. (2017). Disorder‐ and treatment‐specific therapeutic competence scales for posttraumatic stress disorder intervention: Development and psychometric properties. Journal of Traumatic Stress. https://doi.org/10.1002/jts.22236.

* Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24(4), 295–300. https://doi.org/10.1111/j.2044-8260.1985.tb00662.x.

Duffy, L., Gajree, S., Langhorne, P., Stott, D. J., & Quinn, T. J. (2013). Reliability (inter-rater agreement) of the Barthel Index for assessment of stroke survivors: Systematic review and meta-analysis. Stroke,44(2), 462–468. https://doi.org/10.1161/strokeaha.112.678615.CrossRefPubMed

Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal Open,315, 629–634.CrossRef

Fairburn, C. G., & Cooper, Z. (2011). Therapist competence, therapy quality, and therapist training. Behaviour Research and Therapy,49, 373–378. https://doi.org/10.1016/j.brat.2011.03.005.CrossRefPubMedPubMedCentral

Fisher, Z., & Tipton, E. (2015). robumeta: An R-package for robust variance estimation in meta-analysis. http://arxiv.org/abs/1503.02220.

Hempel, S., Miles, J. N., Booth, M. J., Wang, Z., Morton, S. C., & Shekelle, P. G. (2013). Risk of bias: A simulation study of power to detect study-level moderator effects in meta-analysis. Systematic Reviews,2, 107. https://doi.org/10.1186/2046-4053-2-107.CrossRefPubMedPubMedCentral

Higgins, J., & Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0 [updated March 2011]. The Cochrane Collaboration, 2011. Retrieved August, 29 from www.cochrane-handbook.org.

* Hoffart, A., Sexton, H., Nordahl, H. M., & Stiles, T. C. (2005). Connection between patient and therapist and therapist’s competence in schema-focused therapy of personality problems. Psychotherapy Research, 15(4), 409–419. https://doi.org/10.1080/10503300500091702.

* Karterud, S., Pedersen, G., Engen, M., Johansen, M. S., Johansson, P. N., Schlüter, C., …, Bateman, A. W. (2013). The MBT Adherence and Competence Scale (MBT-ACS): Development, structure and reliability. Psychotherapy Research, 23(6), 705–717. https://doi.org/10.1080/10503307.2012.708795.

Kazantzis, N. (2003). Therapist competence in cognitive-behavioural therapies: Review of the contemporary empirical evidence. Behaviour Change,20(1), 1–12. https://doi.org/10.1375/bech.20.1.1.24845.CrossRef

* Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42(4), 385–397. https://doi.org/10.1007/s10608-018-9919-4.

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine,15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012.CrossRefPubMedPubMedCentral

Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajeweski, B. J., Hróbjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International Journal of Nursing Studies,48, 661–671. https://doi.org/10.1016/j.ijnurstu.2011.01.016.CrossRefPubMed

Kuyken, W., & Tsivrikos, D. (2009). Therapist competence, comorbidity and cognitive-behavioral therapy for depression. Psychotherapy and Psychosomatics,78(1), 42–48. https://doi.org/10.1159/000172619.CrossRefPubMed

Machmutow, K., Holtforth, M. G., Krieger, T., & Watzke, B. (2018). Identifying relapse prevention elements during psychological treatment of depression: Development of an observer-based rating instrument. Journal of Affective Disorders,227, 358–365. https://doi.org/10.1016/j.jad.2017.11.009.CrossRefPubMed

* McGrath, K. B. (2013). Validation of the Drexel University ACT/tCBT Adherence and Competence Rating Scale: Revised for use in a clinical population. Doctoral dissertation. Retrieved from https://idea.library.drexel.edu/islandora/object/idea%3A3803.

Meister, R., Jansen, A., Härter, M., Nestoriuc, Y., & Kriston, L. (2017). Placebo and nocebo reactions in randomized trials of pharmacological treatments for persistent depressive disorder. A meta-regression analysis. Journal of Affective Disorders,215, 288–298.CrossRef

Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med,6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097.CrossRefPubMedPubMedCentral

Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review,33(3), 484–499. https://doi.org/10.1016/j.cpr.2013.01.010.CrossRefPubMed

Muse, K., & McManus, F. (2016). Expert insight into the assessment of competence in cognitive-behavioural therapy: A qualitative exploration of experts’ experiences, opinions and recommendations. Clinical Psychology and Psychotherapy,23, 246–259. https://doi.org/10.1002/cpp.1952.CrossRefPubMed

Muse, K., McManus, F., Rakovshik, S., & Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): An observation-based tool for assessing cognitive behavioral therapy competence. Psychological Assessment,29(5), 542–555. https://doi.org/10.1037/pas0000372.CrossRefPubMed

Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Applications to practice. Upper Saddle River, NJ: Pearson/Prentice Hall.

Quintana, D. S. (2015). From pre-registration to publication: A non-technical primer for conducting a meta-analysis to synthesize correlational data. Frontiers in Psychology,6, 1549. https://doi.org/10.3389/fpsyg.2015.01549.CrossRefPubMedPubMedCentral

R-Core-Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Retrieved from https://www.R-project.org/.

* Reichelt, F. K., James, I. A., & Blackburn, I. M. (2003). Impact of training on rating competence in cognitive therapy. Journal of Behavior Therapy and Experimental Psychiatry, 34(2), 87–99.

Roth, A. D., & Pilling, S. (2007). The competences required to deliver effective cognitive and behavioural therapy for people with depression and with anxiety disorders. Retrieved from https://www.ucl.ac.uk/pals/research/clinical-ed.

Santelmann, H., Franklin, J., Bußhoff, J., & Baethge, C. (2016). Interrater reliability of schizoaffective disorder compared with schizophrenia, bipolar disorder, and unipolar depression—A systematic review and meta-analysis. Schizophrenia Research,176(2), 357–363. https://doi.org/10.1016/j.schres.2016.07.012.CrossRefPubMed

* Schmidt, I. D., Strunk, D. R., DeRubeis, R. J., Conklin, L. R., & Braun, J. D. (2018). Revisiting how we assess therapist competence in cognitive therapy. Cognitive Therapy and Research, 42(4), 369–384. https://doi.org/10.1007/s10608-018-9908-7.

* Strunk, D. R., Brotman, M. A., DeRubeis, R. J., & Hollon, S. D. (2010). Therapist competence in cognitive therapy for depression: Predicting subsequent symptom change. J Consult Clin Psychol, 78(3), 429–437. https://doi.org/10.1037/a0019631.

* Soygüt, G., Uluç, S., & Tüzün, Z. (2008). A pilot study of the reliability and validity of the turkish cognitive therapy adherence and competence scale. Turkish Journal of Psychiatry, 19(2).

* Svartberg, M. (1989). Manualization and competence monitoring of short-term anxiety-provoking psychotherapy. Psychotherapy: Theory, Research, Practice, Training, 26(4), 564–571. https://doi.org/10.1037/h0085477.

* Tadic, M., Drapeau, M., Solai, S., de Roten, Y., & Despland, J. N. (2003). Development of a competence scale for brief psychodynamic investigation: A pilot study. Schweizer Archiv für Neurologie und Psychiatrie, 154(1), 28–35. https://doi.org/10.4414/sanp.2003.01333.

Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., et al. (2011). Reliability of the Hamilton Rating Scale for depression: A meta-analysis over a period of 49years. Psychiatry Research,189(1), 1–9. https://doi.org/10.1016/j.psychres.2010.12.007.CrossRefPubMed

* Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54(3), 381–385. https://doi.org/10.1037/0022-006x.54.3.381.

* Vallis, T. M., Shaw, B. F., & McCabe, S. B. (1988). The relationship between therapist competency and cognitive therapy and general therapy skill. Journal of Cognitive Psychotherapy: An International Quarterly, 2(4), 237–249.

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software,36, 1–48.CrossRef

* von Consbruch, K., Clark, D. M., & Stangier, U. (2012). Assessing therapeutic competence in cognitive therapy for social phobia: Psychometric properties of the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP). Behavioural and Cognitive Psychotherapy, 40(2), 149–161. https://doi.org/10.1017/s1352465811000622.

Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology,61(4), 620–630.CrossRef

Warshaw, M. G., Dyck, I., Allsworth, J., Stout, R. L., & Keller, M. B. (2001). Maintaining reliability in a long-term psychiatric study: An ongoing inter-rater reliability monitoring program using the longitudinal interval follow-up evaluation. Journal of Psychiatric Research,35(5), 297–305.CrossRef

Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology,78, 200–211. https://doi.org/10.1037/a0018912.CrossRefPubMedPubMedCentral

* Weck, F., Bohn, C., Ginzburg, D. M., & Stangier, U. (2011). Assessment of adherence and competence in cognitive therapy: Comparing session segments with entire sessions. Psychotherapy Research, 21(6), 658–669. https://doi.org/10.1080/10503307.2011.602751.

* Weck, F., Grikscheit, F., Höfling, V., & Stangier, U. (2014). Assessing treatment integrity in cognitive-behavioral therapy: Comparing session segments with entire sessions. Behavior Therapy, 45(4), 541–552. https://doi.org/10.1016/j.beth.2014.03.003.

* Weck, F., Hautzinger, M., Heidenreich, T., & Stangier, U. (2011). Psychoedukation bei depressiven störungen -erfassung von interventionsmerkmalen und behandlungskompetenzen = Psychoeducation for depression - features of interventions and therapeutic competencies. PPmP: Psychotherapie Psychosomatik Medizinische Psychologie, 61(3-4), 148–153. https://doi.org/10.1055/s-0030-1269902.

* Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(4), 276–279. https://doi.org/10.1097/nmd.0b013e318212461.

* Weck, F., Weigel, M., Richtberg, S., & Stangier, U. (2011). Reliability of adherence and competence assessment in psychoeducational treatment: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(12), 983–986. https://doi.org/10.1097/nmd.0b013e3182392da1.

WHO. (1992). The ICD-10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines (6th ed.). Geneva: World Health Organization.

Wickham, H., Francois, R., Henry, L., & Müller, K. (2015). dplyr: A grammar of data manipulation. R package version 0.4, 3.

Wirtz, M. A. (2017). Interrater Reliability. In V. Zeigler-Hill & T. K. Shackelford (Eds.), Encyclopedia of personality and individual differences (pp. 1–4). New York: Springer.

Wirtz, M., & Caspar, F. (2002). Interrater agreement and interrater reliability. Göttingen: Hogrefe.

* Wittorf, A., Jakobi-Malterre, U. E., Beulen, S., Bechdolf, A., Müller, B. W., Sartory, G., …, Klingberg, S. (2013). Associations between therapy skills and patient experiences of change processes in cognitive behavioral therapy for psychosis. Psychiatry Research, 210(3), 702–709. https://doi.org/10.1016/j.psychres.2013.08.01.

Young, J. E., & Beck, A. T. (1980). Cognitive Therapy Scale: Rating manual. Unpublished Manuscript, University of Pennsylvania, Philadelphia, PA.

Zarafonitis-Müller, S., Kuhr, K., & Bechdorf, A. (2014). Der Zusammenhang der Therapeutenkompetenz und Adhärenz zum Therapieerfolg in der Kognitiven Verhaltenstherapie - metaanalytische Ergebnisse. Fortschritte der Neurologie-Psychiatrie,82, 502–510.CrossRef

Titel: How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis
Auteurs: Franziska Kühne
Ramona Meister
Ulrike Maaß
Tatjana Paunov
Florian Weck
Publicatiedatum: 16-11-2019
Uitgeverij: Springer US
Gepubliceerd in: Cognitive Therapy and Research / Uitgave 2/2020
Print ISSN: 0147-5916
Elektronisch ISSN: 1573-2819
DOI: https://doi.org/10.1007/s10608-019-10056-5

Andere artikelen Uitgave 2/2020

Specificity of Disgust in the Prediction of Behavioral Avoidance of Possible Contaminants

Original Article

The Impact of Negative Emotions on Drinking Among Individuals with Social Anxiety Disorder in Daily Life: The Moderating Effect of Maladaptive Emotion Regulation Strategies

Original Article

Emotional Nonacceptance, Distraction Coping and PTSD Symptoms in a Trauma-Exposed Adolescent Inpatient Sample

Original Article

Contextual Changes Influence Attention Flexibility Towards New Goals

Original Article

Beliefs About Emotions, Negative Meta-emotions, and Perceived Emotional Control During an Emotionally Salient Situation in Individuals with Emotional Disorders

Original Article

Experiential Avoidance as a Mechanism of Change Across Cognitive-Behavioral Therapy in a Sample of Participants with Heterogeneous Anxiety Disorders

Original Article