A systematic review of methods for assessing competence in cognitive–behavioural therapy☆
Highlights
► Systematic review of methods for assessing CBT therapist competence. ► Advantages and disadvantages of ten assessment methods are examined in relation to reliability, validity and feasibility. ► Recommendations for assessing CBT therapist competence are made. ► Priorities for future research are highlighted.
Introduction
Significant progress has been made in developing evidence-based psychological treatments for a variety of disorders and problems. Prominent amongst these treatments is Cognitive Behaviour Therapy (CBT), which has been shown to be effective in treating a wide range of psychological disorders (Butler et al., 2006, Stewart and Chambless, 2009). Identifying an optimal strategy for assessing the competence with which CBT is delivered is important to the continued progression of the field for a number of reasons. First, the strong evidence base for CBT and the high economic burden of untreated mental illness has prompted increased demand for the dissemination of CBT (Barlow et al., 1999, Clark, 2011, McHugh and Barlow, 2010). For example, the UK has seen large-scale government investment in the Increasing Access to Psychological Therapies dissemination program (Clark et al., 2009, Department of Health, 2008) and a number of other European countries are considering similar initiatives (Berge, 2011). Similarly, in the USA the Veterans Health Administration recently embarked on a significant program promoting the implementation of evidence-based psychological treatments, such as CBT (McHugh & Barlow, 2010). Effective methods of assessing CBT competence are essential to the success of such dissemination programs as they provide a means of assessing the training of new CBT therapists and ensuring the quality of treatment provision within routine clinical practice (McHugh and Barlow, 2010, Rakovshik and McManus, 2010, Schoenwald et al., 2011).
Second, competence assessment plays a crucial role in the empirical evaluation of CBT as research trials cannot draw valid conclusions regarding the efficacy of CBT protocols unless the competence with which the protocols are delivered can be established (Waltz et al., 1993, Weck et al., 2011b). Third, as evidence suggests that therapist competence may play a role in determining treatment outcomes, at least in the context of depression (Kuyken and Tsivrikos, 2009, Shaw et al., 1999, Strunk et al., 2010, Trepka et al., 2004), assessment of CBT competence could provide a vehicle for ensuring that CBT is optimally effective for patients. However, results from studies examining the relationships between CBT competence and patient outcome are variable, and the poor reliability of existing methods for assessing CBT competence has been suggested as one possible explanation for this (Crits-Christoph et al., 1991, Perepletchikova and Kazdin, 2005, Webb et al., 2010). Hence, improved understanding of the assessment of CBT competence may facilitate much needed future research examining the association between competence and outcome in CBT and has the potential to provide insight into the ‘active ingredients’ responsible for the relationship (Dobson & Singer, 2005). Finally, effective measurement of CBT competence is necessary to provide targeted feedback regarding therapists' strengths and weaknesses (McManus, Rosen, & Jenkins, 2010) and to enable research examining the acquisition of CBT skills, thus informing the training of therapists.
Despite the importance of effective measurement of CBT competence, a lack of consensus regarding the way in which CBT competence should be assessed has resulted in the development of multiple different assessment methods, many of which have been widely criticised (Barber et al., 2007, McGlinchey and Dobson, 2003, Sharpless and Barber, 2009, Waltz et al., 1993). Hence this review outlines and evaluates strengths and weaknesses of existing methods for assessing CBT competence in order to make recommendations about the most effective methods and identify priorities for future research into the development of reliable, valid and cost-effective methods of assessing CBT competence.
Section snippets
What is CBT competence?
Before examining the utility of different methods for assessing therapist1 competence in CBT it is necessary to clarify what is meant by the term competence. In line with Barber et al.'s (2007) concept of ‘limited-domain intervention competence’ and Kaslow's (2004) notion of
Search strategy and procedures
Articles included in the review met the following criteria: (i) English language publication pertaining to a quantifiable CBT competence measure, (ii) relate to individual face-to-face CBT (i.e., not group, family, online CBT) for an adult population experiencing mental health problems, and (iii) not a review or commentary. Peer-reviewed articles published2
Results
The 64 articles included in the review identified ten methods of CBT competence assessment, which are presented below within Miller's (1990) hierarchical framework for assessing clinical skill (see Fig. 2). While it is recognised that there is overlap between categories, Miller's (1990) framework separates skills assessments into four hierarchical levels: knowledge (knows); practical understanding (knows how); skills (shows how), and clinical practice (does). Each assessment method is described
Level 1: knowledge-based assessments
The basic foundation of CBT competence is a sound understanding of the scientific, theoretical and contextual basis of CBT (Roth & Pilling, 2007). CBT knowledge can be assessed using multiple choice questionnaires (MCQs) or essays (for a summary of studies employing knowledge assessments see online Supplemental materials Appendix 2). As well as assessing CBT knowledge, MCQs and essays can be used to assess knowledge of the clinical application of CBT and thus can also be assessments of
Level 2: assessments of practical understanding
CBT competence involves not only understanding of CBT theory and research but also the ability to use this knowledge to inform the implementation of CBT interventions (Miller, 1990, Roth and Pilling, 2007). Essays and MCQs which are grounded in clinically relevant contextual information can be used to assess therapists' understanding of the clinical application of CBT. Practical understanding can also be assessed using short-answer clinical vignettes and case reports (for a summary of studies
Level 3: Assessments of practical application of knowledge (skill)
Understanding of CBT and its clinical application is necessary but not sufficient in demonstrating competence—it is also essential that a therapist has the skills to apply this knowledge in clinical situations (Roth & Pilling, 2007). Furthermore Miller (1990) distinguishes between skill-based assessments which occur within carefully constructed clinical simulations (‘shows how’) and those which occur within real clinical practice settings (‘does’ independently in practice). First we review
Level 4: Clinical practice assessments
The highest level of competence assessment in Miller's (1990) hierarchy is that of ‘does’—being able to use independent judgment and critical thinking to appropriately and effectively deliver CBT interventions within the cultural and organisational context of clinical practice settings (Miller, 1990, Roth and Pilling, 2007). Methods of assessing competence within clinical practice are reviewed below and include: (1) assessor-ratings of treatment sessions, (2) supervisory assessments, (3)
Summary of assessment methods
This review identified ten key methods for assessing CBT competence which were presented within Miller's (1990) framework for assessing clinical skill as those which assess therapists' knowledge (‘knows’), practical understanding (‘knows how’), practical application of knowledge–skill (‘shows how’), and clinical practice (‘does’). In principle, knowledge of CBT is the easiest domain to operationalise and assess. However, whilst MCQ's are a widely used and feasible means of assessing basic CBT
References5 (126)
- et al.
Residential cognitive therapy versus residential interpersonal therapy for social phobia: A randomized clinical trial
Journal of Anxiety Disorders
(2008) - et al.
Temporal relationship between dysfunctional beliefs, self-efficacy and panic apprehension in the treatment of panic disorder with agoraphobia
Journal of Behavior Therapy and Experimental Psychiatry
(2007) - et al.
The empirical status of cognitive–behavioural therapy: A review of meta-analyses
Clinical Psychology Review
(2006) - et al.
A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions
Drug and Alcohol Dependence
(2000) - et al.
Improving access to psychological therapy: Initial evaluation of two UK demonstration sites
Behaviour Research and Therapy
(2009) - et al.
Therapist competence, therapy quality, and therapist training
Behaviour Research and Therapy
(2011) - et al.
Cognitive behavioral therapy in practice: Treatment delivered by trainees at an outpatient clinic is clinically effective
Behavior Therapy
(2011) - et al.
Change processes in residential cognitive and interpersonal psychotherapy for social phobia: A process-outcome study
Behavior Therapy
(2009) - et al.
Can depression be de-medicalized in the 21st century: Scientific revolutions, counter-revolutions and the magnetic field of normal science
Behaviour Research and Therapy
(2000) - et al.
An evaluation of the effectiveness of diploma-level training in cognitive behaviour therapy
Behaviour Research and Therapy
(2010)
Effectiveness of cognitive therapy training
Journal of Behavior Therapy and Experimental Psychiatry
Treatment fidelity in outcome studies
Clinical Psychology Review
Establishing evidence-based training in cognitive behavioral therapy: A review of current empirical findings and theoretical guidance
Clinical Psychology Review
Impact of training on rating competence in cognitive therapy
Journal of Behavior Therapy and Experimental Psychiatry
The effects of therapist competence in assigning homework in cognitive therapy with cluster C personality disorders: Results from a randomized controlled trial
Cognitive and Behavioral Practice
Psychological treatment of panic disorder with or without agoraphobia: A meta-analysis
Clinical Psychology Review
A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology
Clinical Psychology Review
Therapists' adherence and competence and treatment discrimination in the NIDA collaborative cocaine treatment study
Journal of Clinical Psychology
Development of the cognitive therapy adherence and competence scale
Psychotherapy Research
Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature
Professional Psychology: Research and Practice
The dissemination of empirically supported treatments: A view to the future
Behaviour Research and Therapy
Assessing the development of competence during postgraduate cognitive–behavioral therapy training
Journal of Cognitive Psychotherapy
Cognitive therapy of depression
The ups and downs of cognitive therapy training: What happens to trainees' perception of their competence during a cognitive therapy training course?
Behavioural and Cognitive Psychotherapy
The revised cognitive therapy scale (CTS-R): Psychometric properties
Behavioural and Cognitive Psychotherapy
Factors associated with competence in cognitive therapists
Behavioural and Cognitive Psychotherapy
Self-evaluation of cognitive therapy performance: Do therapists know how competent they are?
Behavioural and Cognitive Psychotherapy
Therapist skill and patient variables in homework compliance: Controlling an uncontrolled variable in cognitive therapy outcome research
Cognitive Therapy and Research
A cognitive–behavioral approach: Treating cocaine addiction
Constructing written test questions for the basic and clinical sciences
Systematic reviews: CRD's guidance for undertaking reviews in health care
Evaluating the clinical skills of psychotherapists: A comparison of techniques
Archives of General Psychiatry
Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: The IAPT experience
International Review of Psychiatry
The use of computers in assessment
Meta-analysis of therapist effects in psychotherapy outcome studies
Psychotherapy Research
Training in cognitive, supportive–expressive, and drug counseling therapies for cocaine dependence
Journal of Consulting and Clinical Psychology
Therapist competence and clinical outcome in the Prevention of Parasuicide by Manual Assisted Cognitive Behaviour Therapy Trial: The POPMACT study
Psychological Medicine: A Journal of Research in Psychiatry and the Allied Sciences
Cognitive behaviour therapy for violent men with antisocial personality disorder in the community: An exploratory randomized controlled trial
Psychological Medicine
Therapist training in empirically supported treatments: A review of evaluation methods for short- and long-term outcomes
Administration and Policy in Mental Health and Mental Health Services Research
Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence?
Psychotherapy Research
How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence?
Psychotherapy Research
IAPT implementation plan: National guidelines for regional delivery
National curriculum for high intensity cognitive behavioural therapy courses
Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression
Journal of Consulting and Clinical Psychology
Reliability of a measure of the quality of cognitive therapy
The British Journal of Clinical Psychology
Definitional and practical issues in the assessment of treatment integrity
Clinical Psychology: Science and Practice
Tayside–Fife clinical trial of cognitive–behavioural therapy for medication-resistant psychotic symptoms
The British Journal of Psychiatry
Assessment in medical education
The New England Journal of Medicine
Adherence and competence assessment in studies of CBT for psychosis: Current status and future directions
Epidemiology and Psychiatric Sciences
Cited by (137)
Scaling up quality-assured psychotherapy: The role of therapist competence on perinatal depression and anxiety outcomes
2023, General Hospital PsychiatrySystematic review of training in cognitive-behavioral therapy: Summarizing effects, costs and techniques
2023, Clinical Psychology ReviewHigher-order CBT skills: are there differences in meta-competence between trainee and experienced therapists?
2024, Cognitive Behaviour TherapistDevelopment of competence in cognitive behavioural therapy and the role of metacognition among clinical psychology and psychotherapy students
2023, Behavioural and Cognitive PsychotherapyDevelopment and initial evaluation of a treatment integrity measure for low-intensity group psychoeducational interventions
2023, Behavioural and Cognitive Psychotherapy
- ☆
This research was supported by a grant from theBritish Association for Behavioural and Cognitive Psychotherapies (BABCP) awarded to Kate Muse and Freda McManus.
- 5
References marked with an *asterisk indicate studies included in the review.