Assessment of patient communication skills during OSCE: Examining effectiveness of a training program in minimizing inter-grader variability

https://doi.org/10.1016/j.pec.2011.04.001Get rights and content

Abstract

Objective

To assess effectiveness of a training program in reducing inter-grader variability in grading communication skills during an objective structured clinical exam (OSCE).

Methods

Global communication (GC) skills are assessed by standardized participants (SP) and faculty during each OSCE using a 6 item rubric. Despite criteria delineated in the GC rubric, inter-grader variability was observed. During 2008–2009 academic year, a training program was implemented before each OSCE to achieve more consistent interpretation and grading in GC skills. GC grades between SP and faculty for 2nd and 3rd level student OSCEs during 2008–2009 (post-training) were compared to 2007–2008 (pre-training). Data was analyzed using repeated measures ANOVA.

Results

274 and 281 students participated in OSCEs during 2007–2008 and 2008–2009 academic years, respectively. Training significantly (P < .001) decreased grader variability between SPs and faculty. There was a greater mean difference between faculty vs. SP before training (faculty 14.68, SP 15.87) compared to after training (faculty 13.51, SP 13.78). Mean GC scores for both faculty and SPs also decreased significantly after training.

Conclusion and practice implications

A training program may be necessary to reduce inter-rater variability in assessment of OSCE communication skills if it is to be truly helpful to student pharmacists learning to counsel patients.

Introduction

Effective communication is an important skill for all health care professionals. Pharmacists, in particular, must perform multiple communication tasks with patients and health care professionals as an integral part of everyday pharmacy practice. By law, pharmacists are required to counsel all patients on their new medications in the United States. Pharmacists conduct Medication Therapy Management sessions and communicate with other health care professionals to ensure optimal pharmaceutical care. Hence, pharmacists must possess strong interpersonal communication skills in order to develop an effective pharmacist–patient and pharmacist–other health professional relationship that is needed for high quality health care delivery [1], [2]. Effective communication in turn, is an important factor in minimizing medication related adverse events and improving the use of medications by patients to ensure optimal therapeutic outcomes [3], [4]. In 2007, the Accreditation Council for Pharmacy Education identified the need to teach communication skills in pharmacy schools by including standards pertaining to communication skills development (standards 13.3, 17.1, 17.3, 25.7, and 26.3) [5]. It has been reported that approximately 75% of the Colleges of Pharmacy in the United States include communication skills development in their curriculum [6].

Assessment of oral communication skills is difficult to achieve on a written examination. Therefore, objective structured clinical exam (OSCE) is becoming increasingly common as an assessment tool for oral communication skills among health professional students. OSCEs are defined by Harden, RM. as “an approach to the assessment of clinical competence in which the components of competence are assessed in a planned or structured way with the attention being paid to the objectivity of the examination” [8]. OSCEs are implemented in order to allow ‘students [to] demonstrate their competence under a variety of simulated conditions’ [9]. OSCEs are typically designed by having one or more timed stations, with assessment of either communication skills, clinical skills or both. OSCEs have been shown to be both valid and reliable in assessing communication and clinical skills in a variety of health professions [7], [10], [11], [12], [13], [14]. Programs that conduct OSCEs generally use global impression scales or specific skills check-off lists to assess a student's performance at a station [14]. A global impression scale may assess the general performance of the student during the OSCE or may focus on a specific area such as the impression of the student's communication skills. One type of global impression scale is a global communication scale which does not just focus on verbal communication skills, but also addresses other non-verbal aspects of communication such as body language, empathy, and organization of speech. A skills check-off list will often serve as a tool to evaluate clinical skills, for example a check-off list of the correct steps in measuring blood pressure. Most OSCEs are designed using a standardized participant (SP) as the simulator in an OSCE station [12], [15], [16], [17], [18], [19]. Standardized participants are individuals who are trained to respond uniformly to stimuli from the student. They are typically unknown to students and are trained to portray a consistent role with each student, so as to provide the same scenario under which students are graded.

OSCEs have become a preferred method over traditional written examinations to asses ‘communication and interpersonal skills, professional judgment and moral/ethical reasoning’ [20]. In Canada, OSCE is utilized as part of the pharmacy licensure and recertification process as an effective way to examine communication skills [7]. A recent study examining the use of OSCEs in US colleges and schools of pharmacy found that out of 88 programs sampled, 32 pharmacy programs reported using OSCEs and approximately half of those not using an OSCE expressed an interest in doing so [21]. Many schools will utilize an SP, a faculty member or both to assess a student's performance during an OSCE. Studies looking at SP vs. practitioner grader reliability have inferred a comparable level of rating with either grader [19]. However, given that assessments of communication and cognitive skills tend to be subjective and may be influenced by cultural factors and setting, there is a need to assess inter-grader variability at one's own institution [20], [22].

Western University of Health Sciences, College of Pharmacy has been utilizing OSCEs to assess communication and clinical skills of student pharmacists for the past 7 years. At our institution we utilize both an SP and a faculty to grade the communication skills aspect of the OSCE. Our steps for conducting an OSCE include the following:

  • 1.

    Recruitment of graders:

    Standardized participant and faculty graders are recruited for an OSCE case. Recruitment for SPs occurs through a centralized organization on campus that scouts and recruits individuals interested in being placed on the SP database. SPs are lay people who are typically professional actors/actresses. SPs are recruited for a case based on the age, gender, physical characteristics and ethnicity specific to the case. Faculty members are practicing clinical faculty with a Pharm.D. degree who are employed by the college of pharmacy or are a post-doctoral trainee (pharmacy residents and fellows) with a full-time faculty of the college. We have a cohort of 20 pharmacy practice faculty and approximately 5 pharmacy residents and fellows. All faculty, residents, and fellows grade about 2 OSCEs a year.

  • 2.

    OSCE case development:

    Cases for OSCEs are written by an OSCE team composed of 3 full-time Pharm.D. clinical faculty in the college. Cases are reviewed by a content expert, and are then pilot tested by students on experiential rotations, followed by a final revision. Students are selected who have had real life practice experience on the OSCE topic. Students are blinded to this process; the preceptor will ask the student to counsel on a certain drug for a given patient without disclosing this is a pilot test for an OSCE. During this process we identify additional clinical skills that should be added to the current list as well the ones that should be removed from the list. This process also identifies other unanticipated questions and dialogues to include in the case.

  • 3.

    Setting the # of OSCE stations:

    OSCEs range from 1 station (in the first year) to 3 stations (in years 2 and 3). Each station ranges in time from 7 to 12 min depending on the length and complexity of the case. Time is given in between each student to allow SPs to grade the communication skills.

  • 4.

    OSCE orientation session:

    Both faculty and SPs are required to attend orientation sessions the morning of each scheduled OSCE, regardless of past OSCE participation. The orientation consists of 2 parts; orientation to the grading of the communication skills and orientation to the case. Faculty are made familiar with the case and the clinical skills grading rubric for the case. SPs are made familiar with the case and the character they are expected to portray throughout the interaction.

    Grading of the communication skills utilizes a standardized Global Communication (GC) rubric which is used during all OSCEs. Students, faculty, and SPs are provided with a brief review of the GC rubric prior to each OSCE.

  • 5.

    Student flow:

    Students are expected to dress professionally with white lab coats. Students are brought up to the clinical skills lab facility in groups for orientation prior to entering the lab. To maintain integrity of the cases, students that have completed the OSCE are sequestered from the students waiting to complete the OSCE.

  • 6.

    Faculty and SP grading procedures:

    • a.

      Faculty are required to grade students on both clinical skills and communication skills. Clinical skills are graded using an analytical checklist specific to the case, and communication skills are graded using the GC rubric. Faculty are placed in either a control room with live-video feed to grade students in real-time as they perform, or in front of a 1-way viewing glass with headphones to watch and grade students as they perform.

    • b.

      SPs are required to grade only the communication skills using the same GC rubric. SPs are instructed not to grade during the student's performance as they must be engaged and portraying their character at all times. SPs are given approximately 5 min following each student's counseling session for grading student performance.

  • 7.

    Grading rubrics:

    As mentioned before, students are assessed during the OSCE on their clinical skills using a case specific analytical check-list and communication skills using the GC rubric. The GC rubric assesses the following 6 criteria: mechanics of verbal expression; content of verbal expression; non-verbal expression; interaction with the patient/health professional; organization/logic & coherence; professional appearance and rapport (see Table 1). The GC rubric is graded numerically on a 4-point Likert scale where 3 is excellent and 0 is failure for each of the 6 sections. This rubric is designed to provide an objective measurement to assess the communication skills of the pharmacy student.

A review of past OSCE scores at our institution showed significant (P < .001) inter-grader variability between the SP and the faculty in grading communication skills for the same student on the same OSCE. In attempting to examine reasons for the variability in grader scores, some possible hypotheses emerged. (1) There is subjectivity inherent in the assessment of communication skills [15], [21]. (2) Inherent to the faculty profile, a faculty member may be more familiar with using a grading rubric and assessing students. (3) SPs must rely on their recall of a student's performance since they must wait to grade a student until the end of the student's performance, which may not be as accurate as a faculty grading throughout the session. (4) There are varying interpretations of the GC rubric scoring criteria by the graders. Given there is evidence for reason 1, the 2nd reason is inherent in the faculty profile, and relying on recall to grade (#3) is a procedural issue that cannot be corrected, we sought to address #4 and decrease variability by implementing a training program to provide a uniform understanding of the GC rubric criteria.

The purpose of this study was to assess the effectiveness of the standardized training program in minimizing inter-grader variability while grading communication skills during an OSCE.

Section snippets

Methods

The purpose of this training was to achieve a more consistent interpretation of each of the 6 criteria on the GC rubric among graders. To address this issue a step-wise approach was used to build the training program:

  • 1.

    Training program overview – In addition to case orientation, a 45 min GC rubric training session was incorporated prior to each second and third year pharmacy OSCE. All graders, faculty and SPs, were required to attend prior to grading each OSCE, regardless of past orientation and

Results

Global communication scores from 12 SP graders and 12 faculty graders were collected during the 2007–2008 academic year (pre-training) and 2008–2009 academic year (post-training). These scores were based on assessment of a total number of 274 students pre-training and 281 students post-training from OSCE's for the 2nd year students (L2) and 3rd year students (L3). Scores obtained across the 2 years from faculty and SPs were collapsed so as to obtain average scores for faculty and for SP

Discussion

The results showed that the training program decreased the differences in grading between the faculty and SP graders. Faculty and SPs had a statistically significant difference in grading (−1.18, P < .001) at baseline, and a difference post-training (−0.27, P < .041). Although the latter difference is still significant the variability factor was lower. A possible reason for this continued difference post-training may be because we use different cases for each discrete OSCE. OSCE cases at our

Role of funding

There was no funding source for this project. The authors have nothing to disclose.

Conflict of interest

No conflict of interest relevant to this paper to disclose.

Acknowledgements

The author would like to acknowledge Olivia Phung and Juan Ramirez of Western University of Health Sciences, College of Pharmacy for their support with data analyses, statistics and proof-reading.

References (23)

  • R. Watson et al.

    Clinical competence assessment in nursing: a systematic review of the literature

    J Adv Nurs

    (2002)
  • Cited by (29)

    • Assessing communication skills in student pharmacists—Psychometric validation of Global Communication Rubric

      2021, Patient Education and Counseling
      Citation Excerpt :

      In order to assess responsiveness and predictive validity, the results of the communication skill scored by using the GC rubric was evaluated using Repeated Measures ANOVA. This study does not include the inter-rater reliability of the GC rubric, because this part of the validation step was previously published in the literature [15]. Communication skills of a total of 1,100 PharmD students enrolled in Class 2010 to 2020 were evaluated using the GC rubric during the five OSCE sessions.

    View all citing articles on Scopus
    View full text