Assessment of patient communication skills during OSCE: Examining effectiveness of a training program in minimizing inter-grader variability
Introduction
Effective communication is an important skill for all health care professionals. Pharmacists, in particular, must perform multiple communication tasks with patients and health care professionals as an integral part of everyday pharmacy practice. By law, pharmacists are required to counsel all patients on their new medications in the United States. Pharmacists conduct Medication Therapy Management sessions and communicate with other health care professionals to ensure optimal pharmaceutical care. Hence, pharmacists must possess strong interpersonal communication skills in order to develop an effective pharmacist–patient and pharmacist–other health professional relationship that is needed for high quality health care delivery [1], [2]. Effective communication in turn, is an important factor in minimizing medication related adverse events and improving the use of medications by patients to ensure optimal therapeutic outcomes [3], [4]. In 2007, the Accreditation Council for Pharmacy Education identified the need to teach communication skills in pharmacy schools by including standards pertaining to communication skills development (standards 13.3, 17.1, 17.3, 25.7, and 26.3) [5]. It has been reported that approximately 75% of the Colleges of Pharmacy in the United States include communication skills development in their curriculum [6].
Assessment of oral communication skills is difficult to achieve on a written examination. Therefore, objective structured clinical exam (OSCE) is becoming increasingly common as an assessment tool for oral communication skills among health professional students. OSCEs are defined by Harden, RM. as “an approach to the assessment of clinical competence in which the components of competence are assessed in a planned or structured way with the attention being paid to the objectivity of the examination” [8]. OSCEs are implemented in order to allow ‘students [to] demonstrate their competence under a variety of simulated conditions’ [9]. OSCEs are typically designed by having one or more timed stations, with assessment of either communication skills, clinical skills or both. OSCEs have been shown to be both valid and reliable in assessing communication and clinical skills in a variety of health professions [7], [10], [11], [12], [13], [14]. Programs that conduct OSCEs generally use global impression scales or specific skills check-off lists to assess a student's performance at a station [14]. A global impression scale may assess the general performance of the student during the OSCE or may focus on a specific area such as the impression of the student's communication skills. One type of global impression scale is a global communication scale which does not just focus on verbal communication skills, but also addresses other non-verbal aspects of communication such as body language, empathy, and organization of speech. A skills check-off list will often serve as a tool to evaluate clinical skills, for example a check-off list of the correct steps in measuring blood pressure. Most OSCEs are designed using a standardized participant (SP) as the simulator in an OSCE station [12], [15], [16], [17], [18], [19]. Standardized participants are individuals who are trained to respond uniformly to stimuli from the student. They are typically unknown to students and are trained to portray a consistent role with each student, so as to provide the same scenario under which students are graded.
OSCEs have become a preferred method over traditional written examinations to asses ‘communication and interpersonal skills, professional judgment and moral/ethical reasoning’ [20]. In Canada, OSCE is utilized as part of the pharmacy licensure and recertification process as an effective way to examine communication skills [7]. A recent study examining the use of OSCEs in US colleges and schools of pharmacy found that out of 88 programs sampled, 32 pharmacy programs reported using OSCEs and approximately half of those not using an OSCE expressed an interest in doing so [21]. Many schools will utilize an SP, a faculty member or both to assess a student's performance during an OSCE. Studies looking at SP vs. practitioner grader reliability have inferred a comparable level of rating with either grader [19]. However, given that assessments of communication and cognitive skills tend to be subjective and may be influenced by cultural factors and setting, there is a need to assess inter-grader variability at one's own institution [20], [22].
Western University of Health Sciences, College of Pharmacy has been utilizing OSCEs to assess communication and clinical skills of student pharmacists for the past 7 years. At our institution we utilize both an SP and a faculty to grade the communication skills aspect of the OSCE. Our steps for conducting an OSCE include the following:
- 1.
Recruitment of graders:
Standardized participant and faculty graders are recruited for an OSCE case. Recruitment for SPs occurs through a centralized organization on campus that scouts and recruits individuals interested in being placed on the SP database. SPs are lay people who are typically professional actors/actresses. SPs are recruited for a case based on the age, gender, physical characteristics and ethnicity specific to the case. Faculty members are practicing clinical faculty with a Pharm.D. degree who are employed by the college of pharmacy or are a post-doctoral trainee (pharmacy residents and fellows) with a full-time faculty of the college. We have a cohort of 20 pharmacy practice faculty and approximately 5 pharmacy residents and fellows. All faculty, residents, and fellows grade about 2 OSCEs a year.
- 2.
OSCE case development:
Cases for OSCEs are written by an OSCE team composed of 3 full-time Pharm.D. clinical faculty in the college. Cases are reviewed by a content expert, and are then pilot tested by students on experiential rotations, followed by a final revision. Students are selected who have had real life practice experience on the OSCE topic. Students are blinded to this process; the preceptor will ask the student to counsel on a certain drug for a given patient without disclosing this is a pilot test for an OSCE. During this process we identify additional clinical skills that should be added to the current list as well the ones that should be removed from the list. This process also identifies other unanticipated questions and dialogues to include in the case.
- 3.
Setting the # of OSCE stations:
OSCEs range from 1 station (in the first year) to 3 stations (in years 2 and 3). Each station ranges in time from 7 to 12 min depending on the length and complexity of the case. Time is given in between each student to allow SPs to grade the communication skills.
- 4.
OSCE orientation session:
Both faculty and SPs are required to attend orientation sessions the morning of each scheduled OSCE, regardless of past OSCE participation. The orientation consists of 2 parts; orientation to the grading of the communication skills and orientation to the case. Faculty are made familiar with the case and the clinical skills grading rubric for the case. SPs are made familiar with the case and the character they are expected to portray throughout the interaction.
Grading of the communication skills utilizes a standardized Global Communication (GC) rubric which is used during all OSCEs. Students, faculty, and SPs are provided with a brief review of the GC rubric prior to each OSCE.
- 5.
Student flow:
Students are expected to dress professionally with white lab coats. Students are brought up to the clinical skills lab facility in groups for orientation prior to entering the lab. To maintain integrity of the cases, students that have completed the OSCE are sequestered from the students waiting to complete the OSCE.
- 6.
Faculty and SP grading procedures:
- a.
Faculty are required to grade students on both clinical skills and communication skills. Clinical skills are graded using an analytical checklist specific to the case, and communication skills are graded using the GC rubric. Faculty are placed in either a control room with live-video feed to grade students in real-time as they perform, or in front of a 1-way viewing glass with headphones to watch and grade students as they perform.
- b.
SPs are required to grade only the communication skills using the same GC rubric. SPs are instructed not to grade during the student's performance as they must be engaged and portraying their character at all times. SPs are given approximately 5 min following each student's counseling session for grading student performance.
- a.
- 7.
Grading rubrics:
As mentioned before, students are assessed during the OSCE on their clinical skills using a case specific analytical check-list and communication skills using the GC rubric. The GC rubric assesses the following 6 criteria: mechanics of verbal expression; content of verbal expression; non-verbal expression; interaction with the patient/health professional; organization/logic & coherence; professional appearance and rapport (see Table 1). The GC rubric is graded numerically on a 4-point Likert scale where 3 is excellent and 0 is failure for each of the 6 sections. This rubric is designed to provide an objective measurement to assess the communication skills of the pharmacy student.
A review of past OSCE scores at our institution showed significant (P < .001) inter-grader variability between the SP and the faculty in grading communication skills for the same student on the same OSCE. In attempting to examine reasons for the variability in grader scores, some possible hypotheses emerged. (1) There is subjectivity inherent in the assessment of communication skills [15], [21]. (2) Inherent to the faculty profile, a faculty member may be more familiar with using a grading rubric and assessing students. (3) SPs must rely on their recall of a student's performance since they must wait to grade a student until the end of the student's performance, which may not be as accurate as a faculty grading throughout the session. (4) There are varying interpretations of the GC rubric scoring criteria by the graders. Given there is evidence for reason 1, the 2nd reason is inherent in the faculty profile, and relying on recall to grade (#3) is a procedural issue that cannot be corrected, we sought to address #4 and decrease variability by implementing a training program to provide a uniform understanding of the GC rubric criteria.
The purpose of this study was to assess the effectiveness of the standardized training program in minimizing inter-grader variability while grading communication skills during an OSCE.
Section snippets
Methods
The purpose of this training was to achieve a more consistent interpretation of each of the 6 criteria on the GC rubric among graders. To address this issue a step-wise approach was used to build the training program:
- 1.
Training program overview – In addition to case orientation, a 45 min GC rubric training session was incorporated prior to each second and third year pharmacy OSCE. All graders, faculty and SPs, were required to attend prior to grading each OSCE, regardless of past orientation and
Results
Global communication scores from 12 SP graders and 12 faculty graders were collected during the 2007–2008 academic year (pre-training) and 2008–2009 academic year (post-training). These scores were based on assessment of a total number of 274 students pre-training and 281 students post-training from OSCE's for the 2nd year students (L2) and 3rd year students (L3). Scores obtained across the 2 years from faculty and SPs were collapsed so as to obtain average scores for faculty and for SP
Discussion
The results showed that the training program decreased the differences in grading between the faculty and SP graders. Faculty and SPs had a statistically significant difference in grading (−1.18, P < .001) at baseline, and a difference post-training (−0.27, P < .041). Although the latter difference is still significant the variability factor was lower. A possible reason for this continued difference post-training may be because we use different cases for each discrete OSCE. OSCE cases at our
Role of funding
There was no funding source for this project. The authors have nothing to disclose.
Conflict of interest
No conflict of interest relevant to this paper to disclose.
Acknowledgements
The author would like to acknowledge Olivia Phung and Juan Ramirez of Western University of Health Sciences, College of Pharmacy for their support with data analyses, statistics and proof-reading.
References (23)
- et al.
Conceptualizing and measuring pharmacist–patient communication: a review of published studies
Res Social Adm Pharm
(2006) - et al.
Counseling reduces dispensing accidents
Am Pharm
(1992) - et al.
The use of objective structured clinical examination (OSCE) for evaluation and instruction in graduate medical education
J Surg Res
(1996) Objective structured clinical examination (OSCE): review of literature and implications for nursing education
Nurse Educ Today
(2007)- Berger BA. Communication Skills for pharmacists: building relationships, improving patient care. 3rd ed. Washington,...
Research on the effects of pharmacist–patient communication in institutions and ambulatory care sites, 1969–1994
Am J Health Syst Pharm
(1996)- Acreditation Counsel for Pharmacy Education. Accreditation standards and guidelines [cited 2010 November 1]. Available...
Communication skills development in colleges of pharmacy
Am J Pharm Educ
(2001)- et al.
Development and validation processes for an objective structured clinical examination (OSCE) for entry-to-practice certification in pharmacy: the Canadian experience
Am J Pharm Educ
(2003) What is an OSCE?
Med Teach
(1988)
Clinical competence assessment in nursing: a systematic review of the literature
J Adv Nurs
Cited by (29)
What's been trending with OSCEs in pharmacy education over the last 20 years? A bibliometric review and content analysis
2024, Currents in Pharmacy Teaching and LearningDetermination of Interrater Reliability of a Universal Evaluator Rubric to Assess Student Pharmacist Communication Skills
2022, American Journal of Pharmaceutical EducationAssessing communication skills in student pharmacists—Psychometric validation of Global Communication Rubric
2021, Patient Education and CounselingCitation Excerpt :In order to assess responsiveness and predictive validity, the results of the communication skill scored by using the GC rubric was evaluated using Repeated Measures ANOVA. This study does not include the inter-rater reliability of the GC rubric, because this part of the validation step was previously published in the literature [15]. Communication skills of a total of 1,100 PharmD students enrolled in Class 2010 to 2020 were evaluated using the GC rubric during the five OSCE sessions.
Examiners’ Perceptions in Surgical Education: The Blind Spot in the Assessment of OSCEs
2021, Journal of Surgical EducationExpert consensus to finalize a universal evaluator rubric to assess pharmacy students’ patient communication skills
2020, American Journal of Pharmaceutical EducationBlended learning: Improving the diabetes mellitus counseling skills of German pharmacy students
2020, Currents in Pharmacy Teaching and Learning