Skip to main content
SpringerLink
Log in

How do trained raters take context factors into account when assessing GP trainee communication performance? An exploratory, qualitative study

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Communication assessment in real-life consultations is a complex task. Generic assessment instruments help but may also have disadvantages. The generic nature of the skills being assessed does not provide indications for context-specific behaviour required in practice situations; context influences are mostly taken into account implicitly. Our research questions are: 1. What factors do trained raters observe when rating workplace communication? 2. How do they take context factors into account when rating communication performance with a generic rating instrument? Nineteen general practitioners (GPs), trained in communication assessment with a generic rating instrument (the MAAS-Global), participated in a think-aloud protocol reflecting concurrent thought processes while assessing videotaped real-life consultations. They were subsequently interviewed to answer questions explicitly asking them to comment on the influence of predefined contextual factors on the assessment process. Results from both data sources were analysed. We used a grounded theory approach to untangle the influence of context factors on GP communication and on communication assessment. Both from the think-aloud procedure and from the interviews we identified various context factors influencing communication, which were categorised into doctor-related (17), patient-related (13), consultation-related (18), and education-related factors (18). Participants had different views and practices on how to incorporate context factors into the GP(-trainee) communication assessment. Raters acknowledge that context factors may affect communication in GP consultations, but struggle with how to take contextual influences into account when assessing communication performance in an educational context. To assess practice situations, raters need extra guidance on how to handle specific contextual factors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ander, D. S., Wallenstein, J., Abramson, J. L., Click, L., & Shayne, P. (2012). Reporter-interpreter-manager-educator (RIME) descriptive ratings as an evaluation tool in an emergency medicine clerkship. The Journal of Emergency Medicine, 43, 720–727.

    Article  Google Scholar 

  • Bensing, J., van Dulmen, S., & Tates, K. (2003). Communication in context: New directions in communication research. Patient Education and Counseling, 50, 27–32.

    Article  Google Scholar 

  • Brown, R. F., & Bylund, C. L. (2008). Communication skills training: Describing a new conceptual model. Academic Medicine, 83, 37–44.

    Article  Google Scholar 

  • Chan, C. S., Wun, Y. T., Cheung, A., Dickinson, J. A., Chan, K. W., et al. (2003). Communication skill of general practitioners: Any room for improvement? How much can it be improved? Medical Education, 37, 514–526.

    Article  Google Scholar 

  • Claramita, M., Dalen, J. V., & Van Der Vleuten, C. P. (2011). Doctors in a Southeast Asian country communicate sub-optimally regardless of patients’ educational background. Patient Education and Counseling, 85, e169–e174.

    Article  Google Scholar 

  • Claramita, M., Nugraheni, M. D. F., van Dalen, J., & van der Vleuten, C. (2013). Doctor-patient communication in Southeast Asia: A different culture? Advances in Health Sciences Education, 18, 15–31.

    Article  Google Scholar 

  • Crossley, J., & Davies, H. (2005). Doctors’ consultations with children and their parents: A model of competencies, outcomes and confounding influences. Medical Education, 39, 807–819.

    Article  Google Scholar 

  • Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Medical Education, 45, 560–569.

    Article  Google Scholar 

  • de Haes, H., & Bensing, J. (2009). Endpoints in medical communication research, proposing a framework of functions and outcomes. Patient Education and Counseling, 74, 287–294.

    Article  Google Scholar 

  • Durning, S., Artino, A. R., Jr, Pangaro, L., van der Vleuten, C. P., & Schuwirth, L. (2011). Context and clinical reasoning: Understanding the perspective of the expert’s voice. Medical Education, 45, 927–938.

    Article  Google Scholar 

  • Essers, G., Kramer, A., Andriesse, B., van Weel, C., van der Vleuten, C. P. M., & van Dulmen, A. M. (2013a). Context factors in family physician communication—patient encounters and their impact on assessing communication skills—an exploratory study. BMC Family Practice, 14, 65.

    Article  Google Scholar 

  • Essers, G., van Dulmen, S., van Es, J., van Weel, C., van der Vleuten, C., Kramer, A. (2013b). Context factors in consultations of general practitioner trainees and their impact on communication assessment in the authentic setting. Patient Education and Counseling. doi: 10.1016/j.pec.2013.08.024. [Epub ahead of print].

  • Essers, G., van Dulmen, S., van Weel, C., van der Vleuten, C., & Kramer, A. (2011). Identifying context factors explaining physician’s low performance in communication assessment: An explorative study in general practice. BMC Family Practice, 12, 138.

    Article  Google Scholar 

  • Eva, K. W. (2003). On the generality of specificity. Medical Education, 37, 587–588.

    Article  Google Scholar 

  • Glaser, B. G. (1965). The constant comparative method of qualitative-analysis. Social Problems, 12, 436–445.

    Article  Google Scholar 

  • Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16, 151–165.

    Article  Google Scholar 

  • Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59–82.

    Article  Google Scholar 

  • Hobma, S. O., Ram, P. M., Muijtjens, A. M. M., Grol, R., & van der Vleuten, C. P. M. (2004). Setting a standard for performance assessment of doctor-patient communication in general practice. Medical Education, 38, 1244–1252.

    Article  Google Scholar 

  • Hulsman, R. L. (2009). Shifting goals in medical communication. Determinants of goal detection and response formation. Patient Education and Counseling, 74, 302–308.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45, 1048–1060.

    Article  Google Scholar 

  • Kramer, A. W. M., Dusman, H., Tan, L. H. C., Jansen, J. J. M., Grol, R., & van der Vleuten, C. P. M. (2004). Acquisition of communication skills in postgraduate training for general practice. Medical Education, 38, 158–167.

    Article  Google Scholar 

  • Malau-Aduli, B. S. (2011). Exploring the experiences and coping strategies of international medical students. BMC Medical Education, 11, 40.

    Article  Google Scholar 

  • McGill, D. A., van der Vleuten, C. P. M., & Clarke, M. J. (2011). Supervisor assessment of clinical and professional competence of medical trainees: A reliability study using workplace data and a focused analytical literature review. Advances in Health Sciences Education, 16, 405–425.

    Article  Google Scholar 

  • Pangaro, L. (1999). A new vocabulary and other innovations for improving descriptive in-training evaluations. Academic Medicine, 74, 1203–1207.

    Article  Google Scholar 

  • Pangaro, L., & ten Cate, O. (2013). Frameworks for learner assessment in medicine: AMEE Guide No. 78. Medical Teacher, 35, E1197–E1210.

    Article  Google Scholar 

  • Pelgrim EAM. (2013). Clarifying observation and assessment feedback in workplace-based learning, chapter 7. Dissertation, Radboud UMC, Nijmegen. http://www.eampelgrim.nl/phd_thesis_pelgrim.pdf.

  • Pelgrim, E. A. M., & Kramer, A. W. M. (2013). How can medical education benefit from the evidence on learners seeking and using feedback? Medical Education, 47, 225–227.

    Article  Google Scholar 

  • Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012). The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Medical Education, 46, 604–612.

    Google Scholar 

  • Ram, P., Grol, R., Rethans, J. J., Schouten, B., van der Vleuten, C., & Kester, A. (1999). Assessment of general practitioners by video observation of communicative and medical performance in daily practice: Issues of validity, reliability and feasibility. Medical Education, 33, 447–454.

    Article  Google Scholar 

  • Ram, P., van Thiel, J., & van Dalen, J. (2003). MAAS-global manual 2000. Maastricht: Maastricht University.

    Google Scholar 

  • Ram, P., et al. (2011). National assessment plan for the general practitioner specialty training. Utrecht: General Practitioner Specialty Training in the Netherlands.

    Google Scholar 

  • Reinders, M. E., Blankenstein, A. H., van Marwijk, H. W. J., Knol, D. L., Ram, P., et al. (2011). Reliability of consultation skills assessments using standardised versus real patients. Medical Education, 45, 578–584.

    Article  Google Scholar 

  • Salmon, P., Mendick, N., & Young, B. (2011). Integrative qualitative communication analysis of consultation and patient and practitioner perspectives: Towards a theory of authentic caring in clinical relationships. Patient Education and Counseling, 82, 448–454.

    Article  Google Scholar 

  • Salmon, P., & Young, B. (2011). Creativity in clinical communication: From communication skills to skilled communication. Medical Education, 45, 217–226.

    Article  Google Scholar 

  • Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18, 291–303.

    Article  Google Scholar 

  • Tolsgaard, M. G., Arendrup, H., Lindhardt, B. O., Hillingso, J. G., Stoltenberg, M., & Ringsted, C. (2012). Construct validity of the reporter-interpreter-manager-educator structure for assessing students’ patient encounter skills. Academic Medicine, 87, 799–806.

    Article  Google Scholar 

  • Tweed, M., & Ingham, C. (2010). Observed consultation: Confidence and accuracy of assessors. Advances in Health Sciences Education, 15, 31–43.

    Article  Google Scholar 

  • van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39, 309–317.

    Article  Google Scholar 

  • van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34, 205–214.

    Article  Google Scholar 

  • van Es, J., De Waard-Wieringa, M., & Visser, M. (2013). Differential growth in doctor-patient communication skills in GP trainees. Medical Education, 47, 691–700.

    Article  Google Scholar 

  • van Es, JM., Schrijver, CJW., Oberink, RHH., & Visser, MRM. (2012). Two-dimensional structure of the MAAS-Global rating list for consultation skills of doctors. Medical Teacher, e1–e6. doi:10.3109/0142159X.2012.709652.

  • van Thiel, J., Kraan, H. F., & Vleuten, C. (1991). Reliability and feasibility of measuring medical interviewing skills: The revised Maastricht History-Taking and Advice Checklist. Medical Education, 25(3), 224–229.

    Article  Google Scholar 

  • Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15, 270–292.

    Article  Google Scholar 

  • Winefield, H. R., Murrell, T. G. C., Clifford, J. V., & Farmer, E. A. (1995). Process and outcomes in general practice consultations: Problems in defining high quality care. Social Science & Medicine, 41, 6.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., & Mann, K. (2011). Examining the box’s contents. Medical Education, 45, 970–972.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2012). Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Sciences. doi: 10.1007/s10459-012-9372-1. [Epub ahead of print].

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013). ‘You’re certainly relatively competent’: Assessor bias due to recent experiences. Medical Education, 47, 910–922.

    Article  Google Scholar 

  • Zoppi, K., & Epstein, R. M. (2002). Is communication a skill? Communication behaviors and being in relation. Family Medicine, 34, 319–324.

    Google Scholar 

Download references

Acknowledgments

We thank all GPs who participated in the study. Special thanks go to Vera Atema who interviewed the Leiden raters, and to Dragana Milic, Fleur van den Bogaard, Tim Butterbrod, Ine Smeets, and Maxime Essers, who transcribed the recordings.

Conflict of interests

The study was funded by the SBOH Foundation, employer of Dutch General Practitioner Registrars and funder of the national specialty training for General Practitioners. There are no conflict of interests.

Author information

Authors and Affiliations

  1. Department of Public Health and Primary Care, Leiden University Medical Centre, Hippocratespad 21, 2333 ZP, Leiden, The Netherlands

    Geurt Essers

  2. Department of Primary and Community Care, Radboud University Medical Centre, Nijmegen, The Netherlands

    Geurt Essers, Patrick Dielissen, Chris van Weel, Cees van der Vleuten, Sandra van Dulmen & Anneke Kramer

  3. Australian Primary Health Care Research Institute, Australian National University, Canberra, Australia

    Chris van Weel

  4. Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands

    Cees van der Vleuten

  5. Northumbria University, Newcastle upon Tyne, UK

    Cees van der Vleuten

  6. Copenhagen University, Copenhagen, Denmark

    Cees van der Vleuten

  7. NIVEL (Netherlands Institute for Health Services Research), Utrecht, The Netherlands

    Sandra van Dulmen

  8. Department of Health Science, Buskerud University College, Drammen, Norway

    Sandra van Dulmen

Authors
  1. Geurt Essers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick Dielissen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chris van Weel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cees van der Vleuten
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sandra van Dulmen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anneke Kramer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Geurt Essers.

Appendices

Appendix 1: Think-aloud protocol and interview questions

Introduction and think-aloud protocol

 Assessing GP trainees in their workplace is one of your tasks. It is an important task, for its aim is that the trainees learn from your feedback. We know that assessing workplace performance is not an easy job, and that raters all have their idiosyncratic ways in doing this. The aim of this study is to get a more specific idea of the considerations and thoughts that raters have when assessing communication performance using the MAAS-Global scoring instrument. The results of this study may be relevant to rater training in the future

 We will perform this study by showing you two videotaped consultations consecutively, which will be followed by a number of questions after each video. It should be clear to you that this is not a test of your assessment abilities, nor will you be judged as a result of participating in this study. There is no right or wrong in the answers you give or in the thoughts you express! Nor is it of any concern to us if your judgment is right or wrong. We are merely interested in your thoughts concerning the GP trainees performance in that situation

 We will therefore ask you to think-aloud during your assessment and to express all thoughts and considerations you have. You may act as if you are alone in this room and are talking to yourself. In order to carefully reproduce what you have said, we will record this interview on audio tape. Is it clear what we ask of you?

 Now, you are going to see a consultation of a first-year GP trainee at nine months of his/her training programme. Suppose this is a situation in which you are to assess his/her communication performance. You can do this as you are used to do when assessing a video in a real assessment. However, now you are invited to express your thoughts and feelings aloud while doing so. You may use the papers we provide to make notes and the MAAS-Global scoring form for your rating. Our main leading question is: “What thoughts and considerations (regarding the consultation, the GP trainee, the patient and/or the assessment itself) are relevant to the assessment of the GP trainees communication performance?

 I will start the video and also kindly invite you to express your first impression as soon as you have these

Interview questions

 1. Having seen this consultation on video, what feedback would you give the GP trainee concerning his/her communication performance based on this?

 2. What feedback do you have regarding the assessment itself?

 3. In this consultation, you may have noticed X (e.g. more than one person was attending, or the patient seemed to know the PE he was about to undergo, or the GP trainee and the patient seemed to know each other, or the consultation seemed to take place on the initiative of the GP, or the problem presented seemed easy to solve, or the patient did not seem to master the Dutch language). Does this (in general) make a difference in how you rate the GP trainee’s communication performance?

 4. What do you consider most important when assessing communication performance?

 5. Are there any other factors (concerning the trainee, the patient or the consultation) that you tend to take into account when assessing GP trainee communication performance?

 6. Did the way you now assessed the trainee’s performance differ from the way you usually do this? If so, how is it different?

Appendix 2: MAAS-Global rating list for doctor-patient communication skills

Each item is scored on a scale ranging of 0-6. For the items 2 and 4, the rating ‘not applicable’ is an additional option.

Communication skills for each separate phase

 1. Introduction

  Giving the patient room to tell his story

  General orientation on the reason for visit

  Asking about other reasons for visit

 2. Follow-up consultation

  Naming previous complaints, requests for help and management plan

  Asking about adherence to management plan

  Asking about the course of the complaint

 3. Request for help

  Naming requests for help, wishes or expectations

  Naming reasons that prompted the patient to come now

  Completing exploring request for help

 4. Physical examination

  Instructions to the patient

  Explanation of what is being done

  Treating the patient with care and respect

 5. Diagnosis

  Naming findings and diagnosis/hypothesis

  Naming causes or the relation between findings and diagnosis

  Naming prognosis or expected course

  Asking for the patient’s response

 6. Management

  Shared decision making, discussing alternatives, risks and benefits

  Discussing feasibility and adherence

  Determining who will do what and when

  Asking for patient’s response

 7. Evaluation of consultation

  General question

  Responding to requests for help

  Perspective for the time being

General communication skills

 8. Exploration

  Exploring requests for help, wishes or expectations

  Exploring patient’s response to information given within patient’s frame of reference

  Responding to non-verbal behaviour and cues

 9. Emotions

  Asking about/exploring feelings

  Reflecting feelings (including nature and intensity)

  Sufficiently throughout the entire consultation

 10. Information giving

  Announcing, categorizing

  In small quantities, concrete explanations

  Understandable language

  Asking whether the patient understands

 11. Summarisations

  Content is correct, complete

  Concise, rephrased

  Checking

  Sufficiently throughout the entire consultation

 12. Structuring

  Logical sequence of phases

  Balanced division of time

  Announcing (history taking, examination, other phases)

 13. Empathy

  Concerned, inviting and sincerely empathetic in intonation, gesture and eye contact

  Expressing empathy in brief verbal responses

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Essers, G., Dielissen, P., van Weel, C. et al. How do trained raters take context factors into account when assessing GP trainee communication performance? An exploratory, qualitative study. Adv in Health Sci Educ 20, 131–147 (2015). https://doi.org/10.1007/s10459-014-9511-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-014-9511-y

Keywords

Search

Navigation