Skip to main content
SpringerLink
Log in

How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Learner handover (LH), the process of sharing of information about learners between faculty supervisors, allows for longitudinal assessment fundamental in the competency-based education model. However, the potential to bias future assessments has been raised as a concern. The purpose of this study is to determine whether prior performance information such as LH influences the assessment of learners in the clinical context. Between December 2017 and June 2018, forty-two faculty members and final-year residents from the Department of Medicine at the University of Ottawa were assigned to one of three study groups through quasi-randomisation, taking into account gender, speciality and rater experience. In a counter-balanced design, each group received either positive, negative or no LH prior to watching six simulated learner–patient encounter videos. Participants rated each video using the mini-CEX and completed a questionnaire on the raters’ general impressions of LH. A significant difference in the mean mini-CEX competency scale scores between the negative (M = 5.29) and positive (M = 5.97) LH groups (P < .001, d = 0.81) was noted. Similar findings were found for the single overall clinical competence ratings. In the post-study questionnaire, 22/28 (78%) of participants had correctly deduced the purpose of the study and 14/28 (50%) felt LH did not influence their assessment. LH influenced mini-CEX scores despite raters’ awareness of the potential for bias. These results suggest that LH could influence a rater’s performance assessment and careful consideration of the potential implications of LH is required.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Arts, J. A. R., Gijselaers, W. H., & Boshuizen, H. P. A. (2006). Understanding managerial problem-seolving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology, 31(4), 387–410.

    Article  Google Scholar 

  • Baltes, B. B., & Parker, C. P. (2000). Reducing the effects of performance expectations on behavioral ratings. Organizational Behavior and Human Decision Processes, 82(2), 237–267.

    Article  Google Scholar 

  • Busing, N., Rosenfield, J., Rungta, K., Raegele, M., Warren, A., Wright, B., et al. (2018). Smoothing the transition points in Canadian medical education. Academic Medicine, 93(5), 715–721.

    Article  Google Scholar 

  • Chong, L., Taylor, S., Haywood, M., Adelstein, B.-A., & Shulruf, B. (2018). Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia. Journal of Education Evaluation for Health Professionals, 15, 17.

    Article  Google Scholar 

  • Cleary, L. (2008). “Forward feeding” about students’ progress: The case for longitudinal, progressive, and shared assessment of medical students. Academic Medicine, 83(9), 800.

    Article  Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Psychology Press.

    Google Scholar 

  • Cohen, G. S., & Blumberg, P. (1991). Investigating whether teachers should be given assessments of students made by previous teachers. Academic Medicine, 66(5), 288–289.

    Article  Google Scholar 

  • Cox, S. M. (2008). “Forward feeding” about students’ progress: Information on struggling medical students should not be shared among clerkship directors or with students’ current teachers. Academic Medicine, 83(9), 801.

    Article  Google Scholar 

  • Dory, D., Young, M., Danoff, D., Plotnick, L., Pal, N., Gumuchian, S., et al., (2019). Learner handover: Does providing assessors with information about learners’ weaknesses influence assessment scores? Presented at the Annual Meeting of the Canadian Conference on Medical Education (CCME), Niagara Falls, Ontario.

  • Dudek, N. L., Marks, M. B., & Regehr, G. (2005). Failure to fail: The perspectives of clinical supervisors. Academic Medicine, 80(10), S84–S87.

    Article  Google Scholar 

  • Earl-Slater, A. (2002). The handbook of clinical trials and other research. Oxon: Radcliffe Medical Press Ltd.

    Google Scholar 

  • e-Laws. (2019). Freedom of Information and Protection of Privacy Act, R.S.O. 1990, c.F.31. Retrieved from https://www.ontario.ca/laws/statute/90f31.

  • Epstein, R. M. (2007). Assessment in medical education. New England Journal of Medicine, 356(4), 387–396.

    Article  Google Scholar 

  • Foti, R. J., & Hauenstein, N. M. A. (1993). Processing demands and the effects of prior impressions on subsequent judgments: Clarifying the assimilation/contrast debate. Organizational Behavior and Human Decision Processes, 56(2), 167–189.

    Article  Google Scholar 

  • Frellsen, S. L., Baker, E. A., Papp, K. K., & Durning, S. J. (2008). Medical school policies regarding strugging medical students during the internal medicine clerkships: Results of a national survey. Academic Medicine, 83(9), 876–881.

    Article  Google Scholar 

  • Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50, 511–522.

    Article  Google Scholar 

  • Gold, W. L., McArdle, P., & Federman, D. D. (2002). Should medical school faculty see assessments of students made by previous teachers? Academic Medicine, 77(11), 1096–1100.

    Article  Google Scholar 

  • Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education Theory and Practice, 16(2), 151–165.

    Article  Google Scholar 

  • Hanges, P. J., Braverman, E., & Rentsch, J. R. (1991). Changes in Raters’ perception of subordinates: A catastrophe model. Journal of Applied Psychology, 76(6), 878–888.

    Article  Google Scholar 

  • Hatala, R., Ainslie, M., Kassen, B. O., Mackie, I., & Roberts, J. M. (2006). Assessing the mini-clinical evaluation exercise in comparison to a national specialty examination. Medical Education, 40, 950–956.

    Article  Google Scholar 

  • Holmboe, E., Ginsburg, S., & Bernabeo, E. (2011). The rotational approach to medical education: Time to confront our assumptions? Medical Education, 45(1), 69–80.

    Article  Google Scholar 

  • Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682.

    Article  Google Scholar 

  • Huber, V. L. (1989). Comparison of the effects of specific and general performance standards on performance appraisal decisions. Decision Sciences, 20(3), 545–557.

    Article  Google Scholar 

  • Humphrey-Murto, S., Leblanc, A., Touchie, C., Pugh, D., Wood, T. J., Cowley, L., et al. (2019). The influence of prior performance information on ratings of present performance: Implications for learner handover: A scoping review. Academic Medicine, 94(7), 1050–1057.

    Article  Google Scholar 

  • Ioannidis, J. P. A. (2018). The proposal to lower P value thresholds to.005. Journal of the American Medical Association, 319(14), 1429–1430.

    Article  Google Scholar 

  • Jones, K. T., Hunt, S. C., & Chen, C. C. (2008). Auditors’ performance evaluations: An experimental analysis of the effects of initial impressions and task-specific experience on information later recalled. Accounting Forum, 32(3), 213–224.

    Article  Google Scholar 

  • Kassam, A., Ruetalo, M., Topps, M., et al. (2019). Key stakeholder opinions for a national learner education handover. BMC Medical Education, 19, 150.

    Article  Google Scholar 

  • Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. Journal of the American Medical Association, 302(12), 1316–1326.

    Article  Google Scholar 

  • Kravitz, D. A., & Balzer, W. K. (1992). Context effects in performance appraisal: A methodological critique and empirical study. Journal of Applied Psychology, 77(1), 24–31.

    Article  Google Scholar 

  • Lockyer, J., Carraccio, C., Chang, M.-K., Hart, D., Smee, S., Touchie, C., et al. (2017). Core principles of assessment in competency-based medical education. Medical Teacher, 39(6), 609–616.

    Article  Google Scholar 

  • Mims, L. D., DeCastro, A. O., & Kelly, A. G. (2017). Perspectives of family medicine clerkship directors regarding forward feeding: A CERA study. Family Medicine, 49(9), 699–705.

    Google Scholar 

  • Murphy, K. R., Balzer, W. K., Lockhart, M. C., & Eisenman, E. J. (1985). Effects of previous performance on evaluations of present performance. Journal of Applied Psychology, 70(1), 72–84.

    Article  Google Scholar 

  • Murphy, K. R., Gannett, B. A., Herr, B. M., & Chen, J. A. (1986). Effects of subsequent performance on evaluations of previous performance. Journal of Applied Psychology, 71(3), 427–431.

    Article  Google Scholar 

  • Nickell, L., Mountjoy, M., Bandiera, G., Topps, M., Maggi, J., Forgie, M., et al., (2018, April). Learner Education Handover: Easing the transition from undergraduate to postgraduate medicine. Symposium presented at the Canadian Conference on Medical Education, Halifax, Nova Scotia, Canada.

  • Nieminen, L. R. G., Rudolph, C. W., Baltes, B. B., Casper, C. M., Wynne, K. T., & Kirby, L. C. (2013). The combined effect of Ratee’s bodyweight and past performance information on performance judgments. Journal of Applied Social Psychology, 43, 527–543.

    Article  Google Scholar 

  • Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Medical Teacher, 33(3), 206–214.

    Article  Google Scholar 

  • Reilly, S. P., Smither, J. W., Warech, M. A., & Reilly, R. R. (1998). The influence of indirect knowledge of previous performance on ratings of present performance: The effects of job familiarity and rater training. Journal of Business and Psychology, 12(4), 421–435.

    Article  Google Scholar 

  • Royal College of Physicians and Surgeons of Canada. (2018). Guidelines for Educational Handover in Competence by Design (CBD). Retrieved from http://www.royalcollege.ca/rcsite/documents/cbd/guidelines-for-educational-handover-in-competence-by-design-e.pdf

  • Salvemini, N. J., Reilly, R. R., & Smither, J. W. (1993). The influence of rater motivation on assimilation effects and accuracy in performance ratings. Organizational Behavior and Human Decision Processes, 55(1), 41–60.

    Article  Google Scholar 

  • Smither, J. W., Reilly, R. R., & Buda, R. (1988). Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited. Journal of Applied Psychology, 73(3), 487–496.

    Article  Google Scholar 

  • Sozener, C. B., Lypson, M. L., House, J. B., Hopson, L. R., Dooley-Hash, S. L., Hauff, S., et al. (2016). Reporting achievement of medical student milestones to residency program directors: An educational handover. Academic Medicine, 91(5), 676–684.

    Article  Google Scholar 

  • Stroud, L., Herold, J., Tomlinson, G., & Cavalcanti, R. B. (2011). Who you know of what you know? Effect of examiner familiarity with residents on OSCE scores. Academic Medicine, 86(10), S8–S11.

    Article  Google Scholar 

  • Sullivan, G. M., & Feinn, R. (2012). Using effect size—or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279–282.

    Article  Google Scholar 

  • ten Cate, O., & Regehr, G. (2019). The power of subjectivity in the assessment of medical trainees (2018). Academic Medicine, 94(3), 333–337.

    Article  Google Scholar 

  • U.S Department of Education. (2018). Family Educational Rights and Privacy Act (FERPA). Retrieved from https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html.

  • van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317.

    Article  Google Scholar 

  • van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205–214.

    Article  Google Scholar 

  • Wancata, L. M., Morgan, H., Sandhu, G., Santen, S., & Hughes, D. T. (2017). Using the ACMGE milestones as a handover tool from medical school to surgery residency. Journal of Surgical Education, 74(3), 519–529.

    Article  Google Scholar 

  • Warm, E. J., Englander, R., Pereira, A., & Barach, P. (2017). Improving learner handovers in medical education. Academic Medicine, 92(7), 927–931.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013). ‘You’re certainly relatively competent’: Assessor bias due to recent experiences. Medical Education, 47(9), 910–922.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Lesley Ananny, Katherine Scowcroft and the Research Support Unit at the Department of Innovation in Medical Education for their ongoing support. The authors also thank the Department of Medicine at the University of Ottawa for their participation in this project.

Funding

The authors thank the Department of Medicine at the University of Ottawa, the Canadian Association of Medical Education (CAME) and the Association of Medical Education in Europe for their financial support through the Wooster Family grant and research grant programme respectively.

Author information

Authors and Affiliations

  1. Department of Medicine, University of Ottawa, Ottawa, ON, Canada

    Tammy Shaw, Claire Touchie, Debra Pugh & Susan M. Humphrey-Murto

  2. Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada

    Timothy J. Wood, Claire Touchie & Susan M. Humphrey-Murto

  3. Medical Council of Canada, Ottawa, Canada

    Claire Touchie & Debra Pugh

  4. The Ottawa Hospital – General Campus, 501 Smyth Road Box 209, Ottawa, ON, K1H 8L6, Canada

    Tammy Shaw

Authors
  1. Tammy Shaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timothy J. Wood
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claire Touchie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Debra Pugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Susan M. Humphrey-Murto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tammy Shaw.

Ethics declarations

Conflict of interest

None reported.

Ethical approval

The authors obtained ethical approval from the Ottawa Health Network Research Ethics Board (OHSN-REB).

Previous presentations

The authors have presented the results of this study at the University of Ottawa’s 2019 Meridith Marks Day local conference in Ottawa March 29, 2019, the 2019 Canadian Association of Medical Education (CAME) annual conference in Niagara Falls, Ontario April 14, 2019 and will present at the 2019 Association of Medical Education in Europe annual conference in Vienna, Austria August 27, 2019. The project is also Dr. Shaw’s M.Ed. thesis project at the University of Dundee (manuscript in progress).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Study design

See Table 5.

Table 5 Study design
Full size table

Appendix 2: Mini-CEX form

figure a

Appendix 3: learner–patient clinical encounter videos

See Table 6.

Table 6 Description of medical issues, patients and learners used in learner–patient clinical encounter videos
Full size table

Appendix 4: Learner handover form—example

The Program Director is now forwarding educational information about each learner based on previous evaluations (ITERs, written examinations, OSCEs) over the last 12 months. Here is the information about this learner for you to reflect upon.

figure b
figure c

Appendix 5: Post-video questionnaire—example of questions for groups A&B

  1. 1.

    What did you think was the purpose of this study?

  2. 2.

    Did you recognize any of the learners in the videos? If yes, which ones? If no, skip Questions 3&4.

  3. 3.

    On a scale of 1 (poor) to 10 (excellent), what was your impression of this (these) learner(s) prior to this session?

  4. 4.

    Did your former impression of this (these) learner(s) impact your assessment of the current performance?

You were provided with information about the learner. Please answer the following questions. Please elaborate on your answers.

  1. 5.

    How did you consider the learner handover/forward feeding information as you watched the video?

  2. 6.

    What information, if any, provided to you in the learner handover/forward feeding information was relevant to you in making assessment judgments? If so, in which instance? If varied with each video, please comment.

  3. 7.

    Did you feel the information provided was credible? Why or why not?

  4. 8.

    How do you think the learner handover/forward feeding information impacted your assessment?

  5. 9.

    What is your general opinion regarding learner handover?

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shaw, T., Wood, T.J., Touchie, C. et al. How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover. Adv in Health Sci Educ 26, 199–214 (2021). https://doi.org/10.1007/s10459-020-09979-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-020-09979-6

Keywords

Search

Navigation