Skip to main content
Log in

Towards a program of assessment for health professionals: from training into practice

  • Reflections
  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Despite multifaceted attempts to “protect the public,” including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • ABA: American Board of Anesthesiology. (2014). MOCA minute. http://www.theaba.org/MOCA/MOCA-Minute. Last accessed November 2, 2015.

  • AFMC: Association of Faculties of Medicine in Canada. (2010). The Future of Medical Education in Canada (FMEC): A collective vision for MD education. Retrieved from http://www.afmc.ca/fmec/pdf/collective_vision.pdf.

  • Bernabeo, E., Hood, S., Iobst, W., Holmboe, E., & Caverzagie, K. (2013). Optimizing the implementation of practice improvement modules in training: Lessons from educators. Journal of Graduate Medical Education, 5(1), 74–80.

    Article  Google Scholar 

  • Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

    Google Scholar 

  • Bogo, M., Regehr, C., Logie, C., et al. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education, 47, 5–18.

    Article  Google Scholar 

  • Bordage, G., Meguerditchian, A. N., & Tamblyn, R. (2013). Avoidable adverse events: A content analysis of a national qualifying examination. Academic Medicine, 88, 1493–1498.

    Article  Google Scholar 

  • Boud, D., & Molloy, E. (Eds.). (2013). Feedback in higher and professional education: Understanding it and doing it well. London: Routledge.

    Google Scholar 

  • Butler, R. (1987). Task-involving and ego-involving properties of evaluation: Effects of different feedback conditions on motivational perceptions, interest, and performance. Journal of Educational Psychology, 79, 474–482.

    Article  Google Scholar 

  • Cadieux, G., Tamblyn, R., Dauphinee, D., & Libman, M. (2007). Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ, 177(8), 877–883.

    Article  Google Scholar 

  • Choudhry, N. K., Fletcher, R. H., & Soumerai, S. B. (2005). Systematic review: The relationship between clinical experience and quality of health care. Annals of Internal Medicine, 142(4), 260–273.

    Article  Google Scholar 

  • Cizek, G. J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justification on test use. Psychological Methods, 17, 31–43.

    Article  Google Scholar 

  • Colliver, J. A. (2002). Educational theory and medical education practice: A cautionary note for medical school faculty. Academic Medicine, 77(12), 1217–1220.

    Article  Google Scholar 

  • Cook, D. A. (2014). When I say… validity. Medical Education, 48(10), 948–949.

    Article  Google Scholar 

  • Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49, 560–575.

    Article  Google Scholar 

  • Cote, L., & Bordage, G. (2012). Content and conceptual frameworks of preceptor feedback in response to residents’ educational needs. Academic Medicine, 87(9), 1274–1281.

    Article  Google Scholar 

  • Cruess, R., & Cruess, S. (2014). Updating the Hippocratic Oath to include medicine’s social contract. Medical Education, 48(1), 95–100.

    Article  Google Scholar 

  • Custers, E. (2010). Long-term retention of basic science knowledge: A review study. Advances in Health Sciences Education, 15(1), 109–128.

    Article  Google Scholar 

  • Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830–837.

    Article  Google Scholar 

  • Ellaway, R. H., Pusic, M. V., Galbraith, R. M., & Cameron, T. (2014). Developing the role of big data and analytics in health professional education. Medical Teacher, 36(3), 216–222.

    Article  Google Scholar 

  • Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79, S70–S81.

    Article  Google Scholar 

  • Eva, K. W. (2002). The aging physician: Changes in cognitive processing and their impact on medical practice. Academic Medicine, 77, S1–S6.

    Article  Google Scholar 

  • Eva, K. W. (2003). On the generality of specificity. Medical Education, 37, 587–588.

    Article  Google Scholar 

  • Eva, K. W. (2009). Diagnostic error in medical education: Where wrongs can make rights. Advances in Health Sciences Education, 14, 71–81.

    Article  Google Scholar 

  • Eva, K. W., Bordage, G., Campbell, C., Galbraith, R., Ginsburg, S., Holmboe, E., & Regehr, G. (2013). Medical Education Assessment Advisory Committee report to the Medical Council of Canada on Current Issues in Health Professional and Health Professional Trainee Assessment. Retrieved from http://mcc.ca/wp-content/uploads/Reports-MEAAC.pdf.

  • Eva, K. W., & Cunnington, J. P. (2006). The difficulty with experience: Does practice increase susceptibility to premature closure? Journal of Continuing Education in the Health Professions, 26(3), 192–198.

    Article  Google Scholar 

  • Eva, K. W., & Hodges, B. D. (2012). Scylla or Charbydis? Can we navigate between objectification and judgment in assessment? Medical Education, 46, 914–919.

    Article  Google Scholar 

  • Eva, K. W., Munoz, J., Hanson, M. D., Walsh, A., & Wakefield, J. (2010). Which factors, personal or external, most influence students’ generation of learning goals? Academic Medicine, 85, S102–S105.

    Article  Google Scholar 

  • Eva, K. W., & Regehr, G. (2013). Effective feedback for maintenance of competence: From data delivery to trusting dialogues. CMAJ, 185, 463–464.

    Article  Google Scholar 

  • Eva, K. W., Regehr, G., & Gruppen, L. D. (2012). Blinded by ‘insight’: Self-assessment and its role in performance improvement. In B. D. Hodges & L. Lingard (Eds.), The question of competence: Reconsidering medical education in the twenty-first century (pp. 131–154). Ithaca, NY: Cornell University Press.

    Google Scholar 

  • Farmer, E. A., & Page, G. (2005). A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39, 1188–1194.

    Article  Google Scholar 

  • Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., et al. (2010). Competency-based medical education: theory to practice. Medical Teacher, 32(8), 638–645.

    Article  Google Scholar 

  • Galbraith, R. M., Clyman, S., & Melnick, D. E. (2011). Conceptual perspectives: Emerging changes in the assessment paradigm. In J. P. Hafler (Ed.), Extraordinary learning in the workplace (pp. 87–100). Berlin: Springer.

    Chapter  Google Scholar 

  • Galbraith, R. M., Hawkins, R. E., & Holmboe, E. S. (2008). Making self-assessment more effective. Journal of Continuing Education in the Health Professions, 28(1), 20–24.

    Article  Google Scholar 

  • Gierl, M. J., & Lai, H. (2013). Evaluating the quality of medical multiple-choice items created with automated processes. Medical Education, 47(7), 726–733.

    Article  Google Scholar 

  • Gierl, M. J., Lai, H., & Turner, S. R. (2012). Using automatic item generation to create multiple-choice test items. Medical Education, 46(8), 757–765.

    Article  Google Scholar 

  • Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the ‘black box’ differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.

    Article  Google Scholar 

  • Ginsburg, S., Eva, K., & Regehr, G. (2013). Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Academic Medicine, 88(10), 1539–1544.

    Article  Google Scholar 

  • Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.

    Article  Google Scholar 

  • Ginsburg, S., Regehr, G., & Lingard, L. (2004). Basing the evaluation of professionalism on observable behaviours: A cautionary tale. Academic Medicine, 79(10, Suppl), S1–S4.

    Article  Google Scholar 

  • Goldszmidt, M., Minda, J. P., & Bordage, G. (2013). What physicians reason about during clinical encounters: Time to be more explicit. Academic Medicine, 88(3), 390–394.

    Article  Google Scholar 

  • Guadagnoli, M., Morin, M. P., & Dubrowski, A. (2012). The application of the challenge point framework in medical education. Medical Education, 46(5), 447–453.

    Article  Google Scholar 

  • Harrison, C. J., Könings, K. D., Schuwirth, L., Wass, V., & van der Vleuten, C. (2015). Barriers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education, 20(1), 229–245.

    Article  Google Scholar 

  • Hatala, R., Marr, S., Cuncic, C., & Bacchus, C. M. (2011). Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance. BMC Medical Education, 11, 23.

    Article  Google Scholar 

  • Hawkins, et al. (under review). The ABMS MOC Part III examination: Value, concerns and alternative formats.

  • Hays, R., & Gay, S. (2011). Reflection or ‘pre-reflection’: What are we actually measuring in reflective practice? Medical Education, 45(2), 116–118.

    Article  Google Scholar 

  • Hodges, B. (2003). OSCE! variations on a theme by Harden. Medical Education, 37(12), 1134–1140.

    Article  Google Scholar 

  • Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682.

    Article  Google Scholar 

  • James, J. T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety, 9(3), 122–128.

    Article  Google Scholar 

  • Jarvis-Selinger, S., Pratt, D. D., & Regehr, G. (2012). Competency is not enough: integrating identity formation into the medical education discourse. Academic Medicine, 87(9), 1185–1190.

    Article  Google Scholar 

  • Kane, M. T. (1992). An argument-based approach to validation. Psychological Bulletin, 112, 527–535.

    Article  Google Scholar 

  • Karpicke, J. D., & Roediger, H. L, I. I. I. (2008). The critical importance of retrieval for learning. Science, 319, 966–968.

    Article  Google Scholar 

  • Kennedy, T. J., Regehr, G., Baker, G. R., & Lingard, L. A. (2009). ‘It’s a cultural expectation…’ The pressure on medical trainees to work independently in clinical practice. Medical Education, 43(7), 645–653.

    Article  Google Scholar 

  • Klass, D. A. (2007). Performance-based conception of competence is changing the regulation of physicians’ professional behavior. Academic Medicine, 82(6), 529–535.

    Article  Google Scholar 

  • Kluger, A. N., & van Dijk, D. (2010). Feedback, the various tasks of the doctor, and the feedforward alternative. Medical Education, 44, 1166–1174.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. S. (2011). Opening the black box of postgraduate trainee assessment in the clinical setting via observation: A conceptual model. Medical Education, 45, 1048–1060.

    Article  Google Scholar 

  • Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based assessment. Teaching and Learning in Medicine, 25(Suppl 1), S68–S74.

    Article  Google Scholar 

  • Kogan, J. R., Holmboe, E. S., & Hauer, K. R. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA, 302, 1316–1326.

    Article  Google Scholar 

  • Kohn, L. T., Corrigan, J. M., & Donaldson, M. S. (Eds.). (1999). To err is human: building a safer health system. Washington, DC: National Academy Press, Institute of Medicine.

    Google Scholar 

  • Kornell, N., & Son, L. K. (2009). Learners’ choices and beliefs about self-testing. Memory, 17, 493–501.

    Article  Google Scholar 

  • Kromann, C. B., Bohnstedt, C., Jensen, M. L., & Ringsted, C. (2010). The testing effect on skills learning might last 6 months. Advances in Health Sciences Education, 15(3), 395–401.

    Article  Google Scholar 

  • Krumholz, et al. (under review). Recommendations to the American Board of Internal Medicine (ABIM): A vision for certification in internal medicine in 2020.

  • Larsen, D. P., Butler, A. C., & Roediger, H. L, 3rd. (2008). Test-enhanced learning in medical education. Medical Education, 42(10), 959–966.

    Article  Google Scholar 

  • MacRae, H. M., Cohen, R., Regehr, G., Reznick, R., & Burnstein, M. (1997). A new assessment tool: the patient assessment and management examination. Surgery, 122(2), 335–343.

    Article  Google Scholar 

  • Mann, K., Gordon, J., & MacLeod, A. (2009). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Sciences Education, 14(4), 595–621.

    Article  Google Scholar 

  • Mann, K. V., van der Vleuten, C., Eva, K., Armson, H., Chesluk, B., Dornan, T., et al. (2011). Tensions in informed self-assessment: How the desire for feedback and reticence to collect and use it conflict. Academic Medicine, 86, 1120–1127.

    Article  Google Scholar 

  • Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187–1197.

    Article  Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–104). New York: American Council on Education and Macmillan.

    Google Scholar 

  • Morcke, A. M., Dornan, T., & Elka, B. (2013). Outcome (competency) based education: an exploration of its origins, theoretical basis and empirical evidence. Advances in Health Sciences Education, 18, 851–863.

    Article  Google Scholar 

  • Mutabdzic, D., Mylopoulos, M., Murnaghan, M. L., Patel, P., Zilbert, N., Seemann, N., et al. (2015). Coaching surgeons: Is culture limiting our ability to improve? Annals of Surgery, 262(2), 213–216.

    Article  Google Scholar 

  • Mylopoulos, M., & Regehr, G. (2011). Putting the expert together again. Medical Education, 45(9), 920–926.

    Article  Google Scholar 

  • Mylopoulos, M., & Scardamalia, M. (2008). Doctors’ perspectives on their innovations in daily practice: implications for knowledge building in health care. Medical Education, 42(10), 975–981.

    Article  Google Scholar 

  • Neve, H., & Hanks, S. (2016). When I say … capability. Medical Education, 50 (in press).

  • Newble, D. I., & Jaeger, K. (1983). The effect of assessments and examinations on the learning of medical students. Medical Education, 17(3), 165–171.

    Article  Google Scholar 

  • Newell, K. M., Liu, Y., & Mayer-Kress, G. (2001). Time scales in motor learning and development. Psychological Review, 108, 57–82.

    Article  Google Scholar 

  • Norcini, J. J. (2005). Current perspectives in assessment: The assessment of performance at work. Medical Education, 39(9), 880–889.

    Article  Google Scholar 

  • Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 206–214.

    Article  Google Scholar 

  • Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.

    Article  Google Scholar 

  • Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29(9), 855–871.

    Article  Google Scholar 

  • Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation fidelity and transfer of learning. Medical Education, 46(7), 636–647.

    Article  Google Scholar 

  • Norman, G., Neville, A., Blake, J. M., & Mueller, B. (2010). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32(6), 496–499.

    Article  Google Scholar 

  • Norman, G. R., Norcini, J., & Bordage, G. (2014). Competency-based education: Milestones or millstones. Journal of Graduate Medical Education, 6(1), 1–6.

    Article  Google Scholar 

  • Page, G., & Bordage, G. (1995). The medical council of Canada’s key feature project: A more valid written exam. of clinical decision-making skills. Academic Medicine, 70, 104–110.

    Article  Google Scholar 

  • Pugh, D., Hamstra, S. J., Wood, T. J., Humphrey-Murto, S., Touchie, C., Yudkowsky, R., Bordage, G. (2014). A procedural skills OSCE: Assessing technical and non-technical skills of internal medicine residents. Advances in health sciences education. Retrieved from http://link.springer.com/article/10.1007/s10459-014-9512-x?sa_campaign=email/event/articleAuthor/onlineFirst.

  • Razack, S., Hodges, B., Steinert, Y., & Maguire, M. (2015). Seeking inclusion in an exclusive process: Discourses of medical school student selection. Medical Education, 49, 36–47.

    Article  Google Scholar 

  • RCPSC: Royal College of Physicians and Surgeons of Canada. (2011). Assessment strategies within the revised maintenance of certification program, draft recommendations.

  • Regehr, G. (1994). Chickens and children do not an expert make. Academic Medicine, 69, 970–971.

    Article  Google Scholar 

  • Regehr, G., Eva, K., Ginsburg, S., Halwani, Y., & Sidhu, R. (2011). Future of medical education in Canada postgraduate project environmental scan. Paper 13. Assessment in postgraduate medical education: Trends and issues in assessment in the workplace. Retrieved from http://www.afmc.ca/pdf/fmec/13_Regehr_Assessment.pdf.

  • Rohrer, D., & Pashler, H. (2010). Recent research on human learning challenges conventional instructional strategies. Educational Research, 38, 406–412.

    Article  Google Scholar 

  • Sargeant, J., Eva, K. W., Armson, H., Chesluk, B., Dornan, T., Holmboe, E., et al. (2011). Features of assessment learners use to make informed self-assessments of clinical performance. Medical Education, 45, 636–647.

    Article  Google Scholar 

  • Schön, D. (1983). The reflective practitioner: How professionals think in action. London: Temple Smith.

    Google Scholar 

  • Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153–189.

    Article  Google Scholar 

  • Swanson, D., & Roberts, T. (2016). Trends in national licensing examinations. Medical Education, 50(1) (in press).

  • Tamblyn, R., Abrahamowicz, M., Dauphinee, D., et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA, 298(9), 993–1001.

    Article  Google Scholar 

  • Teunissen, P. W., & Westerman, M. (2011). Opportunity or threat: The ambiguity of the consequences of transitions in medical education. Medical Education, 45(1), 51–59.

    Article  Google Scholar 

  • van der Vleuten, C. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1, 41–67.

    Article  Google Scholar 

  • van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317.

    Article  Google Scholar 

  • van Tartwijk, J., & Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE Guide no. 45. Medical Teacher, 31(9), 790–801.

    Article  Google Scholar 

  • Watling, C., Driessen, E., van der Vleuten, C. P., & Lingard, L. (2014). Learning culture and feedback: An international study of medical athletes and musicians. Medical Education, 48(7), 713–723.

    Article  Google Scholar 

  • Wenghofer, E., Klass, D., Abrahamowicz, M., et al. (2009). Doctor scores on national qualifying examinations predict quality of care in future practice. Medical Education, 43(12), 1166–1173.

    Article  Google Scholar 

  • Williams, R. G., Klamen, D. L., Markwell, S. J., Cianciolo, A. T., Colliver, J. A., & Verhulst, S. J. (2014). Variations in senior medical student diagnostic justification ability. Academic Medicine, 89(5), 790–798.

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the Medical Council of Canada (MCC) through the work of the authors as members of the Medical Education Assessment Advisory Committee. The focus was not constrained to MCC practices, however, and the content of the paper does not necessarily reflect MCC policy.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kevin W. Eva.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Eva, K.W., Bordage, G., Campbell, C. et al. Towards a program of assessment for health professionals: from training into practice. Adv in Health Sci Educ 21, 897–913 (2016). https://doi.org/10.1007/s10459-015-9653-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-015-9653-6

Keywords

Navigation