Swipe om te navigeren naar een ander artikel
The online version of this article (https://doi.org/10.1007/s40037-018-0433-x) contains supplementary material, which is available to authorized users.
Assessment in Medical Education fills many roles and is under constant scrutiny. Assessments must be of good quality, and supported by validity evidence. Given the high-stakes consequences of assessment, and the many audiences within medical education (e. g., training level, specialty-specific), we set out to document the breadth, scope, and characteristics of the literature reporting on validation of assessments within medical education.
Searches in Medline (Ovid), Web of Science, ERIC, EMBASE (Ovid), and PsycINFO (Ovid) identified articles reporting on assessment of learners in medical education published since 1999. Included articles were coded for geographic origin, journal, journal category, targeted assessment, and authors. A map of collaborations between prolific authors was generated.
A total of 2,863 articles were included. The majority of articles were from the United States, with Canada producing the most articles per medical school. Most articles were published in journals with medical categorizations (73.1% of articles), but Medical Education was the most represented journal (7.4% of articles). Articles reported on a variety of assessment tools and approaches, and 89 prolific authors were identified, with a total of 228 collaborative links.
Literature reporting on validation of assessments in medical education is heterogeneous. Literature is produced by a broad array of authors and collaborative networks, reported to a broad audience, and is primarily generated in North American and European contexts. Our findings speak to the heterogeneity of the medical education literature on assessment validation, and suggest that this heterogeneity may stem, at least in part, from differences in constructs measured, assessment purposes, or conceptualizations of validity.
ESM-Appendix 1 Search strategy and Appendix 2 Authors with five or more publications included in the study (author in first, last, or sole author position)40037_2018_433_MOESM1_ESM.docx
ESM-Figure 1 Number of publications reporting validation of assessment across time40037_2018_433_MOESM2_ESM.jpg
ESM-Table 1 Journals with 15 or more published articles included in the study40037_2018_433_MOESM3_ESM.docx
ESM-Table 2 List of categories representing at least 1% of total publications, and the number of journals associated with each category40037_2018_433_MOESM4_ESM.docx
Roediger HL, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006;17:249–55. CrossRef
Roediger HL, Karpicke JD. The power of testing memory: basic research and implications for educational practice. Perspect Psychol Sci. 2006;1:181–210. CrossRef
Larsen DP, Butler AC, Roediger HL III. Test-enhanced learning in medical education. Med Educ. 2008;42:959–66. CrossRef
Larsen DP, Butler AC, Roediger HL III. Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Med Educ. 2009;43:1174–81. CrossRef
Tamblyn R, Abrahamowicz M, Brailovsky C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. J Am Med Assoc. 1998;280:989–96. CrossRef
Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between lincensure examination scores and practice in primary care. J Am Med Assoc. 2002;288:3019–26. CrossRef
Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007;298:993–1001. CrossRef
Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–96. CrossRef
Norcini JJ. Workbased assessment. BMJ. 2003;326:753–5. CrossRef
Wass V, van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357:945–9. CrossRef
Sherbino J, Bandiera G, Frank JR. Assessing competence in emergency medicine trainees : an overview of effective methodologies. CMEJ. 2008;10:365–71.
Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33:206–14. CrossRef
Cizek GJ, Rosenberg SL, Koons HH. Sources of validity evidence for educational and psychological tests. Educ Psychol Meas. 2008;68:397–412. CrossRef
Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013;88:872–83. CrossRef
Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014;19:233–50. CrossRef
Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830–7. CrossRef
Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166e7–166e16. CrossRef
Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49:560–75. CrossRef
Schuwirth LWT, van der Vleuten CPM. Programmatic assessment and Kane’s validity perspective. Med Educ. 2012;46:38–48. CrossRef
Albert M. Understanding the debate on medical education research: a sociological perspective. Acad Med. 2004;79:948–54. CrossRef
Albert M, Hodges B, Regehr G. Research in medical edication: balancing service and science. Adv Health Sci Educ Theory Pract. 2007;12:103–15. CrossRef
St-Onge C, Young M, Eva KW, Hodges B. Validity: One word with a plurality of meanings. Adv Health Sci Educ. 2017;22(4):853. CrossRef
Streiner DL, Norman G, Cairney J. Health measurement scales: a practical guide to their development and use. 5th ed. Oxford: Oxford University Press; 2015. CrossRef
Messick S. Standards of validity and the validity standards in performance assessment. Educ Meas Issue Pract. 1995;14:5–8. CrossRef
Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50:1–73. CrossRef
Sampson M, Horsley T, Doja A. A bibliometric analysis of evaluative medical education studies : characteristics and indexing accuracy. Acad Med. 2013;88(3):421–7. CrossRef
Broadus R. Towards a definition of “bibliometrics.”. Scientometrics. 1987;12:373–9. CrossRef
Doja A, Horsley T, Sampson M. Productivity in medical education research : an examination of countries of origin. BMC Med Educ. 2014;14:1–9. CrossRef
Azer SA. The top-cited articles in medical education : a bibliometric analysis. Acad Med. 2015;90:1147–61. CrossRef
Smith DR. Bibliometrics, citation indexing, and the journals of nursing. Nurs Health Sci. 2008;10:260–4. CrossRef
AERA, APA, NCME (American Educational Research Association & National Council on Measurement in Education), Joint Committee on Standards for Educational and Psychological Testing APA. Standards for educational and psychological testing. Washington, DC: AERA; 1999.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. CrossRef
World Directory of Medical Schools. World Federation for Medical Education (WFME) and the Foundation for Advancement of International Medical Education and Research (FAIMER). ( https://search.wdoms.org). Updated 2015. Accessed May 10, 2017.
U.S. National Library of Medicine. Broad Subject Terms for Indexed Journals. ( https://wwwcf.nlm.nih.gov/serials/journals/index.cfm). Published February 15, 2009. Updated Hanuary 24, 2017. Accessed May 10, 2017.
St-Onge C, Young M. Evolving conceptualisations of validity: impact on the process and outcome of assessment. Med Educ. 2015;49(6):548–50. CrossRef
Young ME, Thomas A, Lubarsky S, et al. Drawing boundaries: The difficulty of defining clinical reasoning. Acad Med. 2018; https://doi.org/10.1097/ACM.0000000000002142.
Cizek G. Defining and distinguishing validity: interpretations of score meaning and justifications of test use. Psychol Methods. 2012;17:31–43. CrossRef
- Characterizing the literature on validity and assessment in medical education: a bibliometric study
Elise Vachon Lachiver
- Bohn Stafleu van Loghum