Swipe om te navigeren naar een ander artikel
In a script concordance test (SCT), examinees are asked to judge the effect of a new piece of clinical information on a proposed hypothesis. Answers are collected using a Likert-type scale (ranging from −2 to +2, with ‘0’ indicating no effect), and compared with those of a reference panel of ‘experts’. It has been argued, however, that SCT may be susceptible to the influences of gaming and guesswork. This study aims to address some of the mounting concern over the response process validity of SCT scores.
Using published datasets from three independent SCTs, we investigated examinee response patterns, and computed the score a hypothetical examinee would obtain on each of the tests if he 1) guessed random answers and 2) deliberately answered ‘0’ on all test items.
A simulated random guessing strategy led to scores 2 SDs below mean scores of actual respondents (Z-scores −3.6 to −2.1). A simulated ‘all-0’ strategy led to scores at least 1 SD above those obtained by random guessing (Z-scores −2.2 to −0.7). In one dataset, stepwise exclusion of items with modal panel response ‘0’ to fewer than 10% of the total number of test items yielded hypothetical scores 2 SDs below mean scores of actual respondents.
Random guessing was not an advantageous response strategy. An ‘all-0’ response strategy, however, demonstrated evidence of artificial score inflation. Our findings pose a significant threat to the SCT’s validity argument. ‘Testwiseness’ is a potential hazard to all testing formats, and appropriate countermeasures must be established. We propose an approach that might be used to mitigate a potentially real and troubling phenomenon in script concordance testing. The impact of this approach on the content validity of SCTs merits further discussion.
Charlin B, van der Vleuten C. Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach. Eval Health Prof. 2014;27:304–19. CrossRef
Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Acad Med. 1990;65:611–21. CrossRef
Lemieux M, Bordage G. Propositional versus structural semantic analyses of medical diagnostic thinking. Cogn Sci. 1992;16:185–204. CrossRef
Feltovich PJ, Barrows HS. Issues of generality in medical problem solving. In: Schmidt H, De Volder ML, editors. Tutorials in problem-based learning: a new direction in teaching the health professions. Assen: Van Gorcum; 1984.
Charlin B, Boshuizen H, Custers E, Feltovitch P. Scripts and clinical reasoning. Med Educ. 2007;41:1178–84. CrossRef
Custers EJFM. Thirty years of illness scripts: theoretical origins and practical applications. Med Teach. 2015;37:457–62. CrossRef
Lubarsky S, Dory V, Duggan P, Gagnon R, Charlin B. Script concordance testing: from theory to practice: AMEE guide no. 75. Med Teach. 2013;35:184–93. CrossRef
Norman GR. Objective measurement of clinical performance. Med Educ. 1985;19:43–7. CrossRef
Norcini JJ, Shea JA, Day SC. The use of the aggregate scoring for a recertification examination. Eval Health Prof. 1990;13:241–51. CrossRef
Charlin B, Brailovsky CA, Leduc C, Blouin D. The diagnosis script questionnaire: a new tool to assess a specific dimension of clinical competence. Adv Health Sci Educ Theory Pract. 1998;3:51–8. CrossRef
Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten C. Script concordance testing: a review of published validity evidence. Med Educ. 2011;45:329–38. CrossRef
Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166.e7–166.e16. CrossRef
Gagnon R, Charlin B, Roy L, et al. The cognitive validity of the script concordance test: a time processing study. Teach Learn Med. 2006;18:22–7. CrossRef
Ahmadi SF, Khoshkish S, Soltani-Arabshahi K. Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios? Int J Emerg Med. 2014;7:34. CrossRef
Kreiter C. Commentary: the response process validity of a script concordance item. Adv Health Sci Educ Theory Pract. 2011;17:7–9. CrossRef
Lineberry M, Kreiter CD, Bordage G. Threats to the validity in the use and interpretation of script concordance test scores. Med Educ. 2013;47:1175–83. CrossRef
Wilson AB, Pike GR, Humbert A. Preliminary factor analyses raise concerns about script concordance test utility. Med Sci Educ. 2014; https://doi.org/10.1007/s40670-014-0013-6.
Bland A, Kreiter C, Gordon J. The psychometric properties of five scoring methods applied to the Script Concordance Test. Acad Med. 2005;80:395–9. CrossRef
Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C. Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test? Med Educ. 2005;39:284–91. CrossRef
Lambert C, Gagnon R, Nguyen D, Charlin B. The script concordance test in radiation oncology: validation study of a new tool to assess clinical reasoning. Radiat Oncol. 2009;4:7. CrossRef
Lubarsky S, Chalk C, Kazitani D, Gagnon R, Charlin B. The Script Concordance Test: a new tool assessing clinical judgement in neurology. Can J Neurol Sci. 2009;36:326–31. CrossRef
Nouh T, Boutros M, Gagnon R, et al. The script concordance test as a measure of clinical reasoning: a national validation study. Am J Surg. 2012;203:530–4. CrossRef
Wilson AB, Pike GR, Humbert A. Analyzing script concordance test: scoring methods and items by difficulty and type. Teach Learn Med. 2014;26:135–45. CrossRef
Downing SM. Threats to the validity of locally developed multiple-choice tests in medical education: construct-irrelevant variance and construct underrepresentation. Adv Health Sci Educ Theory Pract. 2002;7:235–41. CrossRef
Williams RG, Klamen DA, McGaghie WC. Special article: cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003;15:270–92. CrossRef
See KC, Tan KL, Lim TK. The script concordance test for clinical reasoning: re-examining its utility and potential weakness. Med Educ. 2014;48:1069–77. CrossRef
Fournier JP, Demeester A, Charlin B. Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak. 2008;8:18. CrossRef
Friedman Ben-David M. Principles of assessment. In: Dent J, Harden RM, editors. A practical guide for medical teachers. 2nd ed. Edinburgh, Churchill, Livingstone: Elsevier; 2005.
Boulouffe C, Charlin B, Vanpee D. Evaluation of clinical reasoning in basic emergencies using a script concordance test. Am J Pharm Educ. 2010;74:1–6. CrossRef
Ramaekers S, Kremer W, Pilot A, van Keulen H. Assessment of competence in clinical reasoning and decision-making under uncertainty: the script concordance test method. Assess Eval High Educ. 2010;35:661–73. CrossRef
Dawson T, Comer L, Kossick MA, Neubrander J. Can script concordance testing be used in nursing education to accurately assess clinical reasoning skills? J Nurs Educ. 2014;53:281–6. CrossRef
Van den Broek WES, van Asperen MV, Custers EJFM, Valk GD, ten Cate O. Effects of two different instructional formats on scores and reliability of a script concordance test. Perspect Med Educ. 2012;1:119–28. CrossRef
- Examining the effects of gaming and guessing on script concordance test scores
- Bohn Stafleu van Loghum