Skip to main content
main-content
Top

Tip

Swipe om te navigeren naar een ander artikel

23-10-2018 | Eye-Opener | Uitgave 6/2018 Open Access

Perspectives on Medical Education 6/2018

Analysis of question text properties for equality monitoring

Tijdschrift:
Perspectives on Medical Education > Uitgave 6/2018
Auteurs:
Daniel Zahra, Steven A. Burr

Abstract

Introduction

Ongoing monitoring of cohort demographic variation is an essential part of quality assurance in medical education assessments, yet the methods employed to explore possible underlying causes of demographic variation in performance are limited. Focussing on properties of the vignette text in single-best-answer multiple-choice questions (MCQs), we explore here the viability of conducting analyses of text properties and their relationship to candidate performance. We suggest that such analyses could become routine parts of assessment evaluation and provide an additional, equality-based measure of an assessment’s quality and fairness.

Methods

We describe how a corpus of vignettes can be compiled, followed by examples of using Microsoft Word’s native readability statistics calculator and the koRpus text analysis package for the R statistical analysis environment for estimating the following properties of the question text: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (Grade), word count, sentence count, and average words per sentence (WpS). We then provide examples of how these properties can be combined with equality and diversity variables, and the process automated to provide ongoing monitoring.

Conclusions

Given the monitoring of demographic differences in assessment for assurance of equality, the ability to easily include textual analysis of question vignettes provides a useful tool for exploring possible causes of demographic variations in performance where they occur. It also provides another means of evaluating assessment quality and fairness with respect to demographic characteristics. Microsoft Word provides data comparable to the specialized koRpus package, suggesting routine use of word processing software for writing items and assessing their properties is viable with minimal burden, but that automation for ongoing monitoring also provides an additional means of standardizing MCQ assessment items, and eliminating or controlling textual variables as a possible contributor to differential attainment between subgroups.
Literatuur
Over dit artikel

Andere artikelen Uitgave 6/2018

Perspectives on Medical Education 6/2018 Naar de uitgave