Skip to main content
Log in

Composite reliability of a workplace-based assessment toolbox for postgraduate medical education

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

In recent years, postgraduate assessment programmes around the world have embraced workplace-based assessment (WBA) and its related tools. Despite their widespread use, results of studies on the validity and reliability of these tools have been variable. Although in many countries decisions about residents’ continuation of training and certification as a specialist are based on the composite results of different WBAs collected in a portfolio, to our knowledge, the reliability of such a WBA toolbox has never been investigated. Using generalisability theory, we analysed the separate and composite reliability of three WBA tools [mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), and multisource feedback (MSF)] included in a resident portfolio. G-studies and D-studies of 12,779 WBAs from a total of 953 residents showed that a reliability coefficient of 0.80 was obtained for eight mini-CEXs, nine DOPS, and nine MSF rounds, whilst the same reliability was found for seven mini-CEXs, eight DOPS, and one MSF when combined in a portfolio. At the end of the first year of residency a portfolio with five mini-CEXs, six DOPS, and one MSF afforded reliable judgement. The results support the conclusion that several WBA tools combined in a portfolio can be a feasible and reliable method for high-stakes judgements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ahmed, K., Miskovic, D., Darzi, A., Athanasiou, T., & Hanna, G. B. (2011). Observational tools for assessment of procedural skills: a systematic review. American Journal of Surgery, 202, 469–480.

    Article  Google Scholar 

  • Barton, J. R., Corbett, S., & van der Vleuten, C. P. M. (2012). The validity and reliability of a direct observation of procedural skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointestinal Endoscopy, 75, 591–597.

    Article  Google Scholar 

  • Brennan, R. L. (1983). Elements of generalizability theory. Iowa: American College Testing Program.

    Google Scholar 

  • Brennan, R. L. (2001). Generalizability theory. New York: Springer.

    Book  Google Scholar 

  • Crossley, J., Davies, H., Humphris, B., & Jolly, G. (2002). Generalisability: A key to unlock professional assessment. Medical Education, 36, 972–978.

    Article  Google Scholar 

  • Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Medical Education, 45, 560–569.

    Article  Google Scholar 

  • Crossley, J., & Jolly, B. (2012). Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46, 28–37.

    Article  Google Scholar 

  • Dijksterhuis, M. G., Voorhuis, M., Teunissen, P. W., Schuwirth, L. W., ten Cate, O. W., Braat, D. D., et al. (2009). Assessment of competence and progressive independence in postgraduate clinical training. Medical Education, 43, 1156–1165.

    Article  Google Scholar 

  • Donato, A. A., & George, D. L. (2012). A blueprint for implementation of a structured portfolio in an internal medicine residency. Academic Medicine, 87, 185–191.

    Article  Google Scholar 

  • Driessen, E. W., van Tartwijk, J., Teunissen, P. W., Govaerts, M., & van der Vleuten, C. P. M. (2012). The use of programmatic assessment in the clinical workplace: A Maastricht case report. Medical Teacher, 34, 226–231.

    Article  Google Scholar 

  • Hays, R. B., Fabb, W. E., & Van der Vleuten, C. P. M. (1995). Reliability of the fellowship examination of the royal Australian college of general practitioners. Teaching and Learning in Medicine, 7, 43–50.

    Article  Google Scholar 

  • Kogan, J. R., Holmboe, E. S., & Hauer, K. S. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. Journal of the American Medical Association, 302, 1316–1326.

    Article  Google Scholar 

  • McGill, D. A., van der Vleuten, C. P. M., & Clarke, M. J. (2011). Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review. Advances in health sciences education theory and practice, 16, 405–425.

    Article  Google Scholar 

  • Miller, A., & Archer, J. (2010). Impact of workplace based assessment on doctors’ education and performance: A systematic review. British Medical Journal, 341, c5064. doi:10.1136/bmj.c5064.

    Article  Google Scholar 

  • Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138, 476–481.

    Article  Google Scholar 

  • Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE guide no. 31. Medical Teacher, 29, 855–871.

    Article  Google Scholar 

  • Overeem, K., Lombarts, M. J. M. H., Arah, O. A., Klazinga, N. S., Grol, R. P. T. M., & Wollersheim, H. C. (2010). Three methods of multi-source feedback compared: A plea for narrative comments and coworkers’ perspectives. Medical Teacher, 32, 141–147.

    Article  Google Scholar 

  • Pelgrim, E. A., Kramer, A. W., Mokkink, H. G., van den Elsen, L., Grol, R. P. T. M., & van der Vleuten, C. P. M. (2011). In-training assessment using direct observation of single-patient encounters: A literature review. Advances in health sciences education theory and practice, 16, 131–142.

    Article  Google Scholar 

  • Scheele, F., Teunissen, P. W., Van Luijk, S. J., Heineman, E., Fluit, L., Mulder, H., et al. (2008). Introducing competency-based postgraduate medical education in the Netherlands. Medical Teacher, 30, 248–253.

    Article  Google Scholar 

  • Swanson, D. B. (1987). A measurement framework for performance-based tests. In I. Hart & R. Harden (Eds.), Further developments in assessing clinical competence (pp. 11–45). Montreal: Can-Heal publication.

    Google Scholar 

  • Ten Cate, O., & Scheele, F. (2007). Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Academic Medicine, 82, 542–547.

    Article  Google Scholar 

  • Tochel, C., Beggs, K., Haig, A., Roberts, J., Scott, H., Walker, K., et al. (2011). Use of web based systems to support postgraduate medical education. Postgraduate Medical Journal, 87, 800–806.

    Article  Google Scholar 

  • Van der Vleuten, C. P. M., Schuwirth, L. W., Scheele, F., Driessen, E. W., & Hodges, B. (2010). The assessment of professional competence: building blocks for theory development. Best Practice & Research. Clinical Obstetrics & Gynaecology, 24, 703–719.

    Article  Google Scholar 

  • Violato, C., Lockyer, J., & Fidler, H. (2003). Multisource feedback: A method of assessing surgical practice. British Medical Journal, 326, 546–548.

    Article  Google Scholar 

  • Weller, J. M., Jolly, B., Misur, M. P., Merry, A. F., Jones, A., Crossley, J. G., et al. (2009). Mini-clinical evaluation exercise in anaesthesia training. British Journal of Anaesthesia, 102, 633–641.

    Article  Google Scholar 

  • Wilkinson, J. R., Crossley, J. G., Wragg, A., Mills, P., Cowan, G., & Wade, W. (2008). Implementing workplace-based assessment across the medical specialties in the United Kingdom. Medical Education, 42, 364–373.

    Article  Google Scholar 

Download references

Acknowledgments

We thank Mereke Gorsira for correcting and editing the English language of this article.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to J. M. W. Moonen-van Loon.

Appendices

Appendix: Multivariable optimization

Maximizing reliability coefficient

In this appendix, the indices for DOPS, mini-CEX, and MSF are 1, 2, and 3, respectively. Let w 1 , w 2 , w 3 be the weights of the different WBAs used for calculating the composite universe and error scores. Let m be the index for the WBA, \( \sigma_{m}^{2} \left( p \right) \) be the variance for method m and \( \sigma_{{mm^{\prime } }} \left( p \right) \) the covariance for WBAs m and m′. Then the composite universe score variance is given by

$$ \sigma_{C}^{2} \left( p \right) = \mathop \sum \limits_{m} w_{m}^{2} \sigma_{m}^{2} \left( p \right) + \mathop \sum \limits_{m} \mathop \sum \limits_{{m^{'} \ne m}} w_{m} w_{{m^{'} }} \sigma_{{mm^{'} }} \left( p \right) $$
$$ \sigma_{C}^{2} \left( p \right) = w_{1}^{2} \sigma_{1}^{2} \left( p \right) + w_{2}^{2} \sigma_{2}^{2} \left( p \right) + w_{3}^{2} \sigma_{3}^{2} \left( p \right) + 2w_{1} w_{2} \sigma_{12} \left( p \right) + 2w_{2} w_{3} \sigma_{23} \left( p \right) + 2w_{1} w_{3} \sigma_{13} \left( p \right) $$
(1)

The composite error score is

$$ \sigma_{C}^{2} \left( \Updelta \right) = \mathop \sum \limits_{m} w_{m}^{2} \sigma_{m}^{2} (\Updelta ) = w_{1}^{2} \sigma_{1}^{2} \left( \Updelta \right) + w_{2}^{2} \sigma_{2}^{2} \left( \Updelta \right) + w_{3}^{2} \sigma_{3}^{2} \left( \Updelta \right) $$
(2)

where \( \sigma_{m}^{2} (\Updelta ) \) is equal to the absolute error. Using the above equations, the reliability coefficient is defined as

$$ E\varrho^{2} = \frac{{\sigma_{C}^{2} \left( p \right)}}{{\sigma_{C}^{2} \left( p \right) + \sigma_{C}^{2} \left( \Updelta \right)}} $$
(3)

As the sum of the weights is 1 and each weight is positive, these three equations can be rewritten using \( w_{3} = 1 - w_{1} - w_{2} , \) resulting in an equation with two variables, w 1 and w 2 By determining the partial derivative of this equation to w 1 and setting this to zero, the optimal value for w 1 can be found which is expressed in an equation with only w 2 as variable. This function for w 1 is included in the rewritten equation for the reliability coefficient, which can be optimised for w 2. Once the optimal value for w 2 is found, w 1 and w 3 can easily be determined. Entering these weights in Eq. 3 leads to the optimal reliability coefficient, given the variances, co-variances, and harmonic mean.

Minimising SEM

The SEM is the square root of the composite error score. Therefore, minimising SEM is similar to minimising the composite error score. As above, we first rewrite \( w_{3} = 1 - w_{1} - w_{2} \) in the formula of the composite error score and set the partial derivative thereof to w 2 equal to 0. This results in \( w_{2} = \frac{{1 - w_{1} }}{2} \), and consequently, w 3 = w 2. Then, replacing w 2 in the rewritten equation of the composite error score, the only variable is w 1. We can set the derivate to 0 and obtain \( w_{1} = \frac{{\sigma_{2}^{2} \left( \Updelta \right) + \sigma_{3}^{2} \left( \Updelta \right)}}{{4\sigma_{1}^{2} \left( \Updelta \right) + \sigma_{2}^{2} \left( \Updelta \right) + \sigma_{3}^{2} \left( \Updelta \right)}} \). Now, it is easy to obtain w 2 and w 3.

Appendix: WBA tools

This section contains the summative, competency-based, statements on the different assessment tools.

Mini-CEX

Medical expert

  1. 1.

    The (hetero) anamnesis is problem oriented, complete and systematic.

  2. 2.

    General and focused physical examination are problem oriented and complete.

  3. 3.

    General and focused physical examination are conscientious and executed smoothly.

  4. 4.

    Relevant findings are interpreted adequately.

  5. 5.

    Differential diagnosis is complete and relevant to the problem.

  6. 6.

    The proposed policy is tuned to the (situation of) the patient, justified and state-of-the-art.

  7. 7.

    Follow-up appointments are in line with the problem and are clear.

Communicator

  1. 8.

    Communication with patient/family* is empathic and attuned to the patient (active listening, consulting the patient).

  2. 9.

    Communication with patient/family* is problem oriented and effective.

  3. 10.

    Instructions and explanations to the patient/family are complete and checked with the people involved.

Manager

  1. 11.

    The right priorities are chosen, main and side issues are separated correctly.

  2. 12.

    Time is managed correctly.

Professional

  1. 13.

    The patient is treated with respect.

  2. 14.

    The resident’s behaviour during the patient contact creates confidence for the patient/family.

  3. 15.

    The resident is aware of his/her limitations and acts accordingly (e.g. adequately asks for supervision).

Health advocate

  1. 16.

    The resident acts according to the ethical and legal regulations concerning information and confidentiality (WGBO).

  2. 17.

    Follow-up examinations, therapy and guidance are started adequately and with awareness of costs.

Collaborator

  1. 18.

    Communication with colleagues/health professionals is efficient and effective

  2. 19.

    Referrals are adequate

  3. 20.

    Reports are complete and accurate.

Scholar

  1. 21.

    Choices concerning diagnostics, therapy and/or prevention are well-grounded and, if possible, evidence-based.

* If relevant read “caregiver(s)” instead of “family”.

DOPS

Medical expert

  1. 1.

    The resident treats tissues with care; there is minimal tissue damage.

  2. 2.

    The resident moves fluently with maximal efficiency.

  3. 3.

    The resident applies the instruments skillfully.

Communicator

  1. 4.

    The resident communicates adequately with the OR-team, before, during and after the surgery.

  2. 5.

    The resident communicates adequately with the patient, before and after the surgery.

Manager

  1. 6.

    The resident uses the available time efficiently; time management is effective.

  2. 7.

    The resident has a clear time schedule for the surgery and proceeds effortlessly from one step to the next.

Professional

  1. 8.

    The resident shows confidence and is decisive.

  2. 9.

    The resident is open to feedback.

  3. 10.

    The resident recognizes/acknowledges the limits of his/her knowledge and experience.

  4. 11.

    The resident treats the patient and other employees with respect.

Health advocate

  1. 12.

    The resident takes account of ethical and legal regulations (e.g. privacy).

Collaborator

  1. 13.

    The resident gives the assistant(s) adequate instructions.

  2. 14.

    The resident uses the assistance strategically and makes optimal use of it.

  3. 15.

    The resident appreciates the input and expertise of others and makes adequate use of this.

Scholar

  1. 16.

    The resident uses the appropriate terms for the instruments and applies the right instruments at the right moment.

  2. 17.

    The resident demonstrates a great deal of knowledge about the whole procedure.

MSF

Medical expert

  1. 1.

    Independently handles routine patient problems accurately and at an adequate pace.

  2. 2.

    Independently handles complex patient problems accurately and at an adequate pace.

  3. 3.

    Masters medical-technical skills/procedures and applies these adequately.

  4. 4.

    Pays sufficient attention to the psychosocial aspects of disease.

  5. 5.

    Acts in accordance with the current state of affairs in the field.

Communicator

  1. 6.

    Communicates effectively and respectfully with patients/family* (is empathic, clear and listens actively, discusses)

  2. 7.

    Is open to verbal and non-verbal reactions and emotions of others and responds adequately

  3. 8.

    Builds effective therapeutic relationships with patients/family*

Communicator–Collaborator

  1. 9.

    Communicates effectively and respectfully with colleagues (doctors).

  2. 10.

    Communicates effectively and respectfully with other colleagues (nursing staff, obstetricians, paramedic personnel, secretaries, etc.).

  3. 11.

    Is accurate, clear and complete in reporting/written communication (medical record documentation, letters, instructions).

Collaborator

  1. 12.

    Hands over the care for patients effectively as well as carefully.

  2. 13.

    Respects the input and expertise of others, and makes timely and adequately use of this.

  3. 14.

    Is a good colleague and positively contributes to the functioning of a team.

  4. 15.

    Can stimulate and motivate others.

Manager

  1. 16.

    Organises his/her work well. He/she sets the right priorities.

  2. 17.

    Coordinates and manages the care for patients adequately.

  3. 18.

    Is capable of keeping a good balance between work and home.

  4. 19.

    Is available and accessible.

Professional

  1. 20.

    Shows sufficient involvement with the patient and puts the patient’s interest first.

  2. 21.

    Respects the patient’s privacy

  3. 22.

    Is open to feedback and willing to admit mistakes.

  4. 23.

    Is aware of his/her own shortcomings and asks for assistance/supervision in time

  5. 24.

    Functions adequately under stress/time pressure

  6. 25.

    Shows self-confidence.

  7. 26.

    Gives adequate feedback to others.

  8. 27.

    Is reliable and keeps agreements.

Health advocate

  1. 28.

    Weighs costs and benefits for diagnostics, treatments and prevention.

  2. 29.

    Takes initiatives to improve quality in the health sector.

  3. 30.

    Acts according to legal and ethical guidelines and regulations with regard to education, information and privacy.

  4. 31.

    Is capable of involving the patient actively in improving his/her health.

Scholar

  1. 32.

    Takes a scientific approach and uses evidence-based medicine wherever possible.

  2. 33.

    Is willing to and capable of training/educating others.

  3. 34.

    Is capable of presenting clearly and concisely in front of a group (lecture, review of a clinical topic, handover, big round).

  4. 35.

    Is scientifically active.

  5. 36.

    Is aware of the gaps in his/her own knowledge/skills and makes a learning plan based on this.

* If relevant read “caregiver(s)” instead of “family”.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Moonen-van Loon, J.M.W., Overeem, K., Donkers, H.H.L.M. et al. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv in Health Sci Educ 18, 1087–1102 (2013). https://doi.org/10.1007/s10459-013-9450-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-013-9450-z

Keywords

Navigation