Skip to main content
SpringerLink
Log in

Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

In general, researchers attempt to quantify cognitive load using physiologic and psychometric measures. Although the construct measured by both of these metrics is thought to represent overall cognitive load, there is a paucity of studies that compares these techniques to one another. The authors compared data obtained from one physiologic tool (pupillometry) to one psychometric tool (Paas scale) to explore whether they actually measured the construct of cognitive load as purported. Thirty-two participants with a range of resuscitation medicine experience and expertise completed resuscitation-medicine based multiple-choice-questions as well as arithmetic questions. Cognitive load, as measured by both tools, was found to be higher for the more difficult questions as well as for questions that were answered incorrectly (p < 0.001). The group with the least medical experience had higher cognitive load than both the intermediate and experienced groups when answering domain-specific questions (p = 0.023 and p = 0.003 respectively for the physiologic tool; p = 0.006 and p < 0.001 respectively for the psychometric tool). There was a strong positive correlation (Spearman’s ρ = 0.827, p < 0.001 for arithmetic questions; Spearman’s ρ = 0.606, p < 0.001 for medical questions) between the two cognitive load measurement tools. These findings support the validity argument that both physiologic and psychometric metrics measure the construct of cognitive load.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Ayres, P. (2006). Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and Instruction, 16(5), 389–400.

    Article  Google Scholar 

  • Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276–292.

    Article  Google Scholar 

  • Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38(1), 53–61.

    Article  Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: L: Erlbaum.

    Google Scholar 

  • Cook, D. A. (2015). Much ado about differences: Why expert-novice comparisons add little to the validity argument. Advances in Health Sciences Education, 20(3), 829–834.

    Article  Google Scholar 

  • Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166.e7–166.e16.

    Article  Google Scholar 

  • De Jong, T. (2010). Cognitive load theory, educational research, and instructional design: Some food for thought. Instructional Science, 38(2), 105–134.

    Article  Google Scholar 

  • Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37(9), 830–837.

    Article  Google Scholar 

  • Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102(2), 211.

    Article  Google Scholar 

  • Ericsson, K. A., Prietula, M. J., & Cokely, E. T. (2007). The making of an expert. Harvard Business Review, 85(7/8), 114.

    Google Scholar 

  • Gegenfurtner, A., Kok, E., Van Geel, K., De Bruin, A., Jarodzka, H., Szulewski, A., & Van Merriënboer, J. J. G. (in press). The challenges of studying visual expertise in medical image diagnosis. Medical Education.

  • Gegenfurtner, A., Lehtinen, E., & Säljö, R. (2011). Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational Psychology Review, 23(4), 523–552.

    Article  Google Scholar 

  • Gegenfurtner, A., & Seppänen, M. (2013). Transfer of expertise: An eye tracking and think aloud study using dynamic medical visualizations. Computers and Education, 63, 393–403.

    Article  Google Scholar 

  • Gegenfurtner, A., Siewiorek, A., Lehtinen, E., & Säljö, R. (2013). Assessing the quality of expertise differences in the comprehension of medical visualizations. Vocations and Learning, 6(1), 37–54.

    Article  Google Scholar 

  • Gegenfurtner, A., & Szulewski, A. (2016). Visual expertise and the Quiet Eye in sports – comment on Vickers. Current Issues in Sport Science, 1, 108. doi:10.15203/CISS_2016.108.

    Google Scholar 

  • Hess, E. H. (1965). Attitude and pupil size. Scientific American, 212, 46–54.

    Article  Google Scholar 

  • Hess, E. H., & Polt, J. M. (1964). Pupil size in relation to mental activity during simple problem-solving. Science, 143(3611), 1190–1192.

    Article  Google Scholar 

  • Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154(3756), 1583–1585.

    Article  Google Scholar 

  • Klingner, J., Kumar, R., & Hanrahan, P. (2008). Measuring the task-evoked pupillary response with a remote eye tracker. Paper presented at the Proceedings of the 2008 symposium on Eye tracking research and applications.

  • Klingner, J., Tversky, B., & Hanrahan, P. (2011). Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology, 48(3), 323–332.

    Article  Google Scholar 

  • Kok, E. M., Bruin, A. B., Robben, S. G., & Merriënboer, J. J. (2012). Looking in the same manner but seeing it differently: Bottom-up and expertise effects in radiology. Applied Cognitive Psychology, 26(6), 854–862.

    Article  Google Scholar 

  • Laeng, B., Sirois, S., & Gredebäck, G. (2012). Pupillometry a window to the preconscious? Perspectives on Psychological Science, 7(1), 18–27.

    Article  Google Scholar 

  • Laxmisan, A., Hakimzada, F., Sayan, O. R., Green, R. A., Zhang, J., & Patel, V. L. (2007). The multitasking clinician: Decision-making and cognitive demand during and after team handoffs in emergency care. International Journal of Medical Informatics, 76(11), 801–811.

    Article  Google Scholar 

  • Leppink, J., Paas, F., van Gog, T., van der Vleuten, C. P., & van Merriënboer, J. J. (2014). Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction, 30, 32–42.

    Article  Google Scholar 

  • Naismith, L. M., & Cavalcanti, R. B. (2015). Validity of cognitive load measures in simulation-based training: A systematic review. Academic Medicine, 90(11), S24–S35.

    Article  Google Scholar 

  • Naismith, L. M., Cheung, J. J., Ringsted, C., & Cavalcanti, R. B. (2015). Limitations of subjective cognitive load measures in simulation-based procedural training. Medical Education, 49(8), 805–814.

    Article  Google Scholar 

  • Norman, G. (2005). Research in clinical reasoning: Past history and current trends. Medical Education, 39(4), 418–427.

    Article  Google Scholar 

  • Paas, F. G. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429.

    Article  Google Scholar 

  • Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38(1), 63–71.

    Article  Google Scholar 

  • Perry, S. J., Wears, R. L., Croskerry, P., & Shapiro, M. J. (2013). Process Improvement and Patient Safety. In J. Marx, R. Walls & R. Hockberger (Eds.), Rosen's Emergency Medicine - Concepts and Clinical Practice (8 Edn., Vol. 2, pp. 2505–2511). Philadelphia: Elsevier Health Sciences.

    Google Scholar 

  • Schubert, C. C., Denmark, T. K., Crandall, B., Grome, A., & Pappas, J. (2013). Characterizing novice-expert differences in macrocognition: An exploratory study of cognitive work in the emergency department. Annals of Emergency Medicine, 61(1), 96–109.

    Article  Google Scholar 

  • Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Psychology Review, 22(2), 123–138.

    Article  Google Scholar 

  • Sweller, J., Van Merrienboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296.

    Article  Google Scholar 

  • Szulewski, A., Fernando, S. M., Baylis, J., & Howes, D. (2014). Increasing pupil size is associated with increasing cognitive processing demands: A pilot study using a mobile eye-tracking device. Open Journal of Emergency Medicine, 2(1), 8–11.

    Article  Google Scholar 

  • Szulewski, A., Roth, N., & Howes, D. (2015). The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: A new tool for the assessment of expertise. Academic Medicine, 90(7), 981–987.

    Article  Google Scholar 

  • Tuovinen, J., & Paas, F. (2004). Exploring multidimensional approaches to the efficiency of instructional conditions. Instructional Science, 32(1–2), 133–152. doi:10.1023/B:TRUC.0000021813.24669.62.

    Article  Google Scholar 

  • Young, J. Q., Van Merrienboer, J., Durning, S., & Ten Cate, O. (2014). Cognitive load theory: Implications for medical education: AMEE guide no. 86. Medical Teacher, 36(5), 371–384.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Wilma Hopman for assistance with statistical analysis, Bence Linder for development and implementation of the algorithm to smooth the raw pupillometry data and to calculate peak pupillary size, as well as Dr. Jimmie Leppink for advice about experimental design. The authors would also like to acknowledge the Kingston Resuscitation Institute for providing access to the pupillometry device and research assistants.

Author information

Authors and Affiliations

  1. Department of Emergency Medicine, Queen’s University, Kingston General Hospital, 76 Stuart Street, Kingston, ON, K7L 2V7, Canada

    Adam Szulewski

  2. Institut für Qualität und Weiterbildung, Technische Hochschule Deggendorf, Edlmairstraße 9, 94469, Deggendorf, Germany

    Andreas Gegenfurtner

  3. Departments of Emergency Medicine and Critical Care, Queen’s University, Kingston General Hospital, 76 Stuart Street, Kingston, ON, K7L 2V7, Canada

    Daniel W. Howes

  4. Departments of Emergency Medicine and Biomedical and Molecular Sciences, Kingston General Hospital, 76 Stuart Street, Kingston, ON, K7L 2V7, Canada

    Marco L. A. Sivilotti

  5. Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, Room N.5.06, 6200 MD, Maastricht, The Netherlands

    Jeroen J. G. van Merriënboer

Authors
  1. Adam Szulewski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Gegenfurtner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel W. Howes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marco L. A. Sivilotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jeroen J. G. van Merriënboer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel W. Howes.

Appendices

Appendix 1

Psychometric survey used in the study, adapted from Paas (1992).

figure a

Appendix 2

Example of raw pupillometry data obtained from one experienced participant for one medical question. The first arrow represents the time the question appeared on the screen. The second arrow represents the point at which the participant verbalized his answer. As the participant experiences increasing cognitive load during the thought process, the pupil diameter increases in size. When the participant verbalizes the answer to the question, pupil size decreases again. The quantitative cognitive load measurement used in the pupillometry arm of this study can be conceptualized as the area under the curve between these two arrows (referred to as pupillary change index in this manuscript).

figure b

Appendix 3

Data distribution for Paas and Pupillary Change Index scales:

 

Paas

PCI

Valid data points

232

222

Missing data points

0

10

Minimum value

1

2.9

Maximum value

9

597.2

Mean

4.7

92.7

Standard deviation

1.9

95.5

25th percentile

3

27.7

Median

5

59.4

75th percentile

6

124.2

Boxplots of the data distribution of the Paas and Pupillary Change Index scales showing outliers:

figure c

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Szulewski, A., Gegenfurtner, A., Howes, D.W. et al. Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. Adv in Health Sci Educ 22, 951–968 (2017). https://doi.org/10.1007/s10459-016-9725-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-016-9725-2

Keywords

Search

Navigation