Skip to main content

Advertisement

Log in

Exploring the role of first impressions in rater-based assessments

  • Review
  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

A Commentary to this article was published on 22 January 2014

Abstract

Medical education relies heavily on assessment formats that require raters to assess the competence and skills of learners. Unfortunately, there are often inconsistencies and variability in the scores raters assign. To ensure the scores from these assessment tools have validity, it is important to understand the underlying cognitive processes that raters use when judging the abilities of their learners. The goal of this paper, therefore, is to contribute to a better understanding of the cognitive processes used by raters. Representative findings from the social judgment and decision making, cognitive psychology, and educational measurement literature will be used to enlighten the underpinnings of these rater-based assessments. Of particular interest is the impact judgments referred to as first impressions (or thin slices) have on rater-based assessments. These are judgments about people made very quickly and based on very little information. A narrative review will provide a synthesis of research in these three literatures (social judgment and decision making, educational psychology, and cognitive psychology) and will focus on the underlying cognitive processes, the accuracy and the impact of first impressions on rater-based assessments. The application of these findings to the types of rater-based assessments used in medical education will then be reviewed. Gaps in understanding will be identified and suggested directions for future research studies will be discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • AERA, APA, & NCME. (1999). Standards for educational and psychological testing (pp. 9–24). Washington, DC: American Educational Research Association.

  • Ambady, N. (2010). The perils of pondering: Intuition and thin slice judgments. Psychological Inquiry, 21(4), 271–278.

    Article  Google Scholar 

  • Ambady, N., Bernieri, F., & Richeson, J. (2000). Toward a histology of social behavior: Judgmental accuracy from thin slices of the behavioral stream. Advances in Experimental Social Psychology, 32, 201–271.

    Article  Google Scholar 

  • Ambady, N., & Gray, H. M. (2002). On being sad and mistaken: Mood effects on the accuracy of thin-slice judgments. Journal of Personality and Social Psychology, 83(4), 947–961.

    Article  Google Scholar 

  • Ambady, N., Hallahan, M., & Conner, B. (1999). Accuracy of judgments of sexual orientation from thin slices of behavior. Journal of Personality and Social Psychology, 77(3), 538–547.

    Article  Google Scholar 

  • Ambady, N., Hallahan, M., & Rosenthal, R. (1995). On judging and being judged accurately in zero-acquaintance situations. Journal of Personality and Social Psychology, 69(3), 518–529.

    Article  Google Scholar 

  • Ambady, N., & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin, 111(2), 256–274.

    Article  Google Scholar 

  • Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64(3), 431–441.

    Article  Google Scholar 

  • Ames, D. R., Kammrath, L. K., Suppes, A., & Bolger, N. (2010). Not so fast: The (not-quite-complete) dissociation between accuracy and confidence in thin-slice impressions. Personality and Social Psychology Bulletin, 36(2), 264–277.

    Article  Google Scholar 

  • Babad, E., Avni-Babad, D., & Rosenthal, R. (2004). Prediction of students’ evaluations from brief instances of professors’ nonverbal behavior in defined instructional situations. Social Psychology of Education, 7(1), 3–33.

    Article  Google Scholar 

  • Balzer, W. K., & Sulsky, L. M. (1992). Halo and performance appraisal research: A critical examination. Journal of Applied Psychology, 77(6), 975–985.

    Article  Google Scholar 

  • Bargh, J. A. (1992). The ecology of automaticity: Toward establishing the conditions needed to produce automatic processing effects. The American Journal of Psychology, 105(2), 181–199.

    Article  Google Scholar 

  • Barrick, M. R., Shaffer, J. A., & DeGrassi, S. W. (2009). What you see may not be what you get: Relationships among self-presentation tactics and ratings of interview and job performance. The Journal of Applied Psychology, 94(6), 1394–1411.

    Article  Google Scholar 

  • Barrick, M. R., Swider, B. W., & Stewart, G. L. (2010). Initial evaluations in the interview: Relationships with subsequent interviewer evaluations and employment offers. The Journal of Applied Psychology, 95(6), 1163–1172.

    Article  Google Scholar 

  • Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors perspectives. Advances in Health Sciences Education: Theory and Practice. doi:10.1007/s10459-012-9392-x

  • Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65(1), 60–66.

    Article  Google Scholar 

  • Biesanz, J. C., Human, L. J., Paquin, A. C., Chan, M., Parisotto, K. L., Sarracino, J., et al. (2011). Do we know when our impressions of others are valid? Evidence for realistic accuracy awareness in first impressions of personality. Social Psychological and Personality Science, 2(5), 452–459.

    Article  Google Scholar 

  • Borkenau, P., & Liebler, A. (1992). Trait inferences: Sources of validity at zero acquaintance. Journal of Personality and Social Psychology, 62(4), 645–657.

    Article  Google Scholar 

  • Brooks, L. R. (2005). The blossoms and the weeds. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Expérimentale, 59(1), 62–74.

    Article  Google Scholar 

  • Carney, D., Colvin, C., & Hall, J. (2007). A thin slice perspective on the accuracy of first impressions. Journal of Research in Personality, 41(5), 1054–1072.

    Article  Google Scholar 

  • Chan, M., Rogers, K. H., Parisotto, K. L., & Biesanz, J. C. (2011). Forming first impressions: The role of gender and normative accuracy in personality perception. Journal of Research in Personality, 45(1), 117–120.

    Article  Google Scholar 

  • Clauser, B. E., Margolis, M. J., & Swanson, D. B. (2008). Issues of validity and reliability for assessments in Medical Education. In E. S. Holmboe & R. E. Hawkins (Eds.), Practical guide to the evaluation of clinical competence (pp. 10–23). Philadelphia: Mosby Elsevier.

    Google Scholar 

  • Colvin, C. R., & Funder, D. C. (1991). Predicting personality and behavior: A boundary on the acquaintanceship effect. Journal of Personality and Social Psychology, 60(6), 884–894.

    Article  Google Scholar 

  • Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166.e7–166.e16.

    Article  Google Scholar 

  • Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. A. (2008). Effect of rater training on reliability and accuracy of mini-cex scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79.

    Article  Google Scholar 

  • Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218–244.

    Article  Google Scholar 

  • Croskerry, P. (2009). Clinical cognition and diagnostic error: Applications of a dual process model of reasoning. Advances in Health Sciences Education, 14, 27–35.

    Article  Google Scholar 

  • DeNisi, A. S., Cafferty, T. P., & Meglino, B. M. (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behaviour and Human Performance, 33, 360–396.

    Article  Google Scholar 

  • Dijksterhuis, A., Bos, M. W., Nordgren, L. F., & Van Baaren, R. B. (2006). On making the right choice: The deliberation-without-attention effect. Science, 311(5763), 1005–1007.

    Article  Google Scholar 

  • Dipboye, R. L. (1982). Self-fulfilling prophecies in the selection-recruitment interview. The Academy of Management Review, 7(4), 579.

    Google Scholar 

  • Dodson, M., Crotty, B., Prideaux, D., Carne, R., Ward, A., & De Leeuw, E. (2009). The multiple mini-interview: How long is long enough? Medical Education, 43(2), 168–174.

    Article  Google Scholar 

  • Dougherty, T. W., Turban, D. B., & Callender, J. C. (1994). Confirming first impressions in the employment interview: A field study of interview behaviour. Journal of Applied Psychology, 5(5), 659–665.

    Article  Google Scholar 

  • Downing, S. M., & Haladyna, T. M. (2009). Validity and its threats. In S. M. Downing & R. Yudkowsky (Eds.), Assessment in health professions education (pp. 21–56). New York: Routledge.

    Google Scholar 

  • Eva, K. W., & Norman, G. R. (2005). Heuristics and biases—A biased perspective on clinical reasoning. Medical Education, 39(9), 870–872.

    Article  Google Scholar 

  • Eva, K. W., & Regehr, G. (2011). Exploring the divergence between self-assessment and self-monitoring. Advances in Health Sciences Education: Theory and Practice, 16(3), 311–329.

    Article  Google Scholar 

  • Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278.

    Article  Google Scholar 

  • Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Applied Psychology, 66(2), 127–148.

    Article  Google Scholar 

  • Fisicaro, S. A., & Lance, C. E. (1990). Implications of three causal models for the measurement of halo error. Applied Psychological Measurement, 14(4), 419–429.

    Article  Google Scholar 

  • Fiske, S., & Neuberg, S. (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation. In M. Zanna (Ed.), Advances in experimental social psychology (23rd ed., pp. 1–75). San Diego: Academic Press Inc.

    Google Scholar 

  • Funder, D. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgment. Psychological Bulletin, 101(1), 75–90.

    Article  Google Scholar 

  • Funder, D. C., & West, S. G. (1993). Consensus, self-other agreement, and accuracy in personality judgment: A introduction. Journal of Personality, 61(4), 457–476.

    Article  Google Scholar 

  • Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62, 451–482.

    Article  Google Scholar 

  • Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10), S1–S7.

    Article  Google Scholar 

  • Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.

    Article  Google Scholar 

  • Goffin, R. D., Jelley, R. B., & Wagner, S. H. (2003). Is halo helpful? Effects of inducing halo on performance rating accuracy. Social Behaviour and Personality, 31(6), 625–636.

    Article  Google Scholar 

  • Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education: Theory and Practice, 16(2), 151–165.

    Article  Google Scholar 

  • Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education: Theory and Practice. doi:10.1007/s10459-012-9376-x.

  • Harris, M., & Garris, C. (2008). You never get a second chance to make a first impression. In N. Ambady & J. Skowronski (Eds.), First impressions (pp. 147–168). New York, NY: Guilford Press.

    Google Scholar 

  • Hasher, L., & Zacks, R. T. (1979). Automatic and effortful processes in memory. Journal of Experimental Psychology: General, 108(3), 356–388.

    Article  Google Scholar 

  • Hawkins, R. E., & Boulet, J. R. (2008). Direct observation: Standardized patients. In E. S. Holmboe & R. E. Hawkins (Eds.), Evaluation of clinical competence (pp. 102–118). Philadelphia, PA: Mosby Elsevier.

    Google Scholar 

  • Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682.

    Article  Google Scholar 

  • Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5(1), 64–65.

    Article  Google Scholar 

  • Iramaneerat, C., & Yudkowsky, R. (2007). Rater errors in a clinical skills assessment of medical students. Evaluation and the Health Professions, 30(3), 266–283.

    Article  Google Scholar 

  • Jacoby, L. L. (1991). A process dissociation framework: Separating automatic from intentional uses of memory. Journal of Memory and Language, 30, 513–541.

    Article  Google Scholar 

  • Jacoby, L., & Kelley, C. (1990). An episodic view of motivation: Unconscious influences of memory. In E. T. Higgins & R. M. Sorrentino (Eds.), Handbook of motivation and cognition: Foundations of social behavior (Vol. 2, pp. 451–480). New York, NY: Guilford Press.

    Google Scholar 

  • Johnston, J. H., Driskell, J. E., & Salas, E. (1997). Vigilant and hypervigilant decision making. The Journal of Applied Psychology, 82(4), 614–622.

    Article  Google Scholar 

  • Kahneman, D. (2011). Thinking, fast and slow. Canada: Doubleday.

    Google Scholar 

  • Kenny, D. A. (1993). A coming-of-age for research on interpersonal perception. Journal of Personality, 61(4), 789–807.

    Article  Google Scholar 

  • Kenny, D. A., & Albright, L. (1987). Accuracy in interpersonal perception: A social relations analysis. Psychological Bulletin, 102(3), 390–402.

    Article  Google Scholar 

  • Klein, G. (2009). Streetlights and shadows: Searching for the keys to adaptive decision making. Cambridge, MA: MIT Press.

    Google Scholar 

  • Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.

    Article  Google Scholar 

  • Lance, C. E., LaPointe, J. A., & Stewart, A. M. (1994). A test of the context dependency of three causal models of halo rater error. Journal of Applied Psychology, 79(3), 332–340.

    Article  Google Scholar 

  • Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72–107.

    Article  Google Scholar 

  • Lippa, R. A., & Dietz, J. K. (2000). The relation of gender, personality and intelligence to judges’ accuracy in judging strangers’ personality from brief video segments. Journal of Nonverbal Behavior, 24(1), 25–43.

    Article  Google Scholar 

  • Logan, G. D. (1992). Attention and preattention in theories of automaticity. The American Journal of Psychology, 105(2), 317–339.

    Article  Google Scholar 

  • Macan, T. H., & Dipboye, R. L. (1990). The relationship of interviewrs’ preinterview impressions to selection and recruitment outcomes. Personnel Psychology, 43(4), 745–768.

    Article  Google Scholar 

  • Monteiro, S. D., Sherbino J. D., Ilgen, J. S., Dore, K. L. Gaissmaier, W., Wood, T. J., et al. (unpublished manuscript). Diagnosing Fast and Slow: The Effect of Interruptions on Speeded and Reflective Clinical Reasoning.

  • Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78(2), 218–225.

    Article  Google Scholar 

  • Nathan, B. R., & Tippins, N. (1990). The consequences of halo “error” in performance ratings: A field study of the moderating effect of halo on test validation results. Journal of Applied Psychology, 75(3), 290–296.

    Article  Google Scholar 

  • Norman, G. (2009). Dual processing and diagnostic errors. Advances in Health Sciences Education, 14, 37–49.

    Article  Google Scholar 

  • Norman, G. R., & Eva, K. W. (2010). Diagnostic error and clinical reasoning. Medical Education, 44, 94–100.

    Article  Google Scholar 

  • Norman, G. R., Sherbino, J., Dore, K. L., Wood, T. J. Ph. Young, M. E., Gaissmaier, W., et al. (in press). The etiology of diagnostic errors: A controlled trial of System 1 vs. System 2 reasoning. Academic Medicine.

  • Norman, G., Young, M., & Brooks, L. (2007). Non-analytical models of clinical reasoning: The role of experience. Medical Education, 41, 1140–1145.

    Google Scholar 

  • Patterson, M. L., & Stockbridge, E. (1998). Effects of cognitive demand and judgment strategy on person perception accuracy. Journal of Nonverbal Behavior, 22(4), 253–263.

    Article  Google Scholar 

  • Pelaccia, T., Tardif, J., Triby, E., & Charlin, B. (2011). An analysis of clinical reasoning through a recent and comprehensive approach: The dual-process theory. Medical Education Online, 16, 5890.

    Article  Google Scholar 

  • Rosenthal, R. (1994). Interpersonal expectancy effects : A 30-year perspective. Current Directions in Psychological Science, 3(6), 176–179.

    Article  Google Scholar 

  • Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413–428.

    Article  Google Scholar 

  • Schneider, W., & Chein, J. M. (2003). Controlled & automatic processing: Behavior, theory, and biological mechanisms. Cognitive Science, 27(3), 525–559.

    Article  Google Scholar 

  • Sherbino, J., Dore, K. L., Wood, T. J., Young, M. E., Gaissmaier, W., Krueger, S., et al. (2012). On the relation between processing speed and diagnostic error. Academic Medicine, 87(6), 785–791.

    Article  Google Scholar 

  • Smith, H. J., Archer, D., & Costanzo, M. (1991). “Just a hunch”: Accuracy and awareness in person perception. Journal of Nonverbal Behavior, 15(1), 3–18.

    Article  Google Scholar 

  • Snyder, M., Tanke, E., & Berscheid, E. (1977). Social perception and interpersonal behavior : On the self-fulfilling nature of social stereotypes. Journal of Personality and Social Psychology, 35(9), 656–666.

    Article  Google Scholar 

  • Stroud, L., Herold, J., Tomlinson, G., & Cavalcanti, R. B. (2011). Who you know or what you know? Effect of examiner familiarity with residents on OSCE scores. Academic Medicine, 86(10), S8–S11.

    Article  Google Scholar 

  • Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education: Theory and Practice. doi:10.1007/s10459-012-9370-3.

  • Tom, G., Tong, S. T., & Hesse, C. (2009). Thick slice and thin slice teaching evaluations. Social Psychology of Education, 13(1), 129–136.

    Article  Google Scholar 

  • Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.

    Article  Google Scholar 

  • Uleman, J. S., Saribay, S. A., & Gonzalez, C. M. (2008). Spontaneous inferences, implicit impressions, and implicit theories. Annual Review of Psychology, 59, 329–360.

    Article  Google Scholar 

  • Van der Vleuten, C. P. M., & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine, 2(2), 58–76.

    Article  Google Scholar 

  • Van Merriënboer, J. J. G., & Sweller, J. (2010). Cognitive load theory in health professional education: Design principles and strategies. Medical Education, 44(1), 85–93.

    Article  Google Scholar 

  • Wigton, R. (1980). The effects of student personal characteristics on the evaluation of clinical performance. Journal of Medical Education, 55, 423–427.

    Google Scholar 

  • Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.

    Article  Google Scholar 

  • Willis, J., & Todorov, A. (2006). Making up your mind after a 100-ms exposure to a face. Psychological Science, 17(7), 592–598.

    Article  Google Scholar 

  • Wilson, T. D., & Schooler, J. W. (1991). Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60(2), 181–192.

    Article  Google Scholar 

  • Woehr, D. J., Day, D. V., Winfred, A., & Bedeian, A. G. (1998). The systematic distortion hypothesis: A confirmatory test of the implicit covariance and general impression models. Basic and Applied Social Psychology, 16(4), 417–434.

    Article  Google Scholar 

  • Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189–205.

    Article  Google Scholar 

  • Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education: Theory and Practice. doi:10.1007/s10459-012-9396-6

  • Yaphe, J., & Street, S. (2003). How do examiners decide?: A qualitative study of the process of decision making in the oral examination component of the MRCGP examination. Medical Education, 37(9), 764–771.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently. Advances in Health Sciences Education: Theory and Practice. doi:10.1007/s10459-012-9372-1.

Download references

Acknowledgments

The author would like to thank Dr. Andrea Gingerich and Dr Sydney Smee for their helpful discussions that led to this project and Dr. Stan Hamstra, Dr. Susan Humprey-Murto, Kulamakan Mahan Kulasegaram, Sarah Lynch, Dr. Debra Pugh, Dr. Nikki Woods and the journal editor for their thoughtful review and suggestions regarding this manuscript.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Timothy J. Wood.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wood, T.J. Exploring the role of first impressions in rater-based assessments. Adv in Health Sci Educ 19, 409–427 (2014). https://doi.org/10.1007/s10459-013-9453-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-013-9453-9

Keywords

Navigation