Skip to main content

Response Processes and Validity Evidence: Controlling for Emotions in Think Aloud Interviews

  • Chapter
  • First Online:
Understanding and Investigating Response Processes in Validation Research

Part of the book series: Social Indicators Research Series ((SINS,volume 69))

Abstract

The objective of the chapter is to highlight the potentially problematic effects that emotional variables and, in particular, evaluative anxiety may have on response processing data collected using think-aloud interviews. Given the collection of response processing data to inform the nature of processing skills that test-takers use to respond to test items, and related validity arguments, it is imperative that the integrity of these data be considered. Recent research illustrating how distractions and disruptions, in some cases emotionally-induced, can influence response processing data in think-aloud interviews is presented. Methods to control and minimize disruptive emotions for participants in think-aloud interviews are also considered, as well as areas for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, & NCME). (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Beilock, S. L., & Carr, T. H. (2001). On the fragility of skilled performance: What governs choking under pressure? Journal of Experimental Psychology: General, 130, 701–725. doi:10.1037/0096-3445.130.4.701.

    Article  Google Scholar 

  • Beilock, S. L., Kulp, C. A., Holt, L. E., & Carr, T. H. (2004). More on the fragility of performance: Choking under pressure in mathematical problem solving. Journal of Experimental Psychology: General, 133, 584–600. doi:10.1037/0096-3445.133.4.584.

    Article  Google Scholar 

  • Ben-Zeev, T., Fein, S., & Inzlicht, M. (2005). Arousal and stereotype threat. Journal of Experimental Social Psychology, 41, 174–181. doi:10.1016/j.jesp.2003.11.007.

    Article  Google Scholar 

  • Berridge, K. C., & Winkielman, P. (2003). What is an unconscious emotion? (The case for unconscious “liking”). Cognition and Emotion, 17, 181–211. doi:10.1080/02699930244000273.

    Article  Google Scholar 

  • Beukeboom, C. J., & Semin, G. R. (2006). How mood turns on language. Journal of Experimental Social Psychology, 42, 553–566. doi:10.1016/j.jesp.2005.09.005.

    Article  Google Scholar 

  • Bishop, S. J. (2007). Neurocognitive mechanisms of anxiety: An integrative account. Trends in Cognitive Sciences, 11, 307–316. doi:10.1016/j.tics.2007.05.008.

    Article  Google Scholar 

  • Bosson, J. K., Haymovitz, E. L., & Pinel, E. C. (2004). When saying and doing diverge: The effects of stereotype threat on self-reported versus non-verbal anxiety. Journal of Experimental Social Psychology, 40, 247–255. doi:10.1016/S0022-1031(03)00099-4.

    Article  Google Scholar 

  • Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37.

    Google Scholar 

  • Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21, 1363–1368. doi:10.1177/0956797610383437.

    Article  Google Scholar 

  • Carver, C. S., Peterson, L. M., Follansbee, D. J., & Scheier, M. F. (1983). Effects of self-directed attention on performance and persistence among persons high and low in test anxiety. Cognitive Therapy and Research, 7, 333–353. doi:10.1007/BF01177556.

    Article  Google Scholar 

  • Cassady, J. C., & Johnson, R. E. (2002). Cognitive test anxiety and academic performance. Contemporary Educational Psychology, 27, 270–295. doi:10.1006/ceps.2001.1094.

    Article  Google Scholar 

  • Chu, M.-W., Guo, Q., & Leighton, J. P. (2014). Students’ interpersonal trust and attitudes towards standardized tests: Exploring affective variables related to student assessment. Assessment in Education: Principles, Policy & Practice, 21, 167–192. doi:10.1080/0969594X.2013.844094.

    Article  Google Scholar 

  • Chu, M.-W., & Leighton, J. P. (2016). Using errors to enhance learning feedback in computer programming. In S. Tettegah & M. McCreery (Eds.), Emotions and technology: Communication of feelings for, with and through digital media – Volume I: Emotions, learning, and technology (pp. 89–117). Elsevier Publishing.

    Google Scholar 

  • Cizek, G. J., & Burg, S. S. (2006). Addressing test anxiety in a high-stakes environment: Strategies for classrooms and schools. Thousand Oaks, CA: Corwin Press.

    Google Scholar 

  • Cohen, G. L., & Sherman, D. K. (2005). Stereotype threat and the social and scientific contexts of the race achievement gap. American Psychologist, 60, 270–271. doi:10.1037/0003-066X.60.3.270.

    Article  Google Scholar 

  • Cuddy, A. J., Wilmuth, C. A., Yap, A. J., & Carney, D. R. (2015). Preparatory power posing affects nonverbal presence and job interview performance. Journal of Applied Psychology, 100, 1286–1295. doi:10.1037/a0038543.

    Article  Google Scholar 

  • D’Arcey, T., Johnson, M., & Ennis, M. (2012). Assessing the validity of FaceReader using facial electromyography. In Proceedings of APS 24th annual meeting. Retrieved from www.darcey.us/pdf/facereader.pdf

  • D’Mello, S., & Graesser, A. (2009). Automatic detection of learner’s affect from gross body language. Applied Artificial Intelligence, 23, 123–150. doi:10.1080/08839510802631745.

    Article  Google Scholar 

  • DeBellis, V. A., & Goldin, G. A. (2006). Affect and meta-affect in mathematical problem solving: A representational perspective. Educational Studies in Mathematics, 63, 131–147. doi:10.1007/s10649-006-9026-4.

    Article  Google Scholar 

  • Ekman, P. (1972). Universals and cultural differences in facial expressions of emotion. In J. Cole (Ed.), Nebraska symposium on motivation (Vol. 19, pp. 207–283). Lincoln, NE: University of Nebraska Press.

    Google Scholar 

  • Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200. doi:10.1080/02699939208411068.

    Article  Google Scholar 

  • Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 223–241). Cambridge, UK: Cambridge University Press. doi:10.2277/ 0521600812.

    Chapter  Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.

    Google Scholar 

  • Eysenck, M. W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. Emotion, 7, 336–353. doi:10.1037/1528-3542.7.2.336.

    Article  Google Scholar 

  • Fiedler, K. (2001). Affective states trigger processes of assimilation and accommodation. In L. L. Martin & G. L. Clore (Eds.), Theories of mood and cognition: A user’s guidebook (pp. 85–98). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Fiedler, K., & Beier, S. (2014). Affect and cognitive processes in educational contexts. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.

    Google Scholar 

  • Forgas, J. P. (1998). On feeling good and getting your way: Mood effects on negotiator cognition and bargaining strategies. Journal of Personality and Social Psychology, 74, 565–577. doi:10.1037//0022-3514.74.3.565.

    Article  Google Scholar 

  • Frenzel, A. C. (2014). Teacher emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 494–519). New York, NY: Taylor & Francis.

    Google Scholar 

  • Goldin, G. A. (2000). Affective pathways and representation in mathematical problem solving. Mathematical Thinking and Learning, 2, 209–219. doi:10.1207/S15327833MTL0203_3.

    Article  Google Scholar 

  • Gorin, J. S. (2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25, 21–35. doi:10.1111/j.1745-3992.2006.00076.x.

    Article  Google Scholar 

  • Gottman, J., & Levenson, R. W. (2002). A two-factor model for predicting when a couple will divorce: Exploratory analyses using 14-year longitudinal data. Family Process, 41, 83–96. doi:10.1111/j.1545-5300.2002.40102000083.x.

    Article  Google Scholar 

  • Gross, J. J., & Thompson, R. A. (2007). Emotion regulation: Conceptual foundations. In J. J. Gross (Ed.), Handbook of emotion regulation (pp. 3–24). New York, NY: Guildford Press.

    Google Scholar 

  • Hoehn-Saric, R., & McLeod, D. R. (2000). Anxiety and arousal: Physiological changes and their perception. Journal of Affective Disorders, 61, 217–224. doi:10.1016/S0165-0327(00)00339-6.

    Article  Google Scholar 

  • Hsu, K. J., Babeva, K. N., Feng, M. C., Hummer, J. F., & Davison, G. C. (2014). Experimentally induced distraction impacts cognitive but not emotional processes in think-aloud cognitive assessment. Frontiers in Psychology, 5, 1–9. doi:10.3389/fpsyg.2014.00474.

    Google Scholar 

  • Huntsinger, J. R., Clore, G. L., & Bar-Anan, Y. (2010). Mood and global–local focus: Priming a local focus reverses the link between mood and global–local processing. Emotion, 10, 722–726. doi:10.1037/a0019356.

    Article  Google Scholar 

  • Jacobs, S. E., & Gross, J. J. (2014). Emotion regulation in education: Conceptual foundations, current applications, and future directions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.

    Google Scholar 

  • Jamieson, J. P., Mendes, W. B., Blackstock, E., & Schmader, T. (2010). Turning the knots in your stomach into bows: Reappraising arousal improves performance on the GRE. Journal of Experimental Social Psychology, 46, 208–212. doi:10.1016/j.jesp.2009.08.015.

    Article  Google Scholar 

  • Johns, M., Inzlicht, M., & Schmader, T. (2008). Stereotype threat and executive resource depletion: Examining the influence of emotion regulation. Journal of Experimental Psychology: General, 137, 691–705. doi:10.1037/a0013834.

    Article  Google Scholar 

  • King, G., & Wand, J. (2007). Comparing incomparable survey responses: New tools for anchoring vignettes. Political Analysis, 15, 46–66. doi:10.1093/pan/mpl011.

    Article  Google Scholar 

  • Kleinginna Jr., P. R., & Kleinginna, A. M. (1981). A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion, 5, 345–379. doi:10.1007/BF00992553.

    Article  Google Scholar 

  • Kring, A. M., & Sloan, D. M. (2007). The facial expression coding system (FACES): Development, validation and utility. Psychological Assessment, 19, 210–224. doi:10.1037/1040-3590.19.2.210.

    Article  Google Scholar 

  • Kyllonen, P. (2016). Socio-emotional and self-management variables in learning and assessment. In A. A. Rupp & J. P. Leighton (Eds.), Handbook of cognition and assessment (pp. 174–197). Wiley-Blackwell.

    Google Scholar 

  • Kyllonen, P. C., & Bertling, J. (2013). Innovative questionnaire assessment methods to increase crosss-country comparability. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 277–285). Boca Raton, FL: CRC Press. doi:10.1111/jedm.12095.

    Google Scholar 

  • Leighton, J. P. (2004). Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 23, 6–15. doi:10.1111/j.1745-3992.2004.tb00164.x.

    Article  Google Scholar 

  • Leighton, J. P. (2013). Item difficulty and interviewer knowledge effects on the accuracy and consistency of examinee response processes in verbal reports. Applied Measurement in Education, 26, 136–157. doi:10.1080/08957347.2013.765435.

    Article  Google Scholar 

  • Leighton, J. P., & Gierl, M. J. (Eds.). (2007). Cognitive diagnostic assessment for education. Theory and applications. Cambridge, MA: Cambridge University Press. doi:10.1111/j.1745-3984.2008.00072.x.

    Google Scholar 

  • Lewis, B., & Linder, D. (1997). Thinking about choking? Attentional processes and paradoxical performance. Personality and Social Psychology Bulletin, 23, 937–944. doi:10.1177/0146167297239003.

    Article  Google Scholar 

  • Littlewort, G., Whitehill, J., Wu, T., Fasel, I. R., Frank, M., Movellan, J. R., & Bartlett, M. S. (2011). The computer expression recognition toolbox (CERT). In Proceedings of the 9th IEEE conference on Automatic Face and Gesture Recognition (pp. 298–305), Santa Barbara, CA.

    Google Scholar 

  • Maehr, M. L., & Meyer, H. A. (1997). Understanding motivation and schooling: Where we’ve been, where we are, and where we need to go. Educational Psychology Review, 9, 371–409. doi:10.1023/A:1024750807365.

    Article  Google Scholar 

  • Malmivuori, M. L. (2006). Affect and self-regulation. Educational Studies in Mathematics, 63, 149–164. doi:10.1007/s10649-006-9022-8.

    Article  Google Scholar 

  • Mayer, J. D., & Gaschke, Y. N. (1988). The experience and meta-experience of mood. Journal of Personality and Social Psychology, 55, 102–111. doi:10.1037/0022-3514.55.1.102.

    Article  Google Scholar 

  • McCulloch, A. W. (2011). Affect and graphing calculator use. The Journal of Mathematical Behavior, 30, 166–179. doi:10.1016/j.jmathb.2011.02.002.

    Article  Google Scholar 

  • Moors, A., Ellsworth, P. C., Scherer, K. R., & Frijda, N. H. (2013). Appraisal theories of emotion: State of the art and future development. Emotion Review, 5, 119–124. doi:10.1177/1754073912468165.

    Article  Google Scholar 

  • Murphy, S. T., & Zajonc, R. B. (1993). Affect, cognition, and awareness: Affective priming with optimal and suboptimal stimulus exposures. Journal of Personality and Social Psychology, 64, 723–739. doi:10.1037//0022-3514.64.5.723.

    Article  Google Scholar 

  • Norris, S. P. (1990). Effect of eliciting verbal reports of thinking on critical thinking performance. Journal of Educational Measurement, 27, 41–58. doi:10.1111/j.1745-3984.1990.tb00733.x.

    Article  Google Scholar 

  • Osato, E., & Ogawa, N. (2003). Effects of seating positions on heart rates, state anxiety, and estimated interview duration in interview situations. Psychological Reports, 93, 755–770. doi:10.2466/pr0.2003.93.3.755.

    Article  Google Scholar 

  • Paulhus, D. L. (1991). Measurement and control of response biases. In J. Robinson, P. Shaver, & L. Wrightsman (Eds.), Measures of personality and social psychological attitudes (Vol. 1, pp. 17–59). San Diego, CA: Academic Press.

    Chapter  Google Scholar 

  • Pekrun, R. (1992). The impact of emotions on learning and achievement: Towards a theory of cognitive/motivational mediators. Applied Psychology, 41, 359–376. doi:10.1111/j.1464-0597.1992.tb00712.x.

    Article  Google Scholar 

  • Pekrun, R., & Bühner, M. (2014). Self-report measures of academic emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.

    Google Scholar 

  • Pekrun, R., Goetz, T., Daniels, L. M., Stupnisky, R. H., & Perry, R. P. (2010). Boredom in achievement settings: Exploring control-value antecedents and performance outcomes of a neglected emotion. Journal of Educational Psychology, 102, 531–549. doi:10.1037/a0019243.

    Article  Google Scholar 

  • Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of quantitative and qualitative research. Educational Psychologist, 37, 91–106. doi:10.1207/S15326985EP3702_4.

    Article  Google Scholar 

  • Pekrun, R., & Linnenbrink-Garcia, L. (Eds.). (2014). International handbook of emotions in education. New York, NY: Routledge.

    Google Scholar 

  • Pekrun, R., & Meier, E. (2011). Epistemic Emotion Scales (EES). Unpublished manuscript, Department of Psychology, University of Munich, Munich, Germany.

    Google Scholar 

  • Pennebaker, J.W., Chung, C.K., Ireland, M., Gonzales, A., & Booth, R.J. (2007). The development and psychometric properties of LIWC 2007. Retrieved from http://www.liwc.net/LIWC2007LanguageManual.pdf

  • Piaget, J. (1954). Language and thought from a genetic perspective. Acta Psychologica, 10, 51–60. doi:10.1016/0001-6918(54)90004-9.

    Article  Google Scholar 

  • Reisenzein, R., Junge, M., Studtmann, M., & Huber, O. (2014). Observational approaches to the measurement of emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.

    Google Scholar 

  • Rosenthal, R. (2005). Conducting judgment studies: Some methodological issues. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), New handbook of methods in nonverbal behavior research (pp. 199–234). New York, NY: Oxford University Press.

    Google Scholar 

  • Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172. doi:10.1037/0033-295X.110.1.145.

    Article  Google Scholar 

  • Ryan, K. E., & Ryan, A. M. (2005). Psychological processes underlying stereotype threat and standardized math test performance. Educational Psychologist, 40, 53–63. doi:10.1207/s15326985ep4001_4.

    Article  Google Scholar 

  • Sander, D., Grandjean, D., & Scherer, K. R. (2005). A systems approach to appraisal mechanisms in emotion. Neural Networks, 18, 317–352. doi:10.1016/j.neunet.2005.03.001.

    Article  Google Scholar 

  • Sarason, I. G., Sarason, B. R., & Pierce, G. R. (1995). Cognitive interference: At the intelligence-personality crossroads. In D. Saklofske & M. Zeidner (Eds.), International handbook of personality and intelligence (pp. 285–296). New York, NY: Plenum Press.

    Chapter  Google Scholar 

  • Sawyer Jr., T. P., & Hollis-Sawyer, L. A. (2005). Predicting stereotype threat, test anxiety, and cognitive ability test performance: An examination of three models. International Journal of Testing, 5, 225–246. doi:10.1207/s15327574ijt0503_3.

    Article  Google Scholar 

  • Scherer, K. R., Schorr, A., & Johnstone, T. (Eds.). (2001). Appraisal processes in emotion: Theory, methods, research. New York, NY: Oxford University Press.

    Google Scholar 

  • Shuman, V., & Scherer, K. R. (2014). Concepts and structures of emotions. In International handbook of emotions in education (pp. 13–35). New York, NY: Routledge. doi:10.4324/9780203148211.ch2.

    Google Scholar 

  • Sinclair, R. C., & Mark, M. M. (1995). The effects of mood state on judgmental accuracy: Processing strategy as a mechanism. Cognition & Emotion, 9, 417–438. doi:10.1080/02699939508408974.

    Article  Google Scholar 

  • Spielberger, C. D., Gonzalez, H. P., Taylor, C. J., Anton, E. D., Algaze, B., Ross, G. R., & Westberry, L. G. (1980). Manual for the test anxiety inventory (“Test Attitude Inventory”). Redwood City, CA: Consulting Psychologists Press.

    Google Scholar 

  • Steele, C. M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613–629. doi:10.1037//0003-066X.52.6.613.

    Article  Google Scholar 

  • Storbeck, J., & Clore, G. L. (2005). With sadness comes accuracy; With happiness, false memory mood and the false memory effect. Psychological Science, 16, 785–791. doi:10.1111/j.1467-9280.2005.01615.x.

    Article  Google Scholar 

  • Storbeck, J., & Clore, G. L. (2008). The affective regulation of cognitive priming. Emotion, 8, 208–215. doi:10.1037/1528-3542.8.2.208.

    Article  Google Scholar 

  • Turner, J. C., Christensen, A., Kackar-Cam, H. Z., Trucano, M., & Fulmer, S. M. (2014). Enhancing students’ engagement: Report of a 3-Year intervention with middle school teachers. American Educational Research Journal, 51, 1195–1226. doi:10.3102/0002831214532515.

    Article  Google Scholar 

  • Yerkes, R. M., & Dodson, J. D. (1908). The relation of strength of stimulus to rapidity of habit-formation. Journal of Comparative Neurology and Psychology, 18, 459–482. doi:10.1002/cne.920180503.

    Article  Google Scholar 

  • Zeidner, M. (2014). Anxiety in education. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 265–288). New York City, NY: Routledge.

    Google Scholar 

  • Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 39–58. doi:10.1109/TPAMI.2008.52.

    Article  Google Scholar 

  • Zuriff, G. E. (1997). Accommodations for test anxiety under ADA. The Journal of the American Academy of Psychiatry and the Law, 25, 197–206.

    Google Scholar 

Download references

Acknowledgement

Preparation of this chapter was supported by a grant to the first author from the Social Sciences and Humanities Research Council of Canada (SSHRC Grant No. 410-2011-0811). Grantees undertaking such projects are encouraged to express freely their professional judgment. This paper, therefore, does not necessarily represent the positions or the policies of the Canadian government, and no official endorsement should be inferred.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jacqueline P. Leighton .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Leighton, J.P., Tang, W., Guo, Q. (2017). Response Processes and Validity Evidence: Controlling for Emotions in Think Aloud Interviews. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56129-5_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56128-8

  • Online ISBN: 978-3-319-56129-5

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics