Abstract
The objective of the chapter is to highlight the potentially problematic effects that emotional variables and, in particular, evaluative anxiety may have on response processing data collected using think-aloud interviews. Given the collection of response processing data to inform the nature of processing skills that test-takers use to respond to test items, and related validity arguments, it is imperative that the integrity of these data be considered. Recent research illustrating how distractions and disruptions, in some cases emotionally-induced, can influence response processing data in think-aloud interviews is presented. Methods to control and minimize disruptive emotions for participants in think-aloud interviews are also considered, as well as areas for future research.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, & NCME). (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Beilock, S. L., & Carr, T. H. (2001). On the fragility of skilled performance: What governs choking under pressure? Journal of Experimental Psychology: General, 130, 701–725. doi:10.1037/0096-3445.130.4.701.
Beilock, S. L., Kulp, C. A., Holt, L. E., & Carr, T. H. (2004). More on the fragility of performance: Choking under pressure in mathematical problem solving. Journal of Experimental Psychology: General, 133, 584–600. doi:10.1037/0096-3445.133.4.584.
Ben-Zeev, T., Fein, S., & Inzlicht, M. (2005). Arousal and stereotype threat. Journal of Experimental Social Psychology, 41, 174–181. doi:10.1016/j.jesp.2003.11.007.
Berridge, K. C., & Winkielman, P. (2003). What is an unconscious emotion? (The case for unconscious “liking”). Cognition and Emotion, 17, 181–211. doi:10.1080/02699930244000273.
Beukeboom, C. J., & Semin, G. R. (2006). How mood turns on language. Journal of Experimental Social Psychology, 42, 553–566. doi:10.1016/j.jesp.2005.09.005.
Bishop, S. J. (2007). Neurocognitive mechanisms of anxiety: An integrative account. Trends in Cognitive Sciences, 11, 307–316. doi:10.1016/j.tics.2007.05.008.
Bosson, J. K., Haymovitz, E. L., & Pinel, E. C. (2004). When saying and doing diverge: The effects of stereotype threat on self-reported versus non-verbal anxiety. Journal of Experimental Social Psychology, 40, 247–255. doi:10.1016/S0022-1031(03)00099-4.
Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37.
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21, 1363–1368. doi:10.1177/0956797610383437.
Carver, C. S., Peterson, L. M., Follansbee, D. J., & Scheier, M. F. (1983). Effects of self-directed attention on performance and persistence among persons high and low in test anxiety. Cognitive Therapy and Research, 7, 333–353. doi:10.1007/BF01177556.
Cassady, J. C., & Johnson, R. E. (2002). Cognitive test anxiety and academic performance. Contemporary Educational Psychology, 27, 270–295. doi:10.1006/ceps.2001.1094.
Chu, M.-W., Guo, Q., & Leighton, J. P. (2014). Students’ interpersonal trust and attitudes towards standardized tests: Exploring affective variables related to student assessment. Assessment in Education: Principles, Policy & Practice, 21, 167–192. doi:10.1080/0969594X.2013.844094.
Chu, M.-W., & Leighton, J. P. (2016). Using errors to enhance learning feedback in computer programming. In S. Tettegah & M. McCreery (Eds.), Emotions and technology: Communication of feelings for, with and through digital media – Volume I: Emotions, learning, and technology (pp. 89–117). Elsevier Publishing.
Cizek, G. J., & Burg, S. S. (2006). Addressing test anxiety in a high-stakes environment: Strategies for classrooms and schools. Thousand Oaks, CA: Corwin Press.
Cohen, G. L., & Sherman, D. K. (2005). Stereotype threat and the social and scientific contexts of the race achievement gap. American Psychologist, 60, 270–271. doi:10.1037/0003-066X.60.3.270.
Cuddy, A. J., Wilmuth, C. A., Yap, A. J., & Carney, D. R. (2015). Preparatory power posing affects nonverbal presence and job interview performance. Journal of Applied Psychology, 100, 1286–1295. doi:10.1037/a0038543.
D’Arcey, T., Johnson, M., & Ennis, M. (2012). Assessing the validity of FaceReader using facial electromyography. In Proceedings of APS 24th annual meeting. Retrieved from www.darcey.us/pdf/facereader.pdf
D’Mello, S., & Graesser, A. (2009). Automatic detection of learner’s affect from gross body language. Applied Artificial Intelligence, 23, 123–150. doi:10.1080/08839510802631745.
DeBellis, V. A., & Goldin, G. A. (2006). Affect and meta-affect in mathematical problem solving: A representational perspective. Educational Studies in Mathematics, 63, 131–147. doi:10.1007/s10649-006-9026-4.
Ekman, P. (1972). Universals and cultural differences in facial expressions of emotion. In J. Cole (Ed.), Nebraska symposium on motivation (Vol. 19, pp. 207–283). Lincoln, NE: University of Nebraska Press.
Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200. doi:10.1080/02699939208411068.
Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 223–241). Cambridge, UK: Cambridge University Press. doi:10.2277/ 0521600812.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.
Eysenck, M. W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. Emotion, 7, 336–353. doi:10.1037/1528-3542.7.2.336.
Fiedler, K. (2001). Affective states trigger processes of assimilation and accommodation. In L. L. Martin & G. L. Clore (Eds.), Theories of mood and cognition: A user’s guidebook (pp. 85–98). Mahwah, NJ: Lawrence Erlbaum.
Fiedler, K., & Beier, S. (2014). Affect and cognitive processes in educational contexts. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.
Forgas, J. P. (1998). On feeling good and getting your way: Mood effects on negotiator cognition and bargaining strategies. Journal of Personality and Social Psychology, 74, 565–577. doi:10.1037//0022-3514.74.3.565.
Frenzel, A. C. (2014). Teacher emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 494–519). New York, NY: Taylor & Francis.
Goldin, G. A. (2000). Affective pathways and representation in mathematical problem solving. Mathematical Thinking and Learning, 2, 209–219. doi:10.1207/S15327833MTL0203_3.
Gorin, J. S. (2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25, 21–35. doi:10.1111/j.1745-3992.2006.00076.x.
Gottman, J., & Levenson, R. W. (2002). A two-factor model for predicting when a couple will divorce: Exploratory analyses using 14-year longitudinal data. Family Process, 41, 83–96. doi:10.1111/j.1545-5300.2002.40102000083.x.
Gross, J. J., & Thompson, R. A. (2007). Emotion regulation: Conceptual foundations. In J. J. Gross (Ed.), Handbook of emotion regulation (pp. 3–24). New York, NY: Guildford Press.
Hoehn-Saric, R., & McLeod, D. R. (2000). Anxiety and arousal: Physiological changes and their perception. Journal of Affective Disorders, 61, 217–224. doi:10.1016/S0165-0327(00)00339-6.
Hsu, K. J., Babeva, K. N., Feng, M. C., Hummer, J. F., & Davison, G. C. (2014). Experimentally induced distraction impacts cognitive but not emotional processes in think-aloud cognitive assessment. Frontiers in Psychology, 5, 1–9. doi:10.3389/fpsyg.2014.00474.
Huntsinger, J. R., Clore, G. L., & Bar-Anan, Y. (2010). Mood and global–local focus: Priming a local focus reverses the link between mood and global–local processing. Emotion, 10, 722–726. doi:10.1037/a0019356.
Jacobs, S. E., & Gross, J. J. (2014). Emotion regulation in education: Conceptual foundations, current applications, and future directions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.
Jamieson, J. P., Mendes, W. B., Blackstock, E., & Schmader, T. (2010). Turning the knots in your stomach into bows: Reappraising arousal improves performance on the GRE. Journal of Experimental Social Psychology, 46, 208–212. doi:10.1016/j.jesp.2009.08.015.
Johns, M., Inzlicht, M., & Schmader, T. (2008). Stereotype threat and executive resource depletion: Examining the influence of emotion regulation. Journal of Experimental Psychology: General, 137, 691–705. doi:10.1037/a0013834.
King, G., & Wand, J. (2007). Comparing incomparable survey responses: New tools for anchoring vignettes. Political Analysis, 15, 46–66. doi:10.1093/pan/mpl011.
Kleinginna Jr., P. R., & Kleinginna, A. M. (1981). A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion, 5, 345–379. doi:10.1007/BF00992553.
Kring, A. M., & Sloan, D. M. (2007). The facial expression coding system (FACES): Development, validation and utility. Psychological Assessment, 19, 210–224. doi:10.1037/1040-3590.19.2.210.
Kyllonen, P. (2016). Socio-emotional and self-management variables in learning and assessment. In A. A. Rupp & J. P. Leighton (Eds.), Handbook of cognition and assessment (pp. 174–197). Wiley-Blackwell.
Kyllonen, P. C., & Bertling, J. (2013). Innovative questionnaire assessment methods to increase crosss-country comparability. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 277–285). Boca Raton, FL: CRC Press. doi:10.1111/jedm.12095.
Leighton, J. P. (2004). Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 23, 6–15. doi:10.1111/j.1745-3992.2004.tb00164.x.
Leighton, J. P. (2013). Item difficulty and interviewer knowledge effects on the accuracy and consistency of examinee response processes in verbal reports. Applied Measurement in Education, 26, 136–157. doi:10.1080/08957347.2013.765435.
Leighton, J. P., & Gierl, M. J. (Eds.). (2007). Cognitive diagnostic assessment for education. Theory and applications. Cambridge, MA: Cambridge University Press. doi:10.1111/j.1745-3984.2008.00072.x.
Lewis, B., & Linder, D. (1997). Thinking about choking? Attentional processes and paradoxical performance. Personality and Social Psychology Bulletin, 23, 937–944. doi:10.1177/0146167297239003.
Littlewort, G., Whitehill, J., Wu, T., Fasel, I. R., Frank, M., Movellan, J. R., & Bartlett, M. S. (2011). The computer expression recognition toolbox (CERT). In Proceedings of the 9th IEEE conference on Automatic Face and Gesture Recognition (pp. 298–305), Santa Barbara, CA.
Maehr, M. L., & Meyer, H. A. (1997). Understanding motivation and schooling: Where we’ve been, where we are, and where we need to go. Educational Psychology Review, 9, 371–409. doi:10.1023/A:1024750807365.
Malmivuori, M. L. (2006). Affect and self-regulation. Educational Studies in Mathematics, 63, 149–164. doi:10.1007/s10649-006-9022-8.
Mayer, J. D., & Gaschke, Y. N. (1988). The experience and meta-experience of mood. Journal of Personality and Social Psychology, 55, 102–111. doi:10.1037/0022-3514.55.1.102.
McCulloch, A. W. (2011). Affect and graphing calculator use. The Journal of Mathematical Behavior, 30, 166–179. doi:10.1016/j.jmathb.2011.02.002.
Moors, A., Ellsworth, P. C., Scherer, K. R., & Frijda, N. H. (2013). Appraisal theories of emotion: State of the art and future development. Emotion Review, 5, 119–124. doi:10.1177/1754073912468165.
Murphy, S. T., & Zajonc, R. B. (1993). Affect, cognition, and awareness: Affective priming with optimal and suboptimal stimulus exposures. Journal of Personality and Social Psychology, 64, 723–739. doi:10.1037//0022-3514.64.5.723.
Norris, S. P. (1990). Effect of eliciting verbal reports of thinking on critical thinking performance. Journal of Educational Measurement, 27, 41–58. doi:10.1111/j.1745-3984.1990.tb00733.x.
Osato, E., & Ogawa, N. (2003). Effects of seating positions on heart rates, state anxiety, and estimated interview duration in interview situations. Psychological Reports, 93, 755–770. doi:10.2466/pr0.2003.93.3.755.
Paulhus, D. L. (1991). Measurement and control of response biases. In J. Robinson, P. Shaver, & L. Wrightsman (Eds.), Measures of personality and social psychological attitudes (Vol. 1, pp. 17–59). San Diego, CA: Academic Press.
Pekrun, R. (1992). The impact of emotions on learning and achievement: Towards a theory of cognitive/motivational mediators. Applied Psychology, 41, 359–376. doi:10.1111/j.1464-0597.1992.tb00712.x.
Pekrun, R., & Bühner, M. (2014). Self-report measures of academic emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.
Pekrun, R., Goetz, T., Daniels, L. M., Stupnisky, R. H., & Perry, R. P. (2010). Boredom in achievement settings: Exploring control-value antecedents and performance outcomes of a neglected emotion. Journal of Educational Psychology, 102, 531–549. doi:10.1037/a0019243.
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of quantitative and qualitative research. Educational Psychologist, 37, 91–106. doi:10.1207/S15326985EP3702_4.
Pekrun, R., & Linnenbrink-Garcia, L. (Eds.). (2014). International handbook of emotions in education. New York, NY: Routledge.
Pekrun, R., & Meier, E. (2011). Epistemic Emotion Scales (EES). Unpublished manuscript, Department of Psychology, University of Munich, Munich, Germany.
Pennebaker, J.W., Chung, C.K., Ireland, M., Gonzales, A., & Booth, R.J. (2007). The development and psychometric properties of LIWC 2007. Retrieved from http://www.liwc.net/LIWC2007LanguageManual.pdf
Piaget, J. (1954). Language and thought from a genetic perspective. Acta Psychologica, 10, 51–60. doi:10.1016/0001-6918(54)90004-9.
Reisenzein, R., Junge, M., Studtmann, M., & Huber, O. (2014). Observational approaches to the measurement of emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 348–367). New York, NY: Yarlor & Francis.
Rosenthal, R. (2005). Conducting judgment studies: Some methodological issues. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), New handbook of methods in nonverbal behavior research (pp. 199–234). New York, NY: Oxford University Press.
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172. doi:10.1037/0033-295X.110.1.145.
Ryan, K. E., & Ryan, A. M. (2005). Psychological processes underlying stereotype threat and standardized math test performance. Educational Psychologist, 40, 53–63. doi:10.1207/s15326985ep4001_4.
Sander, D., Grandjean, D., & Scherer, K. R. (2005). A systems approach to appraisal mechanisms in emotion. Neural Networks, 18, 317–352. doi:10.1016/j.neunet.2005.03.001.
Sarason, I. G., Sarason, B. R., & Pierce, G. R. (1995). Cognitive interference: At the intelligence-personality crossroads. In D. Saklofske & M. Zeidner (Eds.), International handbook of personality and intelligence (pp. 285–296). New York, NY: Plenum Press.
Sawyer Jr., T. P., & Hollis-Sawyer, L. A. (2005). Predicting stereotype threat, test anxiety, and cognitive ability test performance: An examination of three models. International Journal of Testing, 5, 225–246. doi:10.1207/s15327574ijt0503_3.
Scherer, K. R., Schorr, A., & Johnstone, T. (Eds.). (2001). Appraisal processes in emotion: Theory, methods, research. New York, NY: Oxford University Press.
Shuman, V., & Scherer, K. R. (2014). Concepts and structures of emotions. In International handbook of emotions in education (pp. 13–35). New York, NY: Routledge. doi:10.4324/9780203148211.ch2.
Sinclair, R. C., & Mark, M. M. (1995). The effects of mood state on judgmental accuracy: Processing strategy as a mechanism. Cognition & Emotion, 9, 417–438. doi:10.1080/02699939508408974.
Spielberger, C. D., Gonzalez, H. P., Taylor, C. J., Anton, E. D., Algaze, B., Ross, G. R., & Westberry, L. G. (1980). Manual for the test anxiety inventory (“Test Attitude Inventory”). Redwood City, CA: Consulting Psychologists Press.
Steele, C. M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613–629. doi:10.1037//0003-066X.52.6.613.
Storbeck, J., & Clore, G. L. (2005). With sadness comes accuracy; With happiness, false memory mood and the false memory effect. Psychological Science, 16, 785–791. doi:10.1111/j.1467-9280.2005.01615.x.
Storbeck, J., & Clore, G. L. (2008). The affective regulation of cognitive priming. Emotion, 8, 208–215. doi:10.1037/1528-3542.8.2.208.
Turner, J. C., Christensen, A., Kackar-Cam, H. Z., Trucano, M., & Fulmer, S. M. (2014). Enhancing students’ engagement: Report of a 3-Year intervention with middle school teachers. American Educational Research Journal, 51, 1195–1226. doi:10.3102/0002831214532515.
Yerkes, R. M., & Dodson, J. D. (1908). The relation of strength of stimulus to rapidity of habit-formation. Journal of Comparative Neurology and Psychology, 18, 459–482. doi:10.1002/cne.920180503.
Zeidner, M. (2014). Anxiety in education. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 265–288). New York City, NY: Routledge.
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 39–58. doi:10.1109/TPAMI.2008.52.
Zuriff, G. E. (1997). Accommodations for test anxiety under ADA. The Journal of the American Academy of Psychiatry and the Law, 25, 197–206.
Acknowledgement
Preparation of this chapter was supported by a grant to the first author from the Social Sciences and Humanities Research Council of Canada (SSHRC Grant No. 410-2011-0811). Grantees undertaking such projects are encouraged to express freely their professional judgment. This paper, therefore, does not necessarily represent the positions or the policies of the Canadian government, and no official endorsement should be inferred.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Leighton, J.P., Tang, W., Guo, Q. (2017). Response Processes and Validity Evidence: Controlling for Emotions in Think Aloud Interviews. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-56129-5_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56128-8
Online ISBN: 978-3-319-56129-5
eBook Packages: Social SciencesSocial Sciences (R0)