Skip to main content
SpringerLink
Log in

Observing Testing Situations: Validation as Jazz

  • Chapter
  • First Online:
Understanding and Investigating Response Processes in Validation Research

Part of the book series: Social Indicators Research Series ((SINS,volume 69))

Abstract

Observations of real-life testing situations can provide important insights into test validation and assessment response processes. We consider observations of face-to-face interaction as a starting point to investigate how variation in assessment performance is informed by the ecological characteristics of the testing situation. This perspective offers a contrast to conventional ‘think aloud’ protocols. Observations of real-life testing situations capture in-vivo perspectives on assessment interaction and performance as it occurs, rather than as if occurring in-vitro. Our aim is to deliberately capture and consider the “noisy” ecological characteristics in the testing situation (e.g., interaction, setting, affect) that may influence response processes, but that are not part of the trait under investigation. Observations of real-life testing situations reveal observable structures and patterns of behaviour, but every performance is somewhat different. Like jazz, the testing situation involves elements of improvisation. This is illustrated using transcripts from the OECD Survey of Adult Skills (PIAAC).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In the transcription, I = interviewer (test administrator), R = respondent (test taker), [square bracket indicates overlapping speech, and ((double parentheses indicate descriptions of what is going on.

References

  • Bronfenbrenner, U. (1979). The ecology of human development. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Bronfenbrenner, U. (1994). Ecological models of human development. In T. Huston & T. N. Postlethwaith (Eds.), International encyclopedia of education (Vol. 3, 2nd ed., pp. 1643–1647). New York: Elsevier Science.

    Google Scholar 

  • Bronfenbrenner, U. (2005). Making human beings human: Bioecological perspectives on human development. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Du Bois, J., & Kärkkäinen, E. (2012). Staking a stance on emotion: Affect, sequence and intersubjectivity in dialogic interaction. Text and Talk, 32(4), 433–451.

    Google Scholar 

  • Duranti, A., & Goodwin, C. (Eds.). (1992). Rethinking context: Language as an interactive phenomenon (pp. 1–42). Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Eklöf, H. (2010). Skill and will: Test-taking motivation and assessment quality. Assessment in Education: Principles, Policy & Practice, 17(4), 345–356.

    Article  Google Scholar 

  • Fox, J. (2003). From products to process: An ecological approach to bias detection. International Journal of Testing, 3(1), 21–48.

    Article  Google Scholar 

  • Giddens, A. (1988). Goffman as a systematic social theorist. In P. Drew & A. Wootton (Eds.), Erving Goffman: Exploring the interaction order (pp. 25–279). Oxford, UK: Polity Press.

    Google Scholar 

  • Goffman, E. (1959). The presentation of self in everyday life. Garden City, NJ: Doubleday.

    Google Scholar 

  • Goffman, E. (1964). The neglected situation. American Anthropologist, 66(6), 133–136.

    Article  Google Scholar 

  • Goffman, E. (1981). Forms of talk. Philadelphia: University of Pennsylvania Press.

    Google Scholar 

  • Goffman, E. (1983). Felicity’s condition. American Journal of Sociology, 89(5), 1–53.

    Google Scholar 

  • Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522.

    Article  Google Scholar 

  • Goodwin, C. (2002). Time in action. Current Anthropology, 43(S4), 1–53.

    Article  Google Scholar 

  • Goodwin, C. (2007a). Participation, stance and sffect in the organisation of activities. Discourse & Society, 18(1), 53–73.

    Article  Google Scholar 

  • Goodwin, C. (2007b). Environmentally coupled gestures. In S. Duncan, J. Cassell, & E. Levy (Eds.), Gesture and the dynamic dimension of language (pp. 195–212). Amsterdam: John Benjamin’s.

    Chapter  Google Scholar 

  • Goodwin, M., Cakaite, A., & Goodwin, C. (2012). Emotion as stance. In M. Leena Sorjonen & A. Perakyla (Eds.), Emotion in interaction (pp. 16–41). Oxford, UK: Oxford University Press.

    Chapter  Google Scholar 

  • Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, R. P. Merenda, & C. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3–38). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Maddox, B. (2014). Globalising assessment: An ethnography of literacy assessment, camels and fast food in the Mongolian Gobi. Comparative Education, 50(4), 474–489.

    Article  Google Scholar 

  • Maddox, B. (2015). The neglected situation: Assessment performance and interaction in context. Assessment in Education, 22(4), 427–443.

    Article  Google Scholar 

  • Maddox, B., Zumbo, B. D., Tay-Lim, B. S.-H., & Demin Qu, I. (2015). An anthropologist among the psychometricians: Assessment events, ethnography and DIF in the Mongolian Gobi. International Journal of Testing, 15(4), 291–309.

    Article  Google Scholar 

  • Malinowski, B. (1923). The problem of meaning in primitive languages. In C. K. Ogden & I. A. Richards (Eds.), The meaning of meaning (pp. 296–336). New York: Harcourt.

    Google Scholar 

  • McNamara, T. F. (1997). “Interaction” in second language performance assessment: Whose performance? Applied Linguistics, 18(4), 446–466.

    Article  Google Scholar 

  • McNamara, T. F. (2007). Language testing: A question of context. In J. Fox, M. Wesche, D. Bayliss, L. Cheng, C. Turner, & C. Doe (Eds.), Language testing reconsidered (pp. 131–137). Ottawa, ON: University of Ottawa Press.

    Google Scholar 

  • McNamara, T. F., & Roever, C. (2006). Language testing: The social dimension. Oxford, UK: Blackwell.

    Google Scholar 

  • McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92(3), 350–371.

    Article  Google Scholar 

  • McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago, IL: The University of Chicago Press.

    Google Scholar 

  • Organization for Economic Cooperation and Development. (2013). Technical report of the survey of adult skills (PIAAC). Paris: Author.

    Google Scholar 

  • Shegloff, E. A. (1988). Goffman and the analysis of conversation. In P. Drew & A. Wootton (Eds.), Erving Goffman: Exploring the interaction order (pp. 89–135). Oxford, UK: Polity Press.

    Google Scholar 

  • Schegloff, E. A., & Sacks, H. (1973). Opening up closings. Semiotica, 8(4), 289–327.

    Article  Google Scholar 

  • Stone, J., & Zumbo, B. D. (2016). Validity as a pragmatist project: A global concern with local application. In V. Aryadoust & J. Fox (Eds.), Trends in language assessment research and practice (pp. 555–573). Newcastle, UK: Cambridge Scholars.

    Google Scholar 

  • Yamamoto, K. (2011, September). Implementation of CBT in the PIAAC field test and CAT in the PIAAC. In: 20th seminar report at the Centre for Research on Educational Testing, Tokyo.

    Google Scholar 

  • Zumbo, B. D. (2007a). Three generations of differential item functioning (DIF) analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233.

    Article  Google Scholar 

  • Zumbo, B. D. (2007b). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Psychometrics (Vol. 26, pp. 45–79). Amsterdam: Elsevier Science B.V.

    Google Scholar 

  • Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte, NC: IAP – Information Age.

    Google Scholar 

  • Zumbo, B. D. (2015a). Tides, rips, and eerie calm at the confluence of data uses, consequences, and validity. Plenary address, “The Production of Data in International Assessments,” research conference organized by the Laboratory of International Assessment Studies, Economic and Social Research Council (ESRC), University of East Anglia, Norwich, UK. URL: https://youtu.be/ahfbnLgUR5E

  • Zumbo, B. D. (2015b). Consequences, side effects and the ecology of testing: Keys to considering assessment ‘in vivo’. Keynote address, the annual meeting of the Association for Educational Assessment – Europe (AEA-Europe), Glasgow, Scotland. URL: https://youtu.be/0L6Lr2BzuSQ

  • Zumbo, B. D., Liu, Y., Wu, A. D., Shear, B. R., Astivia, O. L. O., & Ark, T. K. (2015). A methodology for Zumbo’s third generation DIF analyses and the ecology of item responding. Language Assessment Quarterly, 12(1), 136–151.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of International Development, University of East Anglia, Norwich, NR4 7TJ, UK

    Bryan Maddox

  2. Laboratory of International Assessment Studies, Norwich, UK

    Bryan Maddox

  3. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, 2125 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Bruno D. Zumbo

Authors
  1. Bryan Maddox
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno D. Zumbo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bryan Maddox .

Editor information

Editors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, Vancouver, British Columbia, Canada

    Bruno D. Zumbo

  2. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, Vancouver, British Columbia, Canada

    Anita M. Hubley

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Maddox, B., Zumbo, B.D. (2017). Observing Testing Situations: Validation as Jazz. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56129-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56128-8

  • Online ISBN: 978-3-319-56129-5

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation