Skip to main content
SpringerLink
Log in

Response Processes in the Context of Validity: Setting the Stage

  • Chapter
  • First Online:
Understanding and Investigating Response Processes in Validation Research

Part of the book series: Social Indicators Research Series ((SINS,volume 69))

Abstract

In this first chapter, we set the stage for subsequent chapters that we believe will push the boundaries of our current thinking about response processes as validity evidence. Evidence based on response processes has been an overlooked source of validity evidence, but one that offers much promise and strength to support the inferences we make from test scores and to provide explanations for test score variance. In this chapter, we address the critical, but often overlooked step of defining response processes. We next take a brief look back at some key moments and players in the history of response processes theorizing and research. We also examine the prevalence of validity evidence based on response processes reported in the published research literature. We conclude by setting a course for the future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Henceforth referred to as the Standards.

  2. 2.

    Messick (1989a, 1990) would agree with this view but noted that the dominant view of content validation focuses on expert judgments about test content representativeness and relevance. It is because the dominant view of content validity does not address response consistencies and test scores that Messick (1989b) argued that “so-called content validity does not qualify as validity at all” (p. 7).

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME]. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME]. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • American Psychological Association, American Educational Research Association, & National Council on Measurement in Education [APA, AERA, & NCME]. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

    Google Scholar 

  • Ark, T. K., Ark, N., & Zumbo, B. D. (2014). Validation practices of the Objective Structured Clinical Examination (OSCE). In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 267–288). New York, NY: Springer.

    Google Scholar 

  • Beckman, T. J., Cook, D. A., & Mandrekar, J. N. (2005). What is the validity evidence for assessments of clinical teaching? Journal of General Internal Medicine, 20, 1159–1164. doi:10.1111/j.1525-1497.2005.0258.x.

    Article  Google Scholar 

  • Brückner, S., & Pellegrino, J. W. (2016). Integrating the analysis of mental operations into multilevel models to validate an assessment of higher education students’ competency in business and economics. Journal of Educational Measurement, 53, 293–312. doi:10.1111/jedm.12113.

    Article  Google Scholar 

  • Chan, E. K. H. (2014). Standards and guidelines for validation practices: Development and evaluation of measurement instruments. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 9–24). New York, NY: Springer.

    Google Scholar 

  • Chan, E. K. H., Munro, D. W., Huang, A. H. S., Zumbo, B. D., Vojdanijahromi, R., & Ark, N. (2014). Validation practices in counseling: Major journals, mattering instruments, and the Kuder Occupational Interest Survey (KOIS). In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 67–87). New York, NY: Springer.

    Google Scholar 

  • Chan, E. K. H., Zumbo, B. D., Chen, M. Y., Zhang, W., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of measurement validity in articles published in Quality of Life Research. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 217–228). New York, NY: Springer.

    Google Scholar 

  • Chan, E. K. H., Zumbo, B. D., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of validity evidence in the field of health care: A focus on papers published in Value in Health. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 257–265). New York, NY: Springer.

    Google Scholar 

  • Chan, E. K. H., Zumbo, B. D., Zhang, W., Chen, M. Y., Darmawanti, I., & Mulyana, O. P. (2014). Medical Outcomes Study Short Form-36 (SF-36) and the World Health Organization Quality of Life (WHOQoL) assessment: Reporting of psychometric validity evidence. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 243–255). New York, NY: Springer.

    Google Scholar 

  • Chinni, M., & Hubley, A. M. (2014). The Satisfaction with Life Scale (SWLS): A review of reported validation practice. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 35–66). New York, NY: Springer.

    Google Scholar 

  • Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397–412. doi:10.1177/0013164407310130.

    Article  Google Scholar 

  • Collie, R. J., & Zumbo, B. D. (2014). Validity evidence in the Journal of Educational Psychology: Documenting current practice and a comparison with earlier practice. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 113–135). New York, NY: Springer.

    Google Scholar 

  • Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. American Journal of Medicine, 119, 166.e7–166.e16. http://doi.org/10.1016/j.amjmed.2005.10.036.

    Article  Google Scholar 

  • Cook, D. A., Zendejas, B., Hamstra, S. J., Hatala, R., & Brydges, R. (2014). What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Advances in Health Sciences Education, 19, 233–250. doi:10.1007/s10459-013-9458-4.

    Article  Google Scholar 

  • Cox, D. W., & Owen, J. J. (2014). Validity evidence for a perceived social support measure in a population health context. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 229–241). New York, NY: Springer.

    Google Scholar 

  • Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830–837. doi:10.1046/j.1365-2923.2003.01594.x.

    Article  Google Scholar 

  • Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186. doi:10.1007/BF02294171.

    Article  Google Scholar 

  • Embretson, S. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125–150). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Embretson, S., Schneider, L. M., & Roth, D. L. (1986). Multiple processing strategies and the construct validity of verbal reasoning tests. Journal of Educational Measurement, 23, 13–32. doi:10.1111/j.1745-3984.1986.tb00231.x.

    Article  Google Scholar 

  • Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179–197.

    Article  Google Scholar 

  • Embretson, S. E., & Schneider, L. M. (1989). Cognitive component models for psychometric analogies: Conceptually driven versus interactive process models. Learning and Individual Differences, 1, 155–178. doi:10.1016/1041-6080(89)90001-0.

    Article  Google Scholar 

  • Embretson, S. E., & Yang, X. (2013). A multicomponent latent trait model for diagnosis. Psychometrika, 78, 14–36. doi:10.1007/s11336-012-9296-y.

    Article  Google Scholar 

  • Gorin, J. S., & Embretson, S. E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30, 394–411. doi:10.1177/0146621606288554.

    Article  Google Scholar 

  • Gunnell, K. E., Schellenberg, B. J. I., Wilson, P. M., Crocker, P. R. E., Mack, D. E., & Zumbo, B. D. (2014). A review of validity evidence presented in the Journal of Sport and Exercise Psychology (2002–2012): Misconceptions and recommendations for validation research. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 137–156). New York, NY: Springer.

    Google Scholar 

  • Gunnell, K. E., Wilson, P. M., Zumbo, B. D., Crocker, P. R. E., Mack, D. E., & Schellenberg, B. J. I. (2014). Validity theory and validity evidence for scores derived from the Behavioural Regulation in Exercise Questionnaire. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 175–191). New York, NY: Springer.

    Google Scholar 

  • Hubley, A. M., Zhu, M., Sasaki, A., & Gadermann, A. (2014). A synthesis of validation practices in the journals Psychological Assessment and European Journal of Psychological Assessment. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 193–213). New York, NY: Springer.

    Google Scholar 

  • Ivie, J. L., & Embretson, S. E. (2010). Cognitive process modeling of spatial ability: The assembling objects task. Intelligence, 38, 324–335. doi:10.1016/j.intell.2010.02.002.

    Article  Google Scholar 

  • Lennon, R. T. (1956). Assumptions underlying the use of content validity. Educational and Psychological Measurement, 16, 294–304.

    Article  Google Scholar 

  • Lyons-Thomas, J., Liu, Y., & Zumbo, B. D. (2014). Validation practices in the social, behavioral, and health sciences: A synthesis of syntheses. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (p. 313319). New York, NY: Springer.

    Google Scholar 

  • McBride, H. L., Wiens, R. M., McDonald, M. J., Cox, D. W., & Chan, E. K. H. (2014). The Edinburgh Postnatal Depression Scale (EPDS): A review of the reported validity evidence. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 157–174). New York, NY: Springer.

    Google Scholar 

  • Messick, S. (1972). Beyond structure: In search of functional models of psychological process. Psychometrika, 37, 357–375.

    Article  Google Scholar 

  • Messick, S. (1989a). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5–11.

    Article  Google Scholar 

  • Messick, S. (1989b). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: Macmillan Publishing Co, Inc.

    Google Scholar 

  • Messick, S. (1990). Validity of test interpretation and use. Princeton, NJ: Educational Testing Service.

    Google Scholar 

  • Messick, S. (1995). Validity of psychological assessment. Validation of inferences from persons’ responses and performance as scientific inquiry into score meaning. American Psychologist, 50, 741–749.

    Article  Google Scholar 

  • Mislevy, R. J. (2009). Validity from the perspective of model-based reasoning. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions, and applications (pp. 83–108). Charlotte, NC: Information Age.

    Google Scholar 

  • Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). On the role of task model variables in assessment design. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 97–128). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Padilla, J.-L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26, 136–144.

    Google Scholar 

  • Pellegrino, J. W., Baxter, G. P., & Glaser, R. (1999). Addressing the “two disciplines” problem: Linking theories of cognition and learning with assessment and instructional practice. Review of Research in Education, 24, 307–353.

    Google Scholar 

  • Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for conceptualizing and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51, 59–81. doi:10.1080/00461520.2016.1145550.

  • Pellegrino, J. W., & Glaser, R. (1979). Cognitive correlates and components in the analysis of individual differences. Intelligence, 3, 187–215.

    Article  Google Scholar 

  • Sandilands, D., & Zumbo, B. D. (2014). (Mis)alignment of medical education validation research with contemporary validity theory: The Mini-CEX as an example. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 289–310). New York, NY: Springer.

    Google Scholar 

  • Shear, B. R., & Zumbo, B. D. (2014). What counts as evidence: A review of validity studies in Educational and Psychological Measurement. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 91–111). New York, NY: Springer.

    Google Scholar 

  • Tourangeau, R. (1984). Cognitive science and survey methods: A cognitive perspective. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.

    Google Scholar 

  • Villalobos Coronel, M. (2015). Synthesis of reliability and validation practices used with the Rosenberg self-esteem scale. Master’s thesis, University of British Columbia. Retrieved from https://open.library.ubc.ca/cIRcle/collections/24/items/1.0165784

  • Whitely (Embretson), S. E. (1977). Information-processing on intelligence test items: Some response components. Applied Psychological Measurement, 1, 465–476. doi:10.1177/014662167700100402.

    Article  Google Scholar 

  • Whitely (Embretson), S. E. (1980). Multicomponent latent trait models for ability tests. Psychometrika, 45, 479–494.

    Article  Google Scholar 

  • Zumbo, B. D. (2014). What role does, and should, the test Standards play outside of the United States of America? Educational Measurement: Issues and Practice, 33, 31–33.

    Google Scholar 

  • Zumbo, B. D., & Chan, E. K. H. (Eds.). (2014). Validity and validation in social, behavioral, and health sciences. New York, NY: Springer.

    Google Scholar 

  • Zumbo, B. D., Chan, E. K. H., Chen, M. Y., Zhang, W., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of measurement validity in articles published in Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 27–34). New York, NY: Springer.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, 2125 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Anita M. Hubley & Bruno D. Zumbo

Authors
  1. Anita M. Hubley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno D. Zumbo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anita M. Hubley .

Editor information

Editors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, Vancouver, British Columbia, Canada

    Bruno D. Zumbo

  2. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, Vancouver, British Columbia, Canada

    Anita M. Hubley

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Hubley, A.M., Zumbo, B.D. (2017). Response Processes in the Context of Validity: Setting the Stage. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56129-5_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56128-8

  • Online ISBN: 978-3-319-56129-5

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation