Skip to main content
SpringerLink
Log in

How Cognitive Interviewing can Provide Validity Evidence of the Response Processes to Scale Items

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

The current theory about validity reflected in the Standards for Educational and Psychological Testing (AERA et al. in Standards for educational and psychological testing, American Psychological Association, Washington, DC, 1999), offers no clear indications about the methods for gathering validity evidence about the response processes. Cognitive interviewing (CI) can play an important role answering the current demand about empirical and theoretical analyses of the response processes as a source of validity evidence in psychological testing. CI can provide validity evidence for investigating substantive aspects of construct validity and for contributing to the explanations for item and test scores (Zumbo in Handbook of statistics, vol 26, Elsevier, Amsterdam, pp. 45–79, 2007; The concept of validity: revisions, new directions and applications, IAP—Information Age Publishing Inc., Charlotte, NC, pp. 65–82, 2009). The aim of the study was to illustrate the use of cognitive interviewing method for gathering validity evidence on response processes. The search for evidence about the “response process” was guided by an argument-based approach to validity (Kane in Psychological Bulletin 1992; Educational measurement, American Council on Education/Praeger, Washington, DC, pp. 17–64, 2006). 21 cognitive interviews were carried out during the cognitive testing of the APGAR psychological scale intended to measure the “family support” construct. Cognitive interviewing provided validity evidence that explains how respondents interpret and respond to the APGAR items. Respondents maintained a shared interpretation of “family concept” while answering the APGAR scale items. Nevertheless, they included in the concept of family not only family members they live with but also other family members and even friends. CI participants were also capable of classifying their answers about the family support perception following a polythomous response system. Lastly, the role of CI in the Kane’s argument-based approach and Zumbo’s contextualized view of validity will be discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

    Google Scholar 

  • Beatty, P., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly. doi:10.1093/poq/nfm006.

  • Bellón, J. A., Delgado, J., Luna, P., & Lardelli, P. (1996). Validez y fiabilidad del cuestionario de función familiar Apgar-familiar. Atención Primaria, 18, 289–296.

    Google Scholar 

  • Bickart, B., & Felcher, M. (1996). Expanding and enhancing the use of verbal protocols in survey research. In N. Schwarz & S. Sudman (Eds.), Answering question. Methodology for determining cognitive and communicative processes in survey research (pp. 115–142). San Francisco: Jossey-Bass.

    Google Scholar 

  • Boeije, H., & Willis, G. B. (2011). The cognitive interviewing reporting framework (CIRF): Incorporating the principles of qualitative research. Paper presented at the Fourth Conference of the European Survey Research Association (ESRA), Lausanne, Switzerland. Abstract retrieved from http://surveymethodology.eu/conferences/lausanne-2011/presentation/226/.

  • Castillo, M., Padilla, J. L., Gómez, J., & Andrés, A. (2010). A productivity map of cognitive pretest methods for improving survey questions. Psicothema, 22(3), 475–481.

    Google Scholar 

  • Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument-based approach to validity make a difference? Educational Measurement: Issues and Practice. doi: 10.1111/j.1745-3992.2009.00165.x.

  • Conrad, F. G., & Blair, J. (2009). Sources of error in cognitive interviews. Public Opinion Quarterly. doi:10.1093/poq/nfp01.

  • Conrad, F. G, Brown, N. R., & Cashman, E. (1998). Strategies for estimating behavioural frequency in survey interviews. Memory. doi:10.1080/741942603.

  • DeVellis, R. F. (2003). Scale development: Theory and application (2nd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • DeWalt, D. A., Rothrock, N., Yount, S., & Stone, A. A. (2007). Evaluation of item candidates the PROMIS qualitative item review. Medical Care. doi:10.1097/01.mlr.0000254567.79743.e2.

  • Ding, L., Reay, N. W., Lee, A., & Bao, L. (2009). Are we asking the right questions? Validating clicker question sequences by student interviews. American Journal of Physics. doi: 10.1119/1.3116093.

  • Ercikan, K., Arim, R., & Law, D. (2010). Application on think aloud protocols for examining and confirming sources of differential item functioning identified by experts review. Educational Measurement: Issues and Practices, 29(2), 24–35.

    Article  Google Scholar 

  • Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2011). Investigating the substantive aspect of construct validity for the satisfaction with life scale adapted for children: A focus on cognitive processes. Social Indicator Research. doi:10.1007/s11205-010-9603-x.

  • Huber, G. L. (2008). AQUAD, software for the analysis of qualitative data (version 6.8.1.1). Ingeborg Huber Verlag, University of Tübingen, Germany.

  • Irwin, D. E., Varni, J. W., Yeatts, K., & DeWalt D. A. (2009). Cognitive interviewing methodology in the development of a pediatric item bank: a patient reported outcomes measurement information system (PROMIS) study. Health and Quality of Life Outcomes. doi:10.1186/1477-7525-7-3.

  • Kane, M. (1992). An argument-based approach to validity. Psychological Bulletin. doi:10.1037/0033-2909.112.3.527.

  • Kane, M. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Washington, DC: American Council on Education/Praeger.

    Google Scholar 

  • Kane, M. (2010). Validity and fairness. Language Testing. doi: 10.1177/0265532209349467.

  • Knaff, K., Deatrick, J., Gallo, A., Holcombe, G., Bakitas, M., Dixon, J., & Grey, M. (2007). The analysis and interpretation of cognitive interviews for instrument development. Research in Nursing and Health. doi:10.1002/nur.20195.

  • Lissitz, R. (2009). The concept of validity: revisions, new directions and applications. Charlotte, NC: Information Age Publishing In.

    Google Scholar 

  • Llosa, L. (2008). Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments. Educational Measurement: Issues and Practice. doi:10.1111/j.1745-3992.2008.00126.x.

  • Menon, G. (1993). The effects of accessibility of information in memory on judgments of behavioral frequencies. Journal of Consumer Research, 20(3), 431–440.

    Article  Google Scholar 

  • Messick, S. (1989). Validity. In R. Linn (Ed.), Educational measurement (3rd ed., pp. 3–103). Washington, DC: American Council on Education.

    Google Scholar 

  • Messick, S. (1995). Validity of psychological assessments: Validation of inferences from person’s responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.

    Article  Google Scholar 

  • Olt, H., Jirwe, M., Gustavsson, P., & Emami, A. (2010). Psychometric evaluation of the swedish adaptation of the Inventory for Assessing the Process of Cultural Competence among Healthcare Professional Revised (IAPCC-R). Journal of Transcultural Nursing. doi:10.1177/1043659609349064.

  • Padilla, J. L., Gómez, J., Hidalgo, M. D., & Muñiz, J. (2007). Esquema conceptual y procedimientos para analizar la validez de las consecuencias del uso de los test. Psicothema, 19(1), 173–178.

    Google Scholar 

  • Poole, H. M., Murphy, P., & Nurmikko, T. J. (2009). Development and preliminary validation of the NePIQoL: A quality-of-life measure for neuropathic pain. Journal of Pain and Symptom Management. doi:10.1016/j.jpainsymman.2008.01.012.

  • Sijtsma, K. (2009). Correcting fallacies in validity, reliability and clasification. International Journal of Testing. doi:10.1080/15305050903106883.

  • Sireci, S. G. (2005). Validity theory and application. In B. S. Everitt & D. C. Howell (Eds.), Encyclopedia of statistics in behavioral science (pp. 2103–2107). Nueva, Jersey: Wiley.

    Google Scholar 

  • Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 19–37). Charlotte, NC: Information Age Publishing Inc.

    Google Scholar 

  • Smilkstein, G. (1978). The family APGAR: A proposal for family function test and its use by physicians. Journal of Family Practice, 6, 1231–1239.

    Google Scholar 

  • Smilkstein, G., Ashworth, C., & Montano, D. (1982). Validity and reliability of the family APGAR as a test of family function. Journal of Family Practice, 15, 303–311.

    Google Scholar 

  • Spanish Ministry of Health and Consume. (2006). Encuesta Nacional de Salud de España 2006 [National Health Survey in Spain, 2006]. Madrid: Ministerio de Sanidad y Consumo. http://www.ine.es/jaxi/menu.do?L=0&type=pcaxis&path=%2Ft15/p419&file=inebase (March, 13 2008).

  • Tourangeau, R. (1984). Cognitive science and survey methods: A cognitive perspective. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.

    Google Scholar 

  • Tourangeau, R., Rips, R. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Willis, G. B. (2005). Cognitive interviewing. Thousand Oaks: Sage Publications.

    Google Scholar 

  • Willis, G. B., Royston, P., & Bercini, D. (1991). The use of verbal report methods in the development and testing of survey questionnaires. Applied Cognitive Psychology, 5, 251–267.

    Article  Google Scholar 

  • Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 45–79)., Psychometrics Amsterdam: Elsevier.

    Google Scholar 

  • Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: revisions, new directions and applications (pp. 65–82). Charlotte, NC: IAP—Information Age Publishing Inc.

    Google Scholar 

  • Zumbo, B. D., & Shear B. R. (2011). The concept of validity and some novel validation methods. Paper presented at the 42nd Annual Meeting of the Northeastern Educational Research Association, Rocky Hill, CT. Abstract retrieved from http://www.nera-education.org/Full_2011_NERA_Program.pdf.

Download references

Acknowledgments

This study was funded by the Andalusia Regional Government under the Excellent Research Fund (Project nº SEJ-5188).

Author information

Authors and Affiliations

  1. Department of Methodology of Behavioral Sciences, School of Psychology, University of Granada, Campus de Cartuja, 18071, Granada, Spain

    Miguel Castillo-Díaz & José-Luis Padilla

Authors
  1. Miguel Castillo-Díaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José-Luis Padilla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel Castillo-Díaz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Castillo-Díaz, M., Padilla, JL. How Cognitive Interviewing can Provide Validity Evidence of the Response Processes to Scale Items. Soc Indic Res 114, 963–975 (2013). https://doi.org/10.1007/s11205-012-0184-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-012-0184-8

Keywords

Search

Navigation