Abstract
This article puts forward the case that survey questionnaires, which are a type of measuring instrument, can and should be tested to ensure they meet their purpose. Traditionally survey researchers have been pre-occupied with 'standardising' data collection instruments and procedures such as question wording and have assumed that experience in questionnaire design, coupled with pilot testing of questionnaires, will then ensure valid and reliable results. However, implicit in the notion of standardisation are the assumptions that respondents are able to understand the questions being asked, that questions are understood in the same way by all respondents, and that respondents are willing and able to answer such questions. The development of cognitive question testing methods has provided social researchers with a number of theories and tools to test these assumptions, and to develop better survey instruments and questionnaires. This paper describes some of these theories and tools, and argues that cognitive testing should be a standard part of the development process of any survey instrument.
Similar content being viewed by others
References
Fowler FJ. ImprovingSurvey Questions: Design and Evaluation. Applied Social Research Methods Series: 38. Thousand Oaks: Sage, 1995.
Fowler FJ, Mangione TW. Standardized Survey Interviewing: Minimizing Interviewer-Related Error. Applied Social Research Methods Series: 18. Newbury Park: Sage, 1990.
Oksenberg L, Cannell C, Kalton G. New strategies for pretesting survey questions. J Off Stat 1991; 7(3): 349–365.
Belson WA. The Design and Understanding of Survey Questions. Aldershot: Gower, 1981.
Clark HH, Schober MF. Asking questions and influencing answers. In: Tanur JM (ed.) Questions About Questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation, 1992: 15–48.
Tourangeau R, Rips LJ, Rasinski K. The Psychology of Survey Response. Cambridge: Cambridge University Press, 2000.
Krosnick JA. Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol 1991; 5: 213–236.
Krosnick JA, Alwin DF. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opin Quart 1987; 51: 201–219.
Cannell CF, Oksenberg L, Converse JM (eds). Experiments in Interviewing Techniques: Field Experiments in Health Reporting, 1971-1977. Survey Research Centre, Institute of Social Research, University of Michigan. Ann Arbor, MI: University of Michigan, 1979.
Cantril H. Gauging Public Opinion. Princeton, NJ: Princeton University Press, 1944.
Payne SL. The Art of Asking Questions. Princeton, NJ: Princeton University Press, 1951.
Rugg D. Experiments in wording questions: II. Public Opin Quart. 1941; 5: 91–92.
Sudman S, Bradburn NM. Response Effects in Surveys: A Review and Synthesis. Chicago: Aldine, 1974.
Jabine T, Straf M, Tanur J, Tourangeau R (eds). Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines. Washington, DC: National Academy Press, 1984.
Hippler HJ, Schwarz N, Sudman S (eds). Social Information Processing and Survey Methodology. New York: Springer-Verlag, 1987.
Tanur JM. Looking backwards and forwards at the CASM movement. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 13–19
Aborn M. CASM revisited. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 21–38.
Tourangeau R. Cognitive sciences and survey methods. In: Jabine T, Straf M, Tanur J, Tourangeau R (eds). Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines. Washington, DC: National Academy Press, 1984: 73–100.
Esposito J, Jobe J. A General Model of the Survey Interaction Process. Bureau of the Census Seventh Annual Research Conference Proceedings. Washington DC: US Bureau of the Census, 1991: 537–560.
Schober MF. Making sense of questions: an interactional approach. In: Sirken, MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 77–93.
Suchman L, Jordan B. Validity and the collaborative construction of meaning in face-to-face surveys. In: Tanur JM (ed.) Questions about Questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation, 1992: 241–267.
Sudman S, Bradburn NM, Schwarz N. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco: Josey-Bass, 1996.
Tulving E, Thomson DM. Encoding specificity and retrieval processes in episodic memory. Psychol Rev 1973; 80: 352–373.
Anderson J. The Architecture of Cognition. Cambridge, MA: Harvard University Press, 1983.
Loftus E, Palmer J. Reconstruction of automobile destruction: An example of the interaction between language and memory. J Verb Learn Verb Behav 1974; 13: 585–589
Jobe JB, Tourangeau R, Smith AF. Contributions of survey research to the understanding of memory. Appl Cogn Psychol 1993; 7: 567–584.
Blair EA, Burton S. Cognitive processes used by survey respondents to answer behavioral frequency questions. J Consum Res 1987; 14: 280–288.
Sudman S, Schwarz N. Contributions of cognitive psychology to advertising research. J Advertising Res 1989; 29: 43–53.
Tversky A, Kahneman D. Availability: A heuristic for judging frequency and probability. Cogn Psychol 1973; 5: 207–232.
Tversky A, Kahneman D. Judgement under uncertainty: Heuristics and biases. Science 1974; 185: 1124–1131.
Schwarz N, Hippler HJ. Response alternatives: The impact of their choice and presentation order. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds), Measurement Errors in Surveys. New York: Wiley, 1991: 41–56.
Czaja, R. Questionnaire pretesting comes of age. Market Bull 1998; 9: 52–66.
Forsyth BH, Lessler JT. Cognitive laboratory methods: A taxonomy. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds) Measurement Errors in Surveys. New York: Wiley, 1991: 393–418.
Jobe JB, Mingay DJ. (1991) Cognition and survey measurement: History and overview. Appl Cogn Psychol 1991; 5: 175–192.
Schwarz N. Questionnaire design: The rocky road from concepts to answers. In: LybergL, Biemer P, Collins M, de Leeuw E, Dippo C, Schwarz N, Trewin D (eds), Survey Measurement and Process Quality. New York: Wiley, 1997: 29–45.
Willis GB. Cognitive Interviewing and Questionnaire Design: A training manual. Hyattsville, MD: National Centre for Health Statistics, 1994.
Wilson TD, LaFleur SJ, Anderson DA. The validity and consequences of verbal reports about attitudes. In: Schwarz N, Sudman S (eds), Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: Jossey-Bass, 1995: 91–114.
Willis GB, DeMaio T, Harris-Kojetin B. Is the bandwagon headed to the methodological promised land? Evaluating the validity of cognitive interviewing techniques. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 133–153.
Presser S, Blair J. Survey pretesting: Do different methods produce different results? In: Marsden PV (ed), Sociological Methodology. Beverley Hills, CA: Sage, 1994; 24: 73–104.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Collins, D. Pretesting survey instruments: An overview of cognitive methods. Qual Life Res 12, 229–238 (2003). https://doi.org/10.1023/A:1023254226592
Issue Date:
DOI: https://doi.org/10.1023/A:1023254226592