Abstract
QUAID (question-understanding aid) is a software tool that assists survey methodologists, social scientists, and designers of questionnaires in improving the wording, syntax, and semantics of questions. The tool identifies potential problems that respondents might have in comprehending the meaning of questions on questionnaires. These problems can be scrutinized by researchers when they revise questions to improve question comprehension and, thereby, enhance the reliability and validity of answers. QUAID was designed to identify nine classes of problems, but only five of these problems are addressed in this article: unfamiliar technical term, vague or imprecise relative term, vague or ambiguous noun phrase, complex syntax, and working memory overload. We compared the output of QUAID with ratings of language experts who evaluated a corpus of questions on the five classes of problems. The corpus consisted of 505 questions on 11 surveys developed by the U.S. Census Bureau. Analyses of hit rates, false alarm rates,d′ scores, recall scores, and precision scores revealed that QUAID was able to identify these five problems with questions, although improvements in QUAID’s performance are anticipated in future research and development.
Article PDF
Similar content being viewed by others
References
Abney, S. (1997).The SCOL manual (Version 0.1b). Unpublished manuscript, University of Tübingen (www.sfs.nphil.uni-tuebingen.de/Tilde abney/).
Allen, J. (1995).Natural language understanding. Redwood City, CA: Benjamin/Cummings.
Baddeley, A. D. (1986).Working memory. New York: Oxford University Press.
Bickart, B., &Felcher, E. M. (1996). Expanding and enhancing the use of verbal protocols in survey research. In N. Schwarz & S. Sudman (Eds.),Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 115–142). San Francisco: Jossey-Bass.
Brill, E. (1995). Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging.Computational Linguistics,21, 1–24.
Cannell, C. F., Miller, P. V., &Oksenberg, L. (1981). Research on interviewing techniques. In S. Leinhardt (Ed.),Sociological methodology (pp. 389–437). San Francisco: Jossey-Bass.
Coltheart, M. (1981). The MRC psycholinguistic database.Quarterly Journal of Experimental Psychology,33A, 497–505.
DARPA (1995).Proceedings of the Sixth Message Understanding Conference (MUC-6). San Francisco: Morgan Kaufman.
Fowler, F. J. (1993).Survey research methods. Newbury Park, CA: Sage.
Fowler, F. J., &Cannell, C. F. (1996). Using behavioral coding to identify cognitive problems with survey questions. In N. Schwarz & S. Sudman (Eds.),Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 15–36). San Francisco: Jossey-Bass.
Francis, W. N., &Kučera, H. (1982).Frequency analysis of English usage. Boston: Houghton-Mifflin.
Goldman, S. R., Varma, S., &Cote, N. (1996). Extending capacity-constrained construction integration: Toward “smarter” and flexible models of text comprehension. In B. F. Britton & A. C. Graesser (Eds.),Models of understanding text (pp. 73–114). Hillsdale, NJ: Erlbaum.
Graesser, A. C., Baggett, W., &Williams, K. (1996). Questiondriven explanatory reasoning.Applied Cognitive Psychology,10, S17-S32.
Graesser, A. C., Bommareddy, S., Swamer, S., &Golding, J. M. (1996). Integrating questionnaire design with a cognitive computational model of human question answering. In N. Schwartz & S. Sudman (Eds.),Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 143–174). San Francisco: Jossey-Bass.
Graesser, A. C., &Franklin, S. P. (1990). QUEST: A cognitive model of question answering.Discourse Processes,13, 279–303.
Graesser, A. C., Gordon, S. E., &Brainerd, L. E. (1992). QUEST: A model of question answering.Computers & Mathematics with Applications,23, 733–745.
Graesser, A. C., &Hemphill, D. (1991). Question answering in the context of scientific mechanisms.Journal of Memory & Language,30, 186–209.
Graesser, A. C., Kennedy, T., Wiemer-Hastings, P., &Ottati, V. (1999). The use of computational cognitive models to improve questions on surveys and questionnaires. In M. G. Sirken, D. J. Hermann, S. Schechter, N. Schwarz, J. M. Tanur, & R. Tourangeau (Eds.),Cognition and survey methods research (pp. 199–216). New York: Wiley.
Graesser, A. C., Lang, K. L., &Roberts, R. M. (1991). Question answering in the context of stories.Journal of Experimental Psychology: General,120, 254–277.
Groves, R. M. (1989).Survey errors and survey costs. New York: Wiley.
Jacobs, P. S. (Ed.) (1992).Text-based intelligent systems: Current research and practice in information extraction and retrieval. Hillsdale, NJ: Erlbaum.
Jobe, J. B., &Mingay, D. J. (1991). Cognition and survey measurement: History and overview.Applied Cognitive Psychology,5, 175–192.
Just, M., &Carpenter, P. (1992). A capacity theory of comprehension: Individual differences in working memory.Psychological Review,99, 122–149.
Kintsch, W. (1998).Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press.
Lehnert, W. G. (1997). Information extraction: What have we learned?Discourse Processes,23, 441–470.
Lessler, J. T., &Forsyth, B. H. (1996). A coding system for appraising questionnaires. In N. Schwarz & S. Sudman (Eds.),Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 259–291). San Francisco: Jossey-Bass.
Lessler, J. T., &Kalsbeek, W. (1993).Nonsampling error in surveys. New York: Wiley.
Lessler, J. T., &Sirken, M. G. (1985). Laboratory-based research on the cognitive aspects of survey methodology: The goals and methods of the National Center for Health Statistics study.Milbank Memorial Fund Quarterly/Health & Society,63, 565–581.
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., &Miller, K. (1990).Five papers on WordNet (Rep. No. 43). Princeton, NJ: Princeton University, Cognitive Science Laboratory.
Moxey, L. M., &Sanford, A. J. (in press). Communicating quantities: A review of psycholinguistic evidence of how expressions determine perspectives.Applied Cognitive Psychology.
Sanford, A. J., Moxey, L. M., &Paterson, K. B. (1996). Attentional focusing with quantifiers in production and comprehension.Memory & Cognition,24, 144–155.
Schober, M. F., &Conrad, F. G. (1997). Does conversational interviewing reduce survey measurement error?Public Opinion Quarterly,60, 576–602.
Schwarz, N., &Sudman, S. (Eds.) (1996).Answering questions: Methodology for determining cognitive and communicative processes in survey research. San Francisco: Jossey-Bass.
Sirken, M. G., &Fuchsberg, R. (1984). Laboratory-based research on the cognitive aspects of survey methodology. In T. B. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.),Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 183–197). Washington, DC: National Academy Press.
Sirken, M. G., Hermann, D. J., Schechter, S., Schwarz, N., Tanur, J. M., &Tourangeau, R. (Eds.) (1999).Cognition and survey methods research. New York: Wiley.
Sudman, S., Bradburn, N. M., &Schwarz, M. (1995).Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.
Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. J. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.),Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy of Sciences.
Willis, G., Royston, P., &Bercini, D. (1991). The use of verbal report methods in the development and testing of survey questionnaires.Applied Cognitive Psychology,5, 251–267.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was partially funded by grants from the U.S. Census Bureau (43-YA-BC-802930) and the National Science Foundation (SBR 9720314 and SBR 9977969). Previous versions of the QUAID tool had different names: QQEA (QUEST Questionnaire Evaluation Aid) and Cochlea. We thank Scott Allen for his feedback on an earlier draft of this article.
Rights and permissions
About this article
Cite this article
Graesser, A.C., Wiemer-hastings, K., Kreuz, R. et al. QUAID: A questionnaire evaluation aid for survey methodologists. Behavior Research Methods, Instruments, & Computers 32, 254–262 (2000). https://doi.org/10.3758/BF03207792
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03207792