Skip to main content
Log in

Order Effects in Batteries of Questions

  • Published:
Quality and Quantity Aims and scope Submit manuscript

Abstract

Batteries of questions with identical response items are commonly used in survey research. This paper suggests that question order has the potential to cause systematic positive or negative bias on responses to all questions in a battery. Whilst question order effects have been studied for many decades, almost no attention has been given to this topic. The primary aim is to draw attention to this effect, to demonstrate its possible magnitude, and to discuss a range of mechanisms through which it might occur. These include satisficing, anchoring and cooperativeness. The effect seems apparent in the results of a recent survey. This was a survey of Emergency Department patients presenting to Wollongong Hospital (Australia) with apparently less urgent conditions in 2004. Two samples were taken. Question order was fixed in the first sample (n = 104; response rate RR2 = 94%), but randomised in the second sample (n = 46; response rate RR2 = 96%). Respondents were asked to indicate whether each of 18 reasons for presenting to the ED was a ‘very important reason’ a ‘moderately important reason’ or ‘not a reason’ The mean number of very important reasons selected was 56% higher in the first sample as compared to the second sample.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alwin D.F., Krosnick J.A. (1991). The reliability of survey attitude measurement: the influence of question and respondent attributes. Sociological Methods and Research 20:139–181

    Article  Google Scholar 

  • American Association for Public Opinion Research (2004) Standard Definitions: Final Disposition of Case Codes and Outcome Rates for Surveys, 3rd edn. Lenexa, Kansas: AAPOR. Available online at http://www.aapor.org (accessed July 2005).

  • Dillman, D. A., Smyth, J. D., Christian, L. M. & Stern, M. J. (2003). Multiple Answer Questions in Self-Administered Surveys: The Use of Check-All-That-Apply and Forced-Choice Question Formats Paper presented at the annual meeting of the American Statistical Association, San Fransisco, CA.

  • Duffy B. (2003). Response order effects-how do people read?. International Journal of Market Research 45: 457–475

    Google Scholar 

  • Grice, H. P. (1975). Logic and conversation. In: P. Cole, & J. L. Morgan (eds.), Syntax and Semantics, Vol. 3. Speech Acts. New York: Academic Press, pp. 41–48.

  • Knauper B. (1999). The impact of age and education on response order effects in attitude measurement. Public Opinion Quarterly 63:347–370

    Article  Google Scholar 

  • Krosnick J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5: 213–236

    Article  Google Scholar 

  • Krosnick J.A., Alwin D.F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly 51: 201–219

    Article  Google Scholar 

  • McClendon M. J. (1986). Response order effects for dichotomous questions. Social Science Quarterly 67: 205–211

    Google Scholar 

  • Narayan S., Krosnick J.A. (1996). Education moderates some response effects in attitude measurement. Public Opinion Quarterly 51: 201–219

    Google Scholar 

  • Schwarz N. (1996). Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Lawrence Erlbaum Associates, New Jersey

    Google Scholar 

  • Schwarz, N. & Hippler, H. J. (1991). Response alternatives: the impact of their choice and presentation order. In: P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (eds.), Measurement Error in Surveys. Wiley Series in Probability and Mathematical Statistics New York: Wiley, pp. 41–46.

  • Siminski P., Cragg S., Middleton R., Masso M., Lago L., Green J., Eagar K. (2005). Primary care patients’ views on why they present to Emergency Departments—inappropriate attendances or inappropriate policy?. Australian Journal of Primary Health 11: 87–95

    Article  Google Scholar 

  • Sudman S., Bradburn N.M., Schwarz N. (1996). Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. Jossey-Bass, San Fransisco

    Google Scholar 

  • Tourangeau R. (1984). Cognitive science and survey methods. In: Jabine T., Straf M., Tanur J., Tourangeau R. (eds). Cognitive Aspects of Survey Methodology: Building a Bridge between Disciplines. National Academy Press, Washington, DC, pp. 73–100

    Google Scholar 

  • Tourangeau R., Rasinski K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin 103: 299–314

    Article  Google Scholar 

  • Tourangeau R., Rips L.J., Rasinski K. (2000). The Psychology of Survey Response. Cambridge University Press, Cambridge

    Google Scholar 

  • Wanke M, Schwarz N. (1997). Reducing question order effects: the operation of buffer items. In: Lyberg L.E., Biemer P.P., Collins M., de Leeuw E., Dippo C., Trewin D. (eds). Survey Measurement and Process Quality. Wiley, New York, pp. 115–139

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Peter Siminski.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Siminski, P. Order Effects in Batteries of Questions. Qual Quant 42, 477–490 (2008). https://doi.org/10.1007/s11135-006-9054-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-006-9054-2

Keywords

Navigation