Skip to main content
SpringerLink
Log in

Intra-individual Response Variability as an Indicator of Insufficient Effort Responding: Comparison to Other Indicators and Relationships with Individual Differences

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Surveys are one of the most popular ways to collect employee information. Because of their widespread use, data quality is an increasingly important concern. The purpose of this paper is to (1) introduce the intra-individual response variability (IRV) index as an easily calculated and flexible way to detect insufficient effort responding (IER); (2) examine the extent to which various IER indices detect the same or different respondents engaging in IER behavior; and (3) investigate relationships between individual differences and commonly used IER indices to better understand systematic and theoretically relevant IER behavior. In a two-part study, 199 undergraduates responded to questionnaires online, and various IER indices were calculated. The IRV index identifies different respondents than other IER indices. Values on the IRV index (as well as other IER indices) are related to scores on theoretically meaningful individual differences in conscientiousness, agreeableness, and boredom proneness. This study provides researchers with a robust, easily calculated, and flexible means for screening questionnaire data for IER behavior. Practical recommendations for finding and making decisions about IER behavior patterns are provided. This study introduces the IRV index, an extension of the long string, used to identify survey research participants who likely engaged in one type of IER behavior. It is also one of the first studies to evaluate the extent to which IER indices identify different respondents as having engaged in IER and provides additional evidence that values on these indices are related to individual differences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allen, N. J., & Meyer, J.P. (1990). The measurement and antecedents of affective, continuance, and normative commitment to the organization. Journal of Occupational Psychology, 63, 1–18.

  • Anseel, F., Lievens, F., Schollaert, E., & Choragwicka, B. (2010). Response rates in organizational science, 1995–2008: a meta-analytic review and guidelines for survey researchers. Journal of Business and Psychology, 25, 335–349.

    Article  Google Scholar 

  • Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: black-white differences in response style. Public Opinion Quarterly, 48, 491–509.

    Article  Google Scholar 

  • Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48, 5–37.

    Article  PubMed  Google Scholar 

  • Bardo, J. W., Yeager, S. J., & Klingsporn, M. J. (1982). Preliminary assessment of format-specific central tendency and leniency error in summated rating scales. Perceptual and Motor Skills, 54, 227–234.

    Article  Google Scholar 

  • Barrick, M. A., & Mount, M. K. (1991). The big five personality dimensions and job performance: a meta-analysis. Personnel Psychology, 44, 1–26.

    Article  Google Scholar 

  • Barrick, M. R., Mitchell, T. R., & Stewart, G. L. (2003). Situational and motivational influences on trait-behavior relationships. In M. R. Barrick & A. M. Ryan (Eds.), Personality and work: reconsidering the role of personality in organizations (pp. 60–82). San Francisco: Jossey-Bass.

    Google Scholar 

  • Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMP 2 random responding indices: validation using a self-report methodology. Psychological Assessment, 4, 340–345.

    Article  Google Scholar 

  • Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218–229.

    Article  PubMed  Google Scholar 

  • Christian, L.M., Dillman, D.A., & Smyth, J.D. (2008). The effects of mode and format on answers to scalar questions in telephone and web surveys. Advances in telephone survey methodology, 250–275.

  • Costa Jr., P. T., & McCrae, R. R. (2008). The revised NEO personality inventory (NEO-PI-R). In G. J. Boyle, G. Matthews, & D. H. Saklofske (Eds.), The sage handbook of personality theory and assessment: personality measurement and testing (pp. 179–198). London: Sage.

    Chapter  Google Scholar 

  • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171–181.

    Article  Google Scholar 

  • Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.

    Google Scholar 

  • Farmer, R., & Sundberg, N. D. (1986). Boredom proneness: the development and correlates of a new scale. Journal of Personality Assessment, 50, 4–17.

    Article  PubMed  Google Scholar 

  • Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg: Tilburg University Press.

    Google Scholar 

  • Greene, R. L. (1978). Am empirically derived MMPI carelessness scale. Journal of Clinical Psychology, 34, 407–410.

    Article  Google Scholar 

  • Guay, R. P., Oh, I. S., Choi, D., Mitchell, M. S., Mount, M. K., & Shin, K. (2013). The interactive effect of conscientiousness and agreeableness on job performance dimensions in South Korea. International Journal of Selection and Assessment, 21, 233–238.

    Article  Google Scholar 

  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.

    Article  Google Scholar 

  • Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.

    Article  Google Scholar 

  • Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

    Article  Google Scholar 

  • Little, R. J. A., & Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: treatment of no-shows using Rubin’s causal model. Psychological Methods, 3, 147–159.

    Article  Google Scholar 

  • Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.

    Article  Google Scholar 

  • McGonagle, A. K., Huang, J. L., & Walsh, B. M. (2015). Insufficient effort survey responding: an under-appreciated problem in work and Organisational Health Psychology research. Applied Psychology, 65, 287–321. doi:10.1111/apps.12058.

    Article  Google Scholar 

  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–457.

    Article  PubMed  Google Scholar 

  • Merritt, S. M. (2012). The two-factor solution to Allen and Meyer’s (1990) affective commitment scale: effects of negatively worded items. Journal of Business and Psychology, 27, 421–436.

    Article  Google Scholar 

  • Nichols, D. S., Greene, R. L., & Schmolck, P. (1989). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials. Journal of Clinical Psychology, 45, 239–250.

    Article  PubMed  Google Scholar 

  • Organ, D. W., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional predictors of organizational citizenship behavior. Personnel Psychology, 48, 775–802.

    Article  Google Scholar 

  • Rogelberg, S. G., & Stanton, J. M. (2007). Introduction: understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195–209.

    Article  Google Scholar 

  • Skowronski, M. (2012). When the bored behave badly (or exceptionally). Personnel Review, 41, 143–159.

    Article  Google Scholar 

  • Watt, J. D., & Hargis, M. B. (2010). Boredom proneness: its relationship with subjective underemployment, perceived organizational support, and job performance. Journal of Business and Psychology, 25, 163–174.

    Article  Google Scholar 

  • Weiner, S.P., & Dalessio, A.T. (2006). Oversurveying: Causes, consequences, and cures. Getting action from organizational surveys: New concepts, methods, and applications, 294–311.

  • Woods, C. M. (2006). Careless responding to reverse-worded items: implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186–191. doi:10.1007/s10862-005-9004-7.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Organizational Science, University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC, 28223-0001, USA

    Alexandra M. Dunn

  2. Department of Psychology, University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC, 28223-0001, USA

    Eric D. Heggestad

  3. Department of Psychology, University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC, 28223-0001, USA

    Linda R. Shanock

  4. 6330 Quadrangle Drive, Chapel Hill, NC, 27517, USA

    Nels Theilgard

Authors
  1. Alexandra M. Dunn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric D. Heggestad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linda R. Shanock
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nels Theilgard
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexandra M. Dunn.

Appendix A. Catch Question Index Used in the Present Study.

Appendix A. Catch Question Index Used in the Present Study.

Scoring: Scoring recommendations are based on the use of a 5-point Likert-type response scale from 1 = Strongly Disagree to 5 = Strongly Agree). For items without an * designation, responses other than Strongly Disagree or Disagree are given one point for the Catch Question Index. For the items denoted by an *, responses other than Strongly Agree or Agree are given one point for the Catch Question Index.

Note: Item 11 is specific to a college student population.

  1. 1.

    I lie 100 % of the time.

  2. 2.

    I am under three years old.

  3. 3.

    I have traveled around the world twenty three times.

  4. 4.

    I have been to the moon.

  5. 5.

    I own Starbucks.

  6. 6.

    I am ninety-nine years old.

  7. 7.

    I wrote the questions on this survey.

  8. 8.

    I graduated high school at age eight.

  9. 9.

    I am the president of the United States.

  10. 10.

    I have been eaten by a shark.

  11. 11.

    I have never been to a college class before.

  12. 12.

    I can walk on water.

  13. 13.

    I have never laughed before.

  14. 14.

    I am the richest person in the world.

  15. 15.

    I know that the sky is blue.*

  16. 16.

    I have never used a computer before.

  17. 17.

    I eat food.*

  18. 18.

    I have seen a moving car.*

  19. 19.

    I know how to count to ten.*

  20. 20.

    I know how to spell my name.*

  21. 21.

    I know the months of the year.*

  22. 22.

    I have never been mad before.

  23. 23.

    I never smile.

  24. 24.

    I do not know what dancing is.

  25. 25.

    I have never used a pen before.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dunn, A.M., Heggestad, E.D., Shanock, L.R. et al. Intra-individual Response Variability as an Indicator of Insufficient Effort Responding: Comparison to Other Indicators and Relationships with Individual Differences. J Bus Psychol 33, 105–121 (2018). https://doi.org/10.1007/s10869-016-9479-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-016-9479-0

Keywords

Search

Navigation