Does the Length of a Questionnaire Matter?
Expected and Unexpected Answers From Generalizability Theory
Abstract
Short personality questionnaires are increasingly used in research and practice, with some scales including as few as two to five items per personality domain. Despite the frequency of their use, these short scales are often criticized on the basis of their reduced internal consistencies and their purported failure to assess the breadth of broad constructs, such as the Big 5 factors of personality. One reason for this might be the use of principles routed in Classical Test Theory during test construction. In this study, Generalizability Theory is used to compare psychometric properties of different scales based on the NEO-PI-R and BFI, two widely-used personality questionnaire families. Applying both Classical Test Theory (CTT) and Generalizability Theory (GT) allowed to identify the inner workings of test shortening. CTT-based analyses indicated that longer is generally better for reliability, while GT allowed differentiation between reliability for relative and absolute decisions, while revealing how different variance sources affect test score reliability estimates. These variance sources differed with scale length, and only GT allowed clear description of these internal consequences, allowing more effective identification of advantages and disadvantages of shorter and longer scales. Most importantly, the findings highlight the potential error proneness of focusing solely on reliability and scale length in test construction. Practical as well as theoretical consequences are discussed.
References
(2001a). Generalizability theory. Berlin, Germany: Springer.
(2010). An other perspective on personality: Meta-analytic integration of observers’ accuracy & predictive validity. Psychological Bulletin, 136, 1092–1122.
(1992). Revised NEO personality inventory (NEO-PI-R) and NEO Five-Factor inventory (NEO-FFI): Professional manual. Odessa, FL: Psychological Assessment Resources.
(2003). Bibliography for the revised NEO personality inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO-FFI). Lutz, FL: Psychological Assessment Resources.
(1949). Essentials of psychological testing. New York, NY: Harper.
(1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
(2007). On the consistency of individual classification using short scales. Psychological Methods, 12, 105.
(1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670.
(1990). An alternative “description of personality”: The Big-Five factor structure. Journal of Personality and Social Psychology, 59, 1216–1229.
(2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37, 504–528.
(2008). Life goals matter to happiness: A revision of set-point theory. Social Indicators Research, 86, 213–231.
(2006). Assessing the Big-Five personality domains via short forms. European Journal of Psychological Assessment, 22, 139–148. doi: 10.1027/1015-5759.22.3.139
(1991). The “Big Five” Inventory–Versions 4a and 54. Berkeley, CA: University of California, Berkeley, Institute of Personality and Social Research.
(2012). Test length and decision quality in personnel selection: When is short too short? International Journal of Testing, 12, 321–344.
(2013). Assessing individual change using short tests and questionnaires. Applied Psychological Measurement. doi: 10.1177/0146621613510061.
(2010). Individual differences as predictors of work, educational, and broad life outcomes. Personality and Individual Differences, 49, 331–336. doi: 10.1016/j.paid.2010.03.042
(2005). The relative contribution of trait and social influences to the links among perceived social support, affect, and self-esteem. Journal of Personality, 73, 361–388.
(2010). Investigating an invariant item ordering for polytomously scored items. Educational and Psychological Measurement, 70, 578–595.
(1954). The attenuation paradox in test theory. Psychological Bulletin, 51, 493–504. doi: 10.1037/h0058543
(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
(2001). Sources of structure: Genetic, environmental, and artifactual influences on the covariation of personality traits. Journal of Personality, 69, 511–535.
(2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28–50. doi: 10.1177/1088868310366253
(2002). Transgression-related motivational dispositions: Personality substrates of forgiveness and their links to the Big Five. Personality and Social Psychology Bulletin, 28, 1556.
(2004). NEO-PI-R. NEO Persönlichkeitsinventar nach Costa und McCrae. Revidierte Fassung
[NEO-PI-R. NEO Personality Inventory] . Göttingen, Germany: Hogrefe.(2010). How similar are personality scales of the “same” construct? A meta-analytic investigation. Personality and Individual Differences, 49, 669–676. doi: 10.1016/j.paid.2010.06.014
(2009). A meta-analysis of the Five-Factor model of personality and academic performance. Psychological Bulletin, 135, 322–338. doi: 10.1037/a0014996
. (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria: R Development. ISBN 3-900051-07-0, URL www.R-project.org/
(2007). The 10-Item Big Five inventory: Norm values and investigation of sociodemographic effects based on a German population representative sample. European Journal of Psychological Assessment, 23, 193–201. doi: 10.1027/1015-5759.23.3.193
(2005). Kurzversion des Big Five Inventory (BFI-K): Entwicklung und Validierung eines ökonomischen Inventars zur Erfassung der fünf Faktoren der Persönlichkeit
[Short version of the Big Five Inventory (BFI-K): Development and validation of an economic inventory for assessment of the five factors of personality] . Diagnostica, 51, 195–206.(2007). Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41, 203–212.
(2011). psych: Procedures for personality and psychological research. Evanston, IL: Northwestern University. personality-project.org/r/psych.manual.pdf, 1.1.12.
(2006). Generalizability theory. In , Handbook of complementary methods in education research (pp. 309–322). Mahwah, NJ: Erlbaum.
(1989). Generalizability Theory. American Psychologist, 44(6), 922–932.
(2014). Assessing the validity and reliability of a quick scan for student’s evaluation of teaching. Results from confirmatory factor analysis and G Theory. Studies in Educational Evaluation. doi: 10.1016/j.stueduc.2014.03.001.
(1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251.
(2009). Construct validity: Advances in theory and methodology annual review of clinical psychology, Vol. 5, (pp. 1–25). Palo Alto, CA: Annual Reviews.
. (2010). EDUG user guide. Retrieved from www.irdp.ch/edumetrie/logicielfrancais.htm
(2011). Comparative validity of brief to medium-length Big Five and Big Six personality questionnaires. Psychological Assessment, 23, 995–1009. doi: 10.1037/a0024165
(2013). A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. Psychological Assessment, 25, 94–104. doi: 10.1037/a0029061
(1982). Reconsideration of the “attenuation paradox” – and some new paradoxes in test validity. Journal of Experimental Education, 50, 164–171.
(2013). Assessing the reliability of change: A comparison of two measures of adult attachment. Journal of Research in Personality, 47, 202–208. doi: 10.1016/j.jrp. 2013.01.005
(2013). Getting entangled in the nomological net. European Journal of Psychological Assessment, 29, 157–161. doi: 10.1027/1015-5759/a000173
(2009). Modeling socially desirable responding and its effects. Educational and Psychological Measurement, 69, 548.
(2010). Predicting academic success with the Big 5 rated from different points of view: Self-rated, other rated and faked. European Journal of Personality, 24, 341–355. doi: 10.1002/per.753
(2009). Latent state-trait theory: An application in sport psychology. Psychology of Sport and Exercise, 10, 344–349.
