Evaluating the Construct Validity of Objective Personality Tests Using a Multitrait-Multimethod-Multioccasion-(MTMM-MO)-Approach
Abstract
Although Objective Personality Tests (OPTs) have a long history in psychology and the field of psychological assessment, their validity, and reliability have not yet been sufficiently studied. In this study, we examined the convergent and discriminant validity of objective (personality) tests, Implicit Association Tests (IATs), and self-report measures for the assessment of conscientiousness and intelligence. Moreover, the convergent and discriminant validity of these measures was assessed on the trait (stable) and occasion specific (momentary) level by using the multimethod latent state-trait (MM-LST) model proposed by Courvoisier, Nussbeck, Eid, and Cole (2008) which allows for the decomposition of different sources of variance. Data from 367 students assessed on three different measurement occasions was incorporated. Results indicate generally low convergence of OPTs with data gained by other approaches. Additional analyses revealed that the OPTs used assess stable rather than momentary components of the constructs. Reliabilities of different tests ranged from .54 to .95. Furthermore, a substantial amount of trait method specificity revealed that different methods assess trait components that are not shared between OPTs and other measures. Data on the criterion validity of the objective conscientiousness test revealed that it is related to the punctuality of test takers in the laboratory.
References
1974). Construction of composite measures by the canonical factor regression method. In , Sociological methodology 1973–1974 (pp. 51–78). San Francisco, CA: Jossey-Bass.
(2001). Intelligenz-Struktur-Test 2000 R
([Test of intelligence structure 2000 R] . Göttingen, Germany: Hogrefe.2009). Unraveling the three faces of self-esteem: A new information-processing sociometer perspective. Journal of Research in Personality, 43, 933–937. doi: 10.1016/j.jrp.2009.04.002
(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
(1995). EQS structural equations program manual. Encino, CA: Multivariate Software.
(1987). Practical issues in structural modeling. Sociological Methods & Research, 17, 78–117.
(2003). Self-enhancement tendencies among people with high self-esteem: The moderating role of implicit self-esteem. Self and Identity, 2, 169–187. doi: 10.1080/15298860309029
(2005). Decision-making deficits of Korsakoff patients in a new gambling task with explicit rules: Associations with executive functions. Neuropsychology, 19, 267–277.
(2008). The role of strategies in deciding advantageously in ambiguous and risky situations. Cognitive Processing, 9, 159–173.
(2002). Test d2 Aufmerksamkeits-Belastungs-Test. Manual. 9., überarbeitete und neu normierte Auflage
([The d2 Test of Attention. Manual] . (9th ed.). Göttingen, Germany: Hogrefe.1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
(1993). Human cognitive abilities. New York, NY: Cambridge University Press.
(1890). Mental tests and measurements. Mind, 15, 373–381.
(1941). An objective test of character-temperatment: I. Journal of General Psychology, 25, 59–73.
(1944). An objective test of character-temperatment: II. Journal of Social Psychology, 19, 99–114.
(1946). Description and measurement of personality. New York, NY: World Book.
(1957). Personality and motivation structure and measurement. New York, NY: World Book.
(1968). What is “objective” in “objective personality tests”? In , Readings in psychological tests and measurements (pp. 163–168). Homewood, IL: Dorsey Press.
(1977). The scientific analysis of personality and motivation. London, UK: Academic Press.
(1967). Objective personality and motivation tests: A theoretical introduction and practical compendium. Chicago, IL: University of Illinois Press.
(1992). Neo PI-R professional manual. Odessa, FL: Psychological Assessment Resources.
(2006). Unfolding the constituents of psychological scores: Development and application of mixture and multitrait-multimethod LST models (Doctoral dissertation), University of Geneva, Switzerland. Retrieved from doc.rero.ch/record/9221/files/these.pdf
. (2008). Analyzing the convergent and discriminant validity of states and traits: Development and applications of multimethod latent state-trait models. Psychological Assessment, 20, 270–280. doi: 10.1037/a0012812
(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
(2012). Discrepancies between implicit and explicit self-concepts of intelligence predict performance on tests of intelligence. European Journal of Personality, 26, 212–220.
(2010). Convergence of direct, indirect, and objective risk taking measures in the domain of gambling: The moderating role of impulsiveness and self-control. Journal of Psychology, 218, 20–27.
(2005). Temporal stability of the Implicit Association Test-Anxiety. Journal of Personality Assessment, 84, 82–88.
(2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241–261.
(2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38–60.
(2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230–253.
(1996). Effects of dissimulation on self-report and objective measures of personality. Personality and Individual Differences, 21, 335–343.
(2003). Implicit measures in social cognition research: Their meaning and uses. Annual Review of Psychology, 54, 297–327.
(2006). Non-normal and categorical data in structural equation modeling. In , Structural equation modeling: A second course (pp. 269–314). Greenwich, CT: Information Age Publishing.
(2008). When and why do implicit measures predict behaviour? Empirical evidence for the moderating role of opportunity, motivation, and process reliance. European Review of Social Psychology, 19, 285–338. doi: 10.1080/10463280802556958
(2012). A comparison of method effects in two confirmatory factor models for structurally different methods. Structural Equation Modeling, 19, 409–436. doi: 10.1080/10705511.2012.687658
(2014). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses. Behaviour Research Methods. doi: 10.3758/s13428-014-0457-z
(1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480.
(2003). Understanding and using the Implicit Association Test I: An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197–216.
(2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41. doi: 10.1037/a0015575
(2008). Differential stability: The effects of acute and chronic construct accessibility on the temporal stability of the Implicit Association Test. Journal of Individual Differences, 29, 70–79. doi: 10.1027/1614-0001.29.2.70
(1975). Objektive Testbatterie, OA-TB 75
([Objective Test Battery, OA-TB 75] . Weinheim, Germany: Beltz.1979). Über die Verfälschbarkeit von Persönlichkeitsfragebogen und Objektiven Persönlichkeitstests unter SD-Instruktion und in einer Auslesesituation
([About fakeability of personality questionnaires under SD-instruction and in a situation of selection] . Diagnostica, 25, 7–23.2005). A meta-analysis on the correlation between the implicit association test and explicit self-report measures. Personality and Social Psychology Bulletin, 31, 1369–1385. doi: 10.1177/0146167205275613
(2008). Implicit and explicit attitudes and interracial interaction: The moderating role of situationally available control resources. Group Processes & Intergroup Relations, 11, 69–87.
(2008). Working memory capacity and self-regulatory behavior: Towards an individual differences perspective on behavior determination by automatic versus controlled processes. Journal of Personality and Social Psychology, 95, 962–977.
(2000). Design and evaluation of an adaptive matrices test. Diagnostica, 46, 182–188.
(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424–453. doi: 10.1037/1082-989x.3.4.424
(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi: 10.1080/10705519909540118
(1965). Personality factors in objective test devices. San Diego, CA: Knapp.
(2006). The single category implicit association test as a measure of implicit social cognition. Journal of Personality and Social Psychology, 91, 16–32.
(1996). Arbeitshaltungen – Kurze Testbatterie: Anspruchsniveau, Frustrationstoleranz, Leistungsmotivation, Impulsivität/Reflexivität
([Working Style – A short test-battery: Level of aspiration, achievement motivation, frustration tolerance, impulsiveness/reflexiveness] . Frankfurt/M.: Swets Test Services [Software and Manual]2003). The balloon analogue risk task (BART) differentiates smokers and nonsmokers. Experimental and Clinical Psychopharmacology, 11, 26–33.
(2002). Evaluation of a behavioral measure of risk taking: The Balloon Analogue Risk Task (BART). Journal of Experimental Psychology-Applied, 8, 75–84. doi: 10.1037/1076-898x.8.2.75
(2002, September). Unterschiedliche Definitionen von Methodeneffekten in MTMM Modellen und ihre Implikationen für die Analyse der Validität
([Different definitions of method effects in MTMM models and their implications for validity analysis] . Paper presented at the 43rd meeting of the German Psychological Association, Berlin, Germany2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, 151–173. doi: 10.1207/s15328007sem0902_1
(2012). Risky decision making in adults with ADHD. Journal of Behavior Therapy and Experimental Psychiatry, 43, 938–946.
(1998–2012). Mplus user’s guide (3rd eds.). Los Angeles, CA: Muthén & Muthén.
(2006). Analysing multitrait-multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results? British Journal of Mathematical & Statistical Psychology, 59, 195–213. doi: 10.1348/00071100sx67490
(2007). Standortbestimmung und Zukunft Objektiver Persönlichkeitstests.
([Current situation and future of objective personality tests] Report Psychologie, 32, 64–75.2006). Belastbarkeits-Assessment: Computerisierte Objektive Persönlichkeits-Testbatterie – Deutsch (BAcO-D).
([Stress resistance assessment: Computerized Objective Test-battery – German version] Frankfurt/M.: Harcourt Assessment [Test: Software und Manual]2014). Objective Personality Tests. Manuscript submitted for publication
(1994). Zur Taxonomie deutscher Dispositionsbegriffe. In
([A taxonomy of German dispositional concepts] Handbuch deutschsprachiger Wortnormen[Handbook of German word norms] . (pp. 382–441). Göttingen, Germany: Hogrefe.2003). NEO-Persönlichkeitsinventar nach Costa und McCrae, Revidierte Fassung (NEO-PI-R)
([NEO-Personality Inventory revisited by Costa and McCrae] . Göttingen, Germany: Hogrefe.2010). Modeling common traits and method effects in multitrait-multimethod analysis. Multivariate Behavioral Research, 45, 45–72.
(2008). Modelling method effects as individual causal effects. Journal of the Royal Statistical Society Series A, 171, 41–63.
(2007). Convergence of conventional and behavior-based measures: Towards a multimethod approach in the assessment of vocational interests. Psychology Science Quarterly, 49, 168–183.
(2007). Assessing behavior in standardized settings: The role of objective personality tests. International Journal of Clinical and Health Psychology, 7, 537–546.
(2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing Retrieved from www.R-project.org
. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Nielsen & Lydiche.
(2004). Underlying processes in the Implicit Association Test: Dissociating salience from associations. Journal of Experimental Psychology: General, 133, 139–165.
(2010). Can we predict risk-taking behavior? Two behavioral tests for predicting guessing tendencies in a multiple-choice test. European Journal of Psychological Assessment, 26, 87–94.
(2006). The objective assessment of personality: Cattells’s T-data revisited and more. Psychology Science, 48, 53–68.
(2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research-Online, 8, 23–74. doi: citeulike-article-id:7222182
(1975). Objektive Persönlichkeitsmessung in diagnostischer und klinischer Psychologie
([Objective personality measurement in psychological assessment and clinical psychology] . Weinheim, Germany: Beltz.2004). Objektiver Leistungsmotivations Test (OLMT)
([Objective Achievement Motivation Test] . Mödling, Austria: Dr. G. Schuhfried [Software and Manual]2007). Objektiver Leistungsmotivations Test (OLMT)
([Objective Achievement Motivation Test] Mödling, Austria: Dr. G. Schuhfried [Software and Manual]2006). Conceptual, theoretical, and historical foundations of multimethod assessment. In , Handbook of multimethod measurement in psychology (pp. 9–25). New York, NY: American Psychological Association.
(in press ). A model of moderated convergence between direct, indirect, and behavioral measures of personality traits. In , Behavior based assessment: Going beyond self report in the personality, affective, motivation, and social domains. Göttingen, Germany: Hogrefe.2008). Validity of the five-factor model for the implicit self-concept of personality. European Journal of Psychological Assessment, 24, 263–272. doi: 10.1027/1015-5759.24.4.263
(2005). A latent state-trait analysis of implicit and explicit personality measures. European Journal of Psychological Assessment, 21, 100–107. doi: 10.1027/1015-5759.21.2.100
(1973). Cross-media independence of questionnaire and objective test personality factors. Multivariate Behavioral Research, 8, 23–40.
(2006). Predicting spontaneous big five behavior with implicit association tests. European Journal of Psychological Assessment, 22, 13–20. doi: 10.1027/1015-5759.22.1.13
(1990). Structural model evaluation and modification - An interval estimation approach. Multivariate Behavioral Research, 25, 173–180. doi: 10.1207/s15327906mbr2502_4
(1999). Latent state-trait theory and research in personality and individual differences. European Journal of Personality, 13, 389–408.
(2004). Reflective and impulsive determinants of social behavior. Personality and Social Psychology Review, 8, 220–247.
(2000). A model of dual attitudes. Psychological Review, 107, 101–126.
(2007). Faking susceptibility of different measurement methods: Questionnaire, semi-projective, and objective. Psychology Science, 49, 291–307.
(2010). Investigating Measures of Achievement Motivation(s). Journal of Individual Differences, 31, 15–21. doi: 10.1027/1614-0001/a000002
(