Abstract
Purpose
Psychometric models and statistical techniques are cornerstones of research into latent structures of specific psychopathology and general mental health. We discuss “pivot points” for future research efforts from a psychometric epidemiology perspective, emphasising sampling and selection processes of both indicators that guide data collection as well as samples that are confronted with them.
Method
First, we discuss how a theoretical model of psychopathology determines which empirical indicators (questions, diagnoses, etc.) and modelling methods are appropriate to test its implications. Second, we deal with how different research designs introduce different (co-)variances between indicators, potentially leading to a different understanding of latent structures. Third, we discuss widening the range of statistical models available within the “psychometrics class”: the inclusion of categorical approaches can help to enlighten the debate on the structure of psychopathology and agreement on a minimal set of models might lead to greater convergence between studies. Fourth, we deal with aspects of methodology that introduce spurious (co-)variance in latent structure analysis (response styles, clustered data) and differential item functioning to gather more detailed information and to guard against over-generalisation of results, which renders assessments unfair.
Conclusions
Building on established insights, future research efforts should be more explicit about their theoretical understanding of psychopathology and how the analysis of a given indicator–respondent set informs this theoretical model. A coherent treatment of theoretical assumptions, indicators, and samples holds the key to building a comprehensive account of the latent structures of different types of psychopathology and mental health in general.
Similar content being viewed by others
Notes
This is in line with what is usually considered to be a “psychometric assessment”: a situation in which itemised categorical information is gathered on respondents to provide information on a number of non-observed (“latent”) properties with the aim of reaching a (often quantitative) statement about individual differences.
A topic that we ignore in this section comprises the differences between self-reported symptoms, observer-rated symptoms, and clinician-assessed diagnoses. In our view, the methodological challenges with respect to our discussion are the same across all three assessment strategies, while we agree that depending on the background of the reader these might have very different epistemological values for the debate on the structure of psychopathology.
References
Brodbeck J, Stulz N, Itten S, Regli D, Znoj H, Caspar F (2014) The structure of psychopathological symptoms and the associations with DSM-diagnoses in treatment seeking individuals. Compr Psychiatry 55:714–726
Carragher N, Krueger RF, Eaton NR, Slade T (2015). Disorders without borders: current and future directions in the meta-structure of mental disorders. Soc Psychiatry Psychiatr Epidemiol 50:339–350
Eaton NR, Rodriguez-Seijas C, Carragher N, Krueger RF (2015) Transdiagnostic factors of psychopathology and substance use disorders: a review. Soc Psychiatry Psychiatr Epidemiol 50:171–182
Lahey BB, Waldman ID (2012) Annual research review: phenotypic and causal structure of conduct disorder in the broader context of prevalent forms of psychopathology. J Child Psychol Psychiatry 53:536–557
Markon KE, Chmielewski M, Miller CJ (2011) The reliability and validity of discrete and continuous measures of psychopathology: a quantitative review. Psychol Bull 137:856–879
Goldberg D (2015) Psychopathology and classification in psychiatry. Soc Psychiatry Psychiatr Epidemiol 50:1–5
Krueger RF, Markon KE (2014) The role of the DSM-5 personality trait model in moving toward a quantitative and empirically based approach to classifying personality and psychopathology. Annu Rev Clin Psychol 10:477–501
Brown TA, Barlow DH (2009) A proposal for a dimensional classification system based on the shared features of the DSM-IV anxiety and mood disorders: implications for assessment and treatment. Psychol Assess 21:256–271
Brown TA, Chorpita BF, Barlow DH (1998) Structural relationships among dimensions of the DSM-IV anxiety and mood disorders and dimensions of negative affect, positive affect, and autonomic arousal. J Abnorm Psychol 107:179–192
Caspi A, Houts RM, Belsky DW, Goldman-Mellor SJ, Harrington HL, Israel S et al (2014) The p Factor: one general psychopathology factor in the structure of psychiatric disorders? Clin Psychol Sci 2:119–137
Wirth RJ, Edwards MC (2007) Item factor analysis: current approaches and future directions. Psychol Methods 12:58–79
Reise SP, Bonifay WE, Haviland MG (2013) Scoring and modeling psychological measures in the presence of multidimensionality. J Pers Assess 95:129–140
Castonguay LG, Barkham M, Lutz W, McAleavey A (2013) Practice-oriented research: approaches and applications. In: Lambert MJ (ed) Bergin and Garfield’s handbook of psychotherapy and behavior change, 6th edn. Wiley, Hoboken, pp 85–133
Kamphuis JH, Noordhof A (2009) On categorical diagnoses in “DSM-V”: cutting dimensions at useful points? Psychol Assess 21:294–301
Lutz W, Stulz N, Martinovich Z, Leon SC, Saunders SM (2009) Methodological background of decision rules and feedback tools for outcomes management in psychotherapy. Psychother Res 19:502–510
Edwards JR (2011) The fallacy of formative measurement. Organ Res Methods 14:370–388
Jarvis CB, MacKenzie SB, Podsakoff PM (2003) A critical review of construct indicators and measurement model misspecification in marketing and consumer research. J Consum Res 30:199–218
Willoughby M, Holochwost SJ, Blanton ZE, Blair CB (2014) Executive functions: formative versus reflective measurement. Measurement 12:69–95
Borsboom D, Mellenbergh GJ, van Heerden J (2003) The theoretical status of latent variables. Psychol Rev 110:203–219
Cohen P, Cohen J, Teresi J, Marchi M, Velez CN (1990) Problems in the measurement of latent variables in structural equations causal models. Appl Psychol Meas 14:183–196
Gaskin CJ, Happell B (2014) On exploratory factor analysis: a review of recent evidence, an assessment of current practice, and recommendations for future use. Int J Nurs Stud 51:511–521
Wright AGC, Krueger RF, Hobbs MJ, Markon KE, Eaton NR, Slade T (2013) The structure of psychopathology: toward an expanded quantitative empirical model. J Abnorm Psychol 122:281–294
(2014) What lies beneath. Nature 507:273. doi:10.1038/507273a
Markon KE (2010) Modeling psychopathology structure: a symptom-level analysis of Axis I and II disorders. Psychol Med 40:273–288
Cliff N, Hamburger CD (1967) The study of sampling errors in factor analysis by means of artificial experiments. Psychol Bull 68:430–445
Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ (1999) Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods 4:272–299
Böhnke JR, Lutz W, Delgadillo J (2014) Negative affectivity as a transdiagnostic factor in patients with common mental disorders. J Affect Disord 166:270–278
Fairhurst C, Böhnke JR, Gabe R, Croudace TJ, Tober G, Raistrick D (2014) Factor analysis of treatment outcomes from a UK specialist addiction service: relationship between the Leeds Dependence Questionnaire, Social Satisfaction Questionnaire and 10-item Clinical Outcomes in Routine Evaluation. Drug Alcohol Rev 33:643–650
Paap MCS, Meijer RR, Cohen-Kettenis PT, Richter-Appelt H, de Cuypere G, Kreukels BPC et al (2012) Why the factorial structure of the SCL-90-R is unstable: comparing patient groups with different levels of psychological distress using Mokken Scale Analysis. Psychiatry Res 200:819–826
Eaton NR, Krueger RF, Keyes KM, Wall M, Hasin DS, Markon KE et al (2013) The structure and predictive validity of the internalizing disorders. J Abnorm Psychol 122:86–92
El-Gabalawy R, Tsai J, Harpaz-Rotem I, Hoff R, Sareen J, Pietrzak RH (2013) Predominant typologies of psychopathology in the United States: a latent class analysis. J Psychiatr Res 47:1649–1657
Murphy J, Shevlin M, Adamson G (2007) A latent class analysis of positive psychosis symptoms based on the British Psychiatric Morbidity Survey. Personal Individ Differ 42:1491–1502
Vaidyanathan U, Patrick CJ, Iacono WG (2011) Patterns of comorbidity among mental disorders: a person-centered approach. Compr Psychiatry 52:527–535
Carragher N, Krueger R, Eaton N, Markon K, Keyes K, Blanco C et al (2014) ADHD and the externalizing spectrum: direct comparison of categorical, continuous, and hybrid models of liability in a nationally representative sample. Soc Psychiatry Psychiatr Epidemiol 49:1307–1317
Böhnke JR, Lutz W (2014) Using item and test information to optimize targeted assessments of psychological distress. Assessment 21:679–693
McAleavey AA, Nordberg SS, Hayes JA, Castonguay LG, Locke BD, Lockard AJ (2012) Clinical validity of the Counseling Center Assessment of Psychological Symptoms-62 (CCAPS-62): further evaluation and clinical applications. J Couns Psychol 59:575–590
Lambert MJ (2013) The efficacy and effectiveness of psychotherapy. In: Lambert MJ (ed) Bergin and Garfield’s handbook of psychotherapy and behavior change, 6th edn. Wiley, Hoboken, pp 169–218
Cahill J, Barkham M, Stiles WB, Twigg E, Rees A, Hardy GE et al (2006) Convergent validity of the CORE measures with measures of depression for clients in brief cognitive therapy for depression. J Couns Psychol 53:253–259
Raistrick D, Bradshaw J, Tober G, Weiner J, Allison J, Healey C (1994) Development of the Leeds Dependence Questionnaire (LDQ): a questionnaire to measure alcohol and opiate dependence in the context of a treatment evaluation package. Addict Abingdon Engl 89:563–572
Reininghaus U, Priebe S (2012) Measuring patient-reported outcomes in psychosis: conceptual and methodological review. Br J Psychiatry 201:262–267
Simms LJ, Prisciandaro JJ, Krueger RF, Goldberg DP (2012) The structure of depression, anxiety and somatic symptoms in primary care. Psychol Med 42:15–28
Derogatis LR (1993) BSI. Brief Symptom Inventory: administration, scoring, and procedures manual, 4th edn. Natl Comp Syst, Minneapolis
Adam D (2013) Mental health: on the spectrum. Nature 496(7446):416–418
Craddock N, Owen MJ (2010) The Kraepelinian dichotomy—going, going… but still not gone. Br J Psychiatry 196:92–95
Bühler J, Keller F, Läge D (2014) Activation as an overlooked factor in the BDI–II: a factor model based on core symptoms and qualitative aspects of depression. Psychol Assess 26:970–979
Sullivan PF, Kessler RC, Kendler KS (1998) Latent class analysis of lifetime depressive symptoms in the National Comorbidity Survey. Am J Psychiatry 155:1398–1406
Unick GJ, Snowden L, Hastings J (2009) Heterogeneity in comorbidity between major depressive disorder and generalized anxiety disorder and its clinical consequences. J Nerv Ment Dis 197:215–224
Derks EM, Allardyce J, Boks MP, Vermunt JK, Hijman R, Ophoff RA et al (2012) Kraepelin was right: a latent class analysis of symptom dimensions in patients and controls. Schizophr Bull 38:495–505
Reininghaus U, Priebe S, Bentall RP (2013) Testing the psychopathology of psychosis: evidence for a general psychosis dimension. Schizophr Bull 39:884–895
Russo M, Levine SZ, Demjaha A, Di Forti M, Bonaccorso S, Fearon P, et al. (2013) Association between symptom dimensions and categorical diagnoses of psychosis: a cross-sectional and longitudinal investigation. Schizophr Bull [Internet]. 2013 May 9; Available from: http://schizophreniabulletin.oxfordjournals.org/content/early/2013/05/06/schbul.sbt055.abstract
Cella D, Gershon R, Lai J-S, Choi S (2007) The future of outcomes measurement: item banking, tailored short-forms, and computerized adaptive assessment. Qual Life Res 16:133–141
Croudace TJ, Böhnke JR (2014) Item bank measurement of depression: will one dimension work? J Clin Epidemiol 67:4–6
Wahl I, Löwe B, Bjorner JB, Fischer F, Langs G, Voderholzer U et al (2014) Standardization of depression measurement: a common metric was developed for 11 self-report depression measures. J Clin Epidemiol 67:73–86
Reininghaus U, McCabe R, Burns T, Croudace T, Priebe S (2011) Measuring patients’ views: a bifactor model of distinct patient-reported outcomes in psychosis. Psychol Med 41:277–289
Reise SP (2012) The rediscovery of bifactor measurement models. Multivar Behav Res 47:667–696
Blanco C, Wall MM, He J-P, Krueger RF, Olfson M, Jin CJ et al (2015) The space of common psychiatric disorders in adolescents: comorbidity structure and individual latent liabilities. J Am Acad Child Adolesc Psychiatry 54:45–52
Henry JD, Crawford JR (2005) The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br J Clin Psychol 44:227–239
Page AC, Hooke GR, Morrison DL (2007) Psychometric properties of the Depression Anxiety Stress Scales (DASS) in depressed clinical samples. Br J Clin Psychol 46:283–297
Chen FF, Hayes A, Carver CS, Laurenceau J-P, Zhang Z (2012) Modeling general and specific variance in multifaceted constructs: a comparison of the bifactor model to other approaches. J Pers 80:219–251
Collins LM, Lanza ST (2010) Latent class and latent transition analysis. Wiley, Hoboken
Hagenaars JA, McCutcheon AL (eds) (2002) Applied latent class analysis. Cambridge University Press, Cambridge
Reunanen E, Suikkanen R (1999) Latent class analysis: wandering in latent space [Internet]. Konstanz: KOPS available from: http://kops.ub.uni-konstanz.de/volltexte/1999/117/ [retrieved 2009 Aug 31]
Kempf W. A pragmatic approach to Rasch-modeling: the loss of information index [Internet]. Berlin: Irena Regener Report No.: 74 available from: http://www.regener-online.de/books/diskuss_pdf/74.pdf [retrieved 2013 Aug 30]
Sellbom M (2014) A factor mixture model approach to elaborating on offender mental health classification with the MMPI–2–RF. J Pers Assess 96:293–305
Rost J (2001) The growing family of Rasch models. In: Boomsma A, van Duijn MAJ, Snijders TAB (eds) Essays on item response theory. Springer, New York, pp 25–42
Lubke GH, Muthén B (2005) Investigating population heterogeneity with factor mixture models. Psychol Methods 10:21–39
Hallquist MN, Wright AG (2014) Mixture modeling methods for the assessment of normal and abnormal personality, part I: cross-sectional models. J Pers Assess 96:256–268
Kim S-H, Beretvas SN, Sherry AR (2010) A validation of the factor structure of OQ-45 scores using factor mixture modeling. Meas Eval Couns Dev 42:275–295
Masyn KE, Henderson CE, Greenbaum PE (2010) Exploring the latent structures of psychological constructs in social development using the dimensional–categorical spectrum. Soc Dev 19:470–493
Eaton NR, Krueger RF, Docherty AR, Sponheim SR (2014) Toward a model-based approach to the clinical assessment of personality psychopathology. J Pers Assess 96:283–292
Moors G (2003) Diagnosing response style behavior by means of a latent-class factor approach: socio-demographic correlates of gender role attitudes and perceptions of ethnic discrimination reexamined. Qual Quant 37:277–302
Wetzel E, Böhnke JR, Carstensen CH, Ziegler M, Ostendorf F (2013) Do individual response styles matter? J Individ Differ 34:69–81
Subramaniam M, Abdin E, Vaingankar JA, Verma S, Chong SA (2014) Latent structure of psychosis in the general population: results from the Singapore mental health study. Psychol Med 44:51–60
Stochl J, Khandaker GM, Lewis G, Perez J, Goodyer IM, Zammit S, et al. (in press) Mood, anxiety and psychotic phenomena measure a common psychopathological factor. Psychol Med. doi:10.1017/S003329171400261X
Pickles A, Croudace T (2010) Latent mixture models for multivariate and longitudinal outcomes. Stat Methods Med Res 19:271–289
Wright AG, Hallquist MN (2014) Mixture modeling methods for the assessment of normal and abnormal personality, part II: longitudinal models. J Pers Assess 96:269–282
Podsakoff PM, MacKenzie SB, Lee J-Y, Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 88:879–903
Podsakoff PM, MacKenzie SB, Podsakoff NP (2012) Sources of method bias in social science research and recommendations on how to control it. Annu Rev Psychol 63:539–569
Tackett JL, Lahey BB, van Hulle C, Waldman I, Krueger RF, Rathouz PJ (2013) Common genetic influences on negative emotionality and a general psychopathology factor in childhood and adolescence. J Abnorm Psychol 122:1142–1153
Nisbett RE, Aronson J, Blair C, Dickens W, Flynn J, Halpern DF et al (2012) Intelligence: new findings and theoretical developments. Am Psychol 67:130–159
Just C (2011) A review of literature on the general factor of personality. Personal Individ Differ 50:765–771
Brown A, Ford T, Deighton J, Wolpert M (2014) Satisfaction in child and adolescent mental health services: translating users’ feedback into measurement. Adm Policy Ment Health 41:434–446
Hafner-Fink M, Uhan S (2013) Bipolarity and/or duality of social survey measurement scales and the question-order effect. Qual Quant 47:839–852
Weijters B, De Beuckelaer A, Baumgartner H (2014) Discriminant validity where there should be none: positioning same-scale items in separated blocks of a questionnaire. Appl Psychol Meas 38:450–463
Baumgartner H, Steenkamp J-BEM (2001) Response styles in marketing research: a cross-national investigation. J Mark Res 38:143–156
Bolt DM, Johnson TR (2009) Addressing score bias and differential item functioning due to individual differences in response style. Appl Psychol Meas 33:335–352
Van Vaerenbergh Y, Thomas TD (2013) Response styles in survey research: a literature review of antecedents, consequences, and remedies. Int J Public Opin Res 25:195–217
Weijters B, Geuens M, Schillewaert N (2010) The individual consistency of acquiescence and extreme response style in self-report questionnaires. Appl Psychol Meas 34:105–121
Weijters B, Geuens M, Schillewaert N (2010) The stability of individual response styles. Psychol Methods 15:96–110
Wetzel E, Carstensen CH, Böhnke JR (2013) Consistency of extreme response style and non-extreme response style across traits. J Res Personal 47:178–189
Hoyt WT, Kerns M-D (1999) Magnitude and moderators of bias in observer ratings: a meta-analysis. Psychol Methods 4:403–424
Hoyt WT (2000) Rater bias in psychological research: when is it a problem and what can we do about it? Psychol Methods 5:64–86
Julian MW (2001) The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Struct Equ Model 8:325–352
Pornprasertmanit S, Lee J, Preacher KJ (2014) Ignoring clustering in confirmatory factor analysis: some consequences for model fit and standardized parameter estimates. Multivar Behav Res 49:518–543
Satorra A, Muthen B (1995) Complex sample data in structural equation modeling. Sociol Methodol 25:267–316
Stochl J, Jones PB, Perez J, Khandaker GM, Böhnke JR, Croudace TJ (in press) Effects of ignoring clustered data structure in confirmatory factor analysis of ordered polytomous items: a simulation study based on PANSS. Int J Methods Psychiatr Res (accepted)
Meijer RR, Egberink IJL (2012) Investigating invariant item ordering in personality and clinical scales: some empirical findings and a discussion. Educ Psychol Meas 72:589–607
Sturt E (1981) Hierarchical patterns in the distribution of psychiatric symptoms. Psychol Med 11:783–794
Teresi JA, Ocepek-Welikson K, Kleinman M, Eimicke JP, Crane PK, Jones RN et al (2009) Analysis of differential item functioning in the depression item bank from the Patient Reported Outcome Measurement Information System (PROMIS): an item response theory approach. Psychol Sci Q 51:148–180
Crane P, Gibbons L, Ocepek-Welikson K, Cook K, Cella D, Narasimhalu K et al (2007) A comparison of three sets of criteria for determining the presence of differential item functioning using ordinal logistic regression. Qual Life Res 16:69–84
Penfield RD, Algina J (2006) A generalized DIF effect variance estimator for measuring unsigned differential test functioning in mixed format tests. J Educ Meas 43:295–312
Woods CM (2009) Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivar Behav Res 44:1–27
Uebelacker LA, Strong D, Weinstock LM, Miller IW (2009) Use of item response theory to understand differential functioning of DSM-IV major depression symptoms by race, ethnicity and gender. Psychol Med 39:591–601
Eaton NR (2014) Transdiagnostic psychopathology factors and sexual minority mental health: evidence of disparities and associations with minority stressors. Psychol Sex Orientat Gend Divers 1:244–254
Barnes D, Keyes K, Bates L (2013) Racial differences in depression in the United States: how do subgroup analyses inform a paradox? Soc Psychiatry Psychiatr Epidemiol 48:1941–1949
Helms-Lorenz M, Van de Vijver FJ (1995) Cognitive assessment in education in a multicultural society. Eur J Psychol Assess 11:158–169
Schnohr CW, Kreiner S, Due EP, Currie C, Boyce W, Diderichsen F (2008) Differential item functioning of a family affluence scale: validation study on data from HBSC 2001/02. Soc Indic Res 89:79–95
Sunderland M, Mewton L, Slade T, Baillie AJ (2010) Investigating differential symptom profiles in major depressive episode with and without generalized anxiety disorder: true co-morbidity or symptom similarity? Psychol Med 40:1113–1123
Unternaehrer E, Luers P, Mill J, Dempster E, Meyer AH, Staehli S et al (2012) Dynamic changes in DNA methylation of stress-associated genes (OXTR, BDNF) after acute psychosocial stress. Transl Psychiatry 2:e150
Ptak C, Petronis A (2010) Epigenetic approaches to psychiatric disorders. Dialogues Clin Neurosci 12(1):25
Miller GA, Rockstroh B (2013) Endophenotypes in psychopathology research: where do we stand? Annu Rev Clin Psychol 9:177–213
Crane PK, Narasimhalu K, Gibbons LE, Pedraza O, Mehta KM, Tang Y et al (2008) Composite scores for executive function items: demographic heterogeneity and relationships with quantitative magnetic resonance imaging. J Int Neuropsychol Soc 14:746–759
Meyer-Lindenberg A, Tost H (2012) Neural mechanisms of social risk for psychiatric disorders. Nat Neurosci 15:663–668
Miller GA, Elbert T, Sutton BP, Heller W (2007) Innovative clinical assessment technologies: challenges and opportunities in neuroimaging. Psychol Assess 19:58–73
Schmidt NB, Kotov R, Joiner TE (2004) Taxometrics: toward a new diagnostic scheme for psychopathology. American Psychological Association, Washington, DC
Cramer AOJ, Waldorp LJ, van der Maas HLJ, Borsboom D (2010) Comorbidity: a network perspective. Behav Brain Sci 33:137–150
Eaton N (2015) Latent variable and network models of comorbidity: toward an empirically derived nosology. Soc Psychiatry Psychiatr Epidemiol 20:1–5
Pickles A, Angold A (2003) Natural categories or fundamental dimensions: on carving nature at the joints and the rearticulation of psychopathology. Dev Psychopathol 15:529–551
Box GEP (1979) Some problems of statistics and everyday life. J Am Stat Assoc 74:1–4
Everitt BS, Dunn G (2001) Multivariate data analysis, 2nd edn. Wiley, Chichester
Conflict of interest
The authors declare that they have no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Böhnke, J.R., Croudace, T.J. Factors of psychological distress: clinical value, measurement substance, and methodological artefacts. Soc Psychiatry Psychiatr Epidemiol 50, 515–524 (2015). https://doi.org/10.1007/s00127-015-1022-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00127-015-1022-5