Skip to main content
SpringerLink
Log in

Evidence Based on Relations to Other Variables: Bolstering the Empirical Validity Arguments for Constructs

  • Chapter
  • First Online:
Instrument Development in the Affective Domain

Abstract

In this chapter, the concept of validity is examined using evidence based on the relation of constructs within the instrument to constructs that are external to the instrument. This chapter addresses two major categories of validity evidence based on these external relationships. The first is what has historically been referred to as construct validity, which includes analyses of convergent and divergent validity. This first part of the chapter is dedicated to discussing the methodological framework (correlations and multitrait-multimethod matrices) and statistical techniques [structural equation modeling (SEM)] needed to quantify these relationships. The second half of the chapter discusses what has commonly been referred to as criterion validity and includes evidence with external variables that is often predictive in nature. The final section discusses the complex tasks of gathering incremental validity evidence and gathering evidence for use of the instrument with other populations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Much of the section on Structural Equation Modeling is adapted from McCoach (2003).

  2. 2.

    A researcher who is conducting a mean structure analysis or a growth curve analysis would need the means for all of the observed variables as well as the variance/covariance matrix. However, under normal circumstances, the variance/covariance matrix serves as the sufficient statistic for a SEM analysis.

  3. 3.

    Technically, it is considered proper form to analyze a covariance matrix, but under a variety of conditions analyzing a correlation matrix will produce the same results, as a correlation matrix is simply a standardized version of a covariance matrix.

  4. 4.

    If there are n observed variables in the model, there are (n (n + 1))/2 unique elements in the variance/covariance matrix.

  5. 5.

    There are other estimation methods, but they are beyond the scope of this chapter. For more information about alternative estimation methods, see Kaplan (2009) or Hoyle (1995).

References

  • Allport, G. W., Vernon, P. E., & Gardner, L. (1960). Study of values. Oxford, England: Houghton Mifflin.

    Google Scholar 

  • American Educational Research Association (AERA), American Psychological Association (APA) & National Council on Measurement in Education (NCME). (1999). The standards for educational and psychological testing. Washington: American Educational Research Association.

    Google Scholar 

  • Anderson, R. E., Barnes, G. E., & Murray, R. P. (2011). Psychometric properties and long-term predictive validity of the Addiction-Prone Personality (APP) scale. Personality and Individual Differences, 50(5), 651–656.

    Article  Google Scholar 

  • Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182.

    Article  PubMed  Google Scholar 

  • Beck, C. T., & Gable, R. K. (2000). Postpartum depression screening scale: Development and psychometric testing. Nursing Research, 49, 272–282.

    Article  PubMed  Google Scholar 

  • Beck, C. T., Gable, R. K. (2002). Postpartum depression screening scale. Los Angeles: Western Psychological Services.

    Google Scholar 

  • Beck, C. T. (1992). The lived experience of postpartum depression: A phenomenological study. Nursing Research, 41, 166–170.

    Article  PubMed  Google Scholar 

  • Beck, C. T. (1993). Teetering on the edge: A substantive theory of postpartum depression. Nursing Research, 42, 42–48.

    Article  PubMed  Google Scholar 

  • Beck, C. T. (1995). The effects of postpartum depression on maternal-infant interaction: A meta-analysis. Nursing Research, 44(5), 298–304.

    Article  PubMed  Google Scholar 

  • Beck, C. T. (1996). Postpartum depressed mothers’ experiences interacting with their children. Nursing Research, 45, 98–104.

    Article  PubMed  Google Scholar 

  • Beck, C. T., & Gable, R. K. (2001). Further validation of the postpartum depression screening scale. Nursing Research, 50, 155–164.

    Google Scholar 

  • Beck, A. T., Steer, R. A., & Brown, G. K. (1996). BDI-II manual. San Antonio: The Psychological Corporation.

    Google Scholar 

  • Bennett, G. K., Seashore, H. G., & Westman, A. G. (1997). The differential aptitude test. San Antonio, Texas: Psychological Corporation.

    Google Scholar 

  • Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.

    Google Scholar 

  • Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Boutlton, M. J., & Smith, P. K. (1994). Bully/victim problems in middle school children: Stability, self-perceived competence, peer acceptance. British Journal of Developmental Psychology, 12, 315–325.

    Article  Google Scholar 

  • Bovaird, J. A., & Koziol, N. A. (2012). Measurement models for ordered-categorical indicators. In R. Hoyle (Ed.), Handbook of structural equation modeling (pp. 495–511). New York: The Guilford Press.

    Google Scholar 

  • Campbell, D. P. (1973). The strong vocational interest blank for men. In D. G. Zytowski (Ed.), Contemporary approaches to interest measurement. Minneapolis: University of Minnesota Press.

    Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.

    Article  PubMed  Google Scholar 

  • Campbell, D. T., & O’Connell, E. J. (1967). Method factors in multitrait-multimethod matrices: Multiplicative rather than additive? Multivariate Behavioral Research, 2, 409–426.

    Article  Google Scholar 

  • Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Beverly Hills: Sage.

    Google Scholar 

  • Cox, J. L., Holden, J. M., & Sagovsky, R. (1987). Detection of postnatal depression: Development of the 10-item Edinburgh Postnatal Depression Scale. British Journal of Psychiatry, 150, 782–786.

    Article  PubMed  Google Scholar 

  • Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.). Washington, DC: American Council on Education.

    Google Scholar 

  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.

    Article  PubMed  Google Scholar 

  • Duncan, T. E., Duncan, S. C., Strycker, L. A., Li, F., & Alpert, A. (1999). An introduction to latent variable growth curve modeling: Concepts, issues, and applications. Mahwah: Erlbaum.

    Google Scholar 

  • Edwards, A. L. (1959). Edwards personal preference schedule manual. New York: Psychological Corp.

    Google Scholar 

  • Edwards, M. C., Wirth, R. J., Houts, C. R., & Xi, N. (2012). Categorical data in the structural equation modeling framework. In R. Hoyle (Ed.), Handbook of structural equation modeling (pp. 195–208). New York: Guilford Press.

    Google Scholar 

  • Eid, M., & Nussbeck, F. W. (2009). The multitrait-multimethod matrix at 50! Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 71.

    Article  Google Scholar 

  • Gable, R. K. (1970). A multivariate study of work value orientations. Unpublished doctoral dissertation, State University of New York at Albany.

    Google Scholar 

  • Gordon, L. V. (1960). Survey of interpersonal values. Chicago: Science Research Associates.

    Google Scholar 

  • Grimm, K. J., & Widaman, K. F. (2012). Construct validity. In H. Cooper (Ed.), APA handbook of research methods in psychology. Washington, DC: APA.

    Google Scholar 

  • Hawker, D. S., & Boulton, M. J. (2000). Twenty years’ research on peer victimization and psychosocial maladjustment: A meta-analytic review of cross-sectional studies. Journal of Child Psychology and Psychiatry, 41, 441–455.

    Article  PubMed  Google Scholar 

  • Haynes, S. N., & Lench, H. C. (2003). Incremental validity of new clinical assessment measures. Psychological Assessment, 15(4), 456–466.

    Article  PubMed  Google Scholar 

  • Hovling, V., Schermelleh-Engel, K., & Moosbrugger, H. (2009). Analyzing multitrait-multimethod data: A comparison of three approaches. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 99–111.

    Article  Google Scholar 

  • Hoyle, R. H. (Ed.). (1995). Structural equation modeling: Concepts, issues, and applications. Thousand Oaks: Sage Publications.

    Google Scholar 

  • Kaplan, D. (2000). Structural Equation Modeling: Foundations and Extensions. Newbury Park, CA: Sage.

    Google Scholar 

  • Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). New York: Sage Publications.

    Google Scholar 

  • Kenny, D. A. (1976). An empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology, 65, 507–516.

    Google Scholar 

  • Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165–172.

    Article  Google Scholar 

  • Kenny, D. A., Kashy, D. A., & Bolger, N. (1998). Data analysis in social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), Handbook of social psychology (4th ed., pp. 233–265). New York: McGraw Hill.

    Google Scholar 

  • Kline, R. B. (2010). Principles and practice of structural equation modeling (3rd ed.). New York: The Guilford Press.

    Google Scholar 

  • Kuder, G. F. (1949). Manual for the Kuder preference record (personal). Chicago: Science Research Associates.

    Google Scholar 

  • Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7(2), 228–244.

    Article  PubMed  Google Scholar 

  • Maas, C. J. M., Lensvelt-Mulders, G. J. L. M., & Hox, J. J. (2009). A multilevel multitrait-multimethod analysis. Methodology, 5, 72–77.

    Google Scholar 

  • Marsh, H. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 12, 335–361.

    Article  Google Scholar 

  • Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of the behavior of alternative models. Applied Psychological Measurement, 15, 47–70.

    Article  Google Scholar 

  • Marsh, H. W., Byrne, B. M., & Craven, R. (1992). Overcoming problems in confirmatory factor analysis of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research, 27, 489–507.

    Article  Google Scholar 

  • Marsh, H. W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. Hoyle (Ed.), Structural equation modeling (pp. 177–198). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Marsh, H. W., Nagengast, B., Morin, A. J. S., Parada, R. H., Craven, R. G., & Hamilton, L. R. (2011). Construct validity of the multidimensional structure of bullying and victimization: An application of exploratory structural equation modeling. Journal of Educational Psychology, 103(3), 701–732.

    Article  Google Scholar 

  • Marsh, H. W., Wen, Z., Nagengast, B., & Hau, K. (2012). Handbook of structural equation modeling (pp. 436–455). New York: The Guilford Press.

    Google Scholar 

  • Matarazzo, J. D., Guze, S. B., & Matarazzo, R. G. (1955). An approach to the validity of the Taylor Anxiety Scale: Scores of medical and psychiatric patients. The Journal of Abnormal and Social Psychiatry, 51(2), 276–280.

    Article  Google Scholar 

  • McCoach, D. B. (2003). SEM isn’t just the school wide enrichment model anymore: structural equation modeling (SEM) in gifted education. Journal for the Education of the Gifted, 27, 36–61.

    Google Scholar 

  • McCoach, D. B., & Siegle, D. (2003a). The school attitude assessment survey-revised: A new instrument to identify academically able students who underachieve. Educational and Psychological Measurement, 63(3), 414–429.

    Article  Google Scholar 

  • McCoach, D. B., & Siegle, D. (2003b). Factors that differentiate underachieving gifted students from high-achieving gifted students. Gifted Child Quarterly, 47(2), 144–154.

    Article  Google Scholar 

  • Muthen, L. K., & Muthen, B. O. (1998–2007). Mplus Users Guide (4th Ed.). Los Angeles: Muthen & Muthen.

    Google Scholar 

  • Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

    Google Scholar 

  • Nussbeck, F. W., Eid, M., Geiser, C., Courvoisier, D. S., & Lischetzke, T. (2009). A CTC(M-1) model for different types of raters. Methodology, 5, 88–98.

    Google Scholar 

  • Oort, F. J. (2009). Three-mode models for multitrait-multimethod data. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 78–87.

    Article  Google Scholar 

  • Parada, R. (2000). Adolescent Peer Relations Instrument: A theoretical and empirical basis for the measurement of participant roles in bullying and victimization of adolescence: An interim test manual and a research monograph: A test manual. Publication Unit, Self-concept Enhancement and Learning Facilitation (SELF) Research Centre, University of Western Sydney.

    Google Scholar 

  • Peterson, J. S., & Colangelo, N. (1996). Gifted achievers and underachievers: A comparison of patterns found in school files. Journal of Counseling and Development, 74, 399–406.

    Article  Google Scholar 

  • Rabe-Hesketh, S., & Skrondal, A. (2012). Multilevel and longitudinal modeling using stata (3rd edn ). College Station, TX: Stata Press.

    Google Scholar 

  • Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. New York: Routledge.

    Google Scholar 

  • Raykov, T., & Marcoulides, G. A. (2006). A first course in structural equation modeling (2nd ed.). Mahway: Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Silver, H. A., & Barnette, W. L. (1970). Predictive and concurrent validity of the Minnesota vocational interest inventory. Journal of Applied Psychology, 54(5), 436–440.

    Article  Google Scholar 

  • Sireci, S. G. (2006). Content validity. In N. J. Salkind (Ed.) Encyclopedia of measurement and statistics. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Sireci, S. G., & Parker, P. (2006). Validity on trial: Psychometric and legal conceptualizations of validity. Educational Measurement: Issues and Practice, 25(3), 27–34.

    Article  Google Scholar 

  • Schmidt, F. L. (1988). The problem of group differences in ability scores in employment selection. Journal of Vocational Behavior, 33, 272–292.

    Article  Google Scholar 

  • Schumacker, R. E., & Lomax, R. G. (1996). A beginner’s guide to structural equation modeling. Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Spitzer, R. L., Endicott, J., & Robins, E. (1978). Research diagnostic criteria: Rationale and reliability. Archives of General Psychiatry, 35(6), 773–782.

    Article  PubMed  Google Scholar 

  • Suldo, S. M., Shaffer, E. J., & Shaunessy, E. (2008). An independent investigation of the validity of the School Attitude Assessment Survey–Revised. Journal of Psychoeducational Assessment, 26(1), 69–82.

    Article  Google Scholar 

  • Süss, H.-M., Oberauer, K., Wittmann, W. W., Wilhelm, O., & Schulze, R. (2002). Working-memory capacity explains reasoning ability—And a little bit more. Intelligence, 30, 261–288.

    Article  Google Scholar 

  • Super, D. E. (1970). Work values inventory manual. Boston, MA: Houghton Mifflin Company.

    Google Scholar 

  • Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. New York: Harper Collins.

    Google Scholar 

  • Taylor, J. (1953). A personality scale of manifest anxiety. The Journal of Abnormal and Social Psychology, 48(2), 285–290.

    Article  Google Scholar 

  • Warner, W. L., Meeker, M., & Eells, K. (1949). Social class in America; a manual of procedure for the measurement of social status. Oxford, England: Science Research Associates.

    Google Scholar 

  • Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1–26.

    Article  Google Scholar 

  • Widaman, K. F. (1992). Multitrait-multimethod models in aging research. Experimental Aging Research, 18, 185–201.

    Article  PubMed  Google Scholar 

  • Wilhelm, O., & Schulze, R. (2002). The relation of speeded and unspeeded reasoning with mental speed. Intelligence, 30, 537–554.

    Article  Google Scholar 

  • Wolfle, J. A. (1991). Underachieving gifted males: Are we missing the boat? Roeper Review, 13, 181–184.

    Article  Google Scholar 

  • Wood, A. M., Joseph, S., & Maltby, J. (2008). Gratitude uniquely predicts satisfaction with life: Incremental validity above the domains and facets of the five factor model. Personality and Individual Differences, 45(1), 49–54.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Educational Psychology Department, University of Connecticut, Glenbrook Road, Unit 3064 249, Storrs, CT, 06269-3064, USA

    D. Betsy McCoach

  2. Alan Shawn Feinstein Graduate School, Johnson and Wales University, Quail Run Rd. 26, Storrs, CT, 06268, USA

    Robert K. Gable

  3. Department of Educational Psychology, University of Connecticut, Glenbrook Road, Unit 2064 249, Storrs, CT, 06269-2064, USA

    John P. Madura

Authors
  1. D. Betsy McCoach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert K. Gable
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John P. Madura
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to D. Betsy McCoach .

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

McCoach, D.B., Gable, R.K., Madura, J.P. (2013). Evidence Based on Relations to Other Variables: Bolstering the Empirical Validity Arguments for Constructs. In: Instrument Development in the Affective Domain. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-7135-6_6

Download citation

Publish with us

Policies and ethics

Search

Navigation