Measurement invariance: Review of practice and implications

doi:10.1016/j.hrmr.2008.03.003

Human Resource Management Review

Volume 18, Issue 4, December 2008, Pages 210-222

https://doi.org/10.1016/j.hrmr.2008.03.003 Get rights and content

Abstract

A review of efforts to assess the invariance of measurement instruments across different respondent groups using confirmatory factor analysis (CFA) is provided for the years since the Vandenberg and Lance [Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–69.] review. Investigators are more frequently reporting tests of scalar invariance and tests for differences in latent factor means and partial invariance. Efforts have been made to assess, the impact of the choice of a referent indicator in multi-group studies, the appropriateness of forming partials as indicators of a latent construct, the degree of convergence of item response theory and CFA analyses of measurement differences across groups, and the implications of findings of invariance. In this context, a demonstration of the impact of partial invariance on estimated group differences in reliability and means is provided and discussed.

Section snippets

Invariance defined

A measure is invariant when members of different populations who have the same standing on the construct being measured receive the same observed score on the test. A test violates invariance when two individuals from different populations who are identical on the construct score differently on it. In the CFA model, a series of tests are used to establish that there is invariance across populations. The sequence is outlined in Vandenberg and Lance (2000) and a variety of other sources. Byrne

Assessments of factor invariance

To identify articles that used CFA in assessing factor invariance, we did a search of papers published since 2000 that used the term “measurement invariance.” This produced approximately 88 papers. Of these, 75 conducted empirical analyses of measurement invariance using CFA methods. The remainder was discussions or critiques of the CFA method (e.g., Borsboom, 2006). Each of the empirical articles were read to determine the types of invariance considered, the content area addressed by the

Continued development of methods for the assessment of measurement invariance

Because it is relatively rare that a researcher finds a measure that is invariant across all sets of parameters, it is not surprising that many more are presenting models that include partial invariance across participant groups in one or more sets of parameters. Researchers have begun to evaluate this practice and the extent to which allowing some parameters to vary across groups affects subsequent tests of parameters. Millsap and Kwok (2004) examined the impact of partial invariance on the

Lack of measurement equivalence and corresponding differences in reliability and mean differences

The data set for this illustration is based on the responses of 680 African-American and 1522 Caucasian college students to fifteen items from the short form of the IPIP (Goldberg, 1999). The items, contained in Table 2 are from measures of Conscientiousness, Agreeableness and Emotional Stability. Only five items were taken from each scale to keep our illustration simple. Variance–covariance matrices for the two groups were used as input to LISREL 8.72 and are the subject of the analyses

Conclusions

Our review of studies conducted since 2000 that have assessed measurement invariance suggests that examinations of scalar invariance and factor mean differences are much more frequent than they were in the literature reviewed by Vandenberg and Lance (2000). All investigators estimate configural and metric invariance, though assessments of configural invariance often seem relatively cursory. Few investigators test the significance of the difference of the variance–covariance matrices. It is also

References (105)

AndersonC.B. et al.
Cross-cultural equivalence of feeding beliefs and practices: The psychometric properties of the child feeding questionnaire among Blacks and Hispanics
Preventive Medicine
(2005)
AntonakisJ. et al.
Context and leadership: An examination of the nine-factor full-range leadership theory using the Multifactor Leadership Questionnaire
The Leadership Quarterly
(2003)
AtienzaF.L. et al.
Satisfaction with life scale: Analysis of factorial invariance across sexes
Personality and Individual Differences
(2003)
BonaccioS. et al.
Differentiation of cognitive abilities as a function of neuroticism level: A measurement equivalence/invariance analysis
Intelligence
(2006)
BowdenS.C. et al.
Measurement invariance of core cognitive abilities in heterogeneous neurological and community samples
Intelligence
(2004)
DolanC.V. et al.
Multi-group covariance and mean structure modeling of the relationship between the WAIS-III common factors and sex and educational attainment in Spain
Intelligence
(2006)
GomezR. et al.
The spiritual well-being questionnaire: Testing for model applicability, measurement and structural equivalencies, and latent mean differences across gender
Personality and Individual Differences
(2005)
LeoneL. et al.
The cross-cultural generalizability and validity of the multicultural personality questionnaire
Personality and Individual Differences
(2005)
LubkeG.H. et al.
On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model
Intelligence
(2003)
Maydeu-OlivaresA. et al.
Psychometric properties of the Spanish adaptation of the Social Problem-solving Inventory—Revised (SPSI-R)
Personality and Individual Differences
(2000)

MotlR.W. et al.

Factorial validity and invariance of questionnaires measuring social–cognitive determinants of physical activity among adolescent girls

Preventive Medicine

(2000)

ReeveC.L. et al.

The psychometric paradox of practice effects due to retesting: Measurement invariance and stable ability estimates in the face of observed score changes

Intelligence

(2005)

ScholdererJ. et al.

Cross-cultural validity of the food-related lifestyles instrument (FRL) within Western Europe

Appetite

(2004)

SinL.Y.M. et al.

Relationship marketing orientation: Scale development and cross-cultural validation

Journal of Business Research

(2005)

UeltschyL.C. et al.

Cross-cultural invariance of measures of satisfaction and service quality

Journal of Business Research

(2004)

WichertsJ.M. et al.

Are intelligence tests measurement invariant over time? Investigating the nature of the Flynn effect

Intelligence

(2004)

AndersonN. et al.

A construct-driven investigation of gender differences in a leadership-role assessment center

Journal of Applied Psychology

(2006)

BandalosD.J. et al.

Item parceling issues in structural equation modeling

BentlerP.M.

Comparative fit indexes in structural models

Psychological Bulletin

(1990)

BorsboomD.

When does measurement invariance matter?

Medical Care

(2006)

BowdenS.C. et al.

Age-related invariance of abilities measured with the Wechsler Adult Intelligence Scale-III

Psychological Assessment

(2006)

BurnsG.L. et al.

Measurement and structural invariance of parent ratings of ADHD and ODD symptoms across gender for American and Malaysian children

Psychological Assessment

(2006)

ByrneB.M. et al.

The MACS approach to testing for multigroup invariance of a second-order factor structure: A walk through the process

Structural Equation Modeling

(2006)

ByrneB.M. et al.

The issue of measurement invariance revisited

Journal of Cross-Cultural Psychology

(2003)

CervellonM. et al.

Assessing the cross-cultural applicability of affective and cognitive components of attitude

Journal of Cross-Cultural Psychology

(2002)

ChanD.

Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models

Journal of Applied Psychology

(1998)

ChengC.H.K. et al.

Age and gender invariance of self-concept factor structure: An investigation of a newly developed Chinese self-concept instrument

International Journal of Psychology

(2000)

ChenF.F. et al.

Testing measurement invariance of second-order factor models

Structural Equation Modeling

(2005)

ChenY. et al.

Attitude toward and propensity to engage in unethical behavior: Measurement invariance across major among university students

Journal of Business Ethics

(2006)

CheungG.W. et al.

Assessing extreme and acquiesence response sets in cross-cultural research using SEM

Journal of Cross-Cultural Psychology

(2000)

CheungG.W. et al.

Evaluating goodness-of-fit indexes for testing measurement invariance

Structural Equation Modeling

(2002)

ColeM.S. et al.

The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership: A multinational test

Organizational Research Methods

(2006)

CrockettL.J. et al.

Measurement invariance of the center for epidemiological studies depression scale for Latino and Anglo adolescents: A national study

Journal of Consulting and Clinical Psychology

(2005)

De FriasC.M. et al.

Confirmatory factor structure and measurement invariance of the Memory Compensation Questionnaire

Psychological Assessment

(2005)

Del BarrioV. et al.

Factor structure invariance in the children's Big Five questionnaire

European Journal of Psychological Assessment

(2006)

DierdorffE.C. et al.

Group differences and measurement equivalence: Implications for command climate survey research and practice

Military Psychology

(2006)

DollW.J. et al.

The meaning and measurement of user satisfaction: A multigroup invariance analysis of the end-user computing satisfaction instrument

Journal of Management Information Systems

(2004)

DuL. et al.

Measurement invariance across gender and major: The love of money among university students in People's Republic of China

Journal of Business Ethics

(2005)

DurvasulaS. et al.

Does vanity describe other cultures? A cross-cultural examination of the vanity scale

Journal of Consumer Affairs

(2001)

EidM. et al.

Comparing typological structures across cultures by multigroup latent class analysis

Journal of Cross-Cultural Psychology

(2003)

FeldtT. et al.

Structural invariance and stability of sense of coherence: A longitudinal analysis of two groups with different employment experiences

Work and Stress

(2005)

FrenchB.F. et al.

Confirmatory factor-analytic procedures for the determination of measurement invariance

Structural Equation Modeling

(2006)

FrenzelA.C. et al.

Achievement emotions in Germany and China: A cross-cultural validation of the Academic Emotions Questionnaire-Mathematics

Journal of Cross-Cultural Psychology

(2007)

GaudreauP. et al.

Positive and negative affective states in a performance-related setting

European Journal of Psychological Assessment

(2006)

GoldbergL.R.

A broad-bandwidth public-domain personality inventory measuring the lower-level facets of several five-factor models

GregorichS.E.

Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory analysis framework

Medical Care

(2006)

GrouzetF.M.E. et al.

Longitudinal cross-gender factorial invariance of the academic motivation scale

Structural Equation Modeling

(2006)

GuppyA. et al.

The psychometric properties of the short version of the Cybernetic Coping Scale: A multigroup CFA across four samples

Journal of Occupational and Organizational Psychology

(2004)

HollandP.W. et al.

Differential item functioning

(1993)

HornJ.L. et al.

A practical and theoretical guide in measurement invariance in aging research

Experimental Aging Research

(1992)

Cited by (481)

Values and pro-environmental behavior: What is the role of trust?
2024, Journal of Outdoor Recreation and Tourism
Freshwater ecosystems provide invaluable natural resources for outdoor recreationists but are at risk of degradation from biological invasions. An understanding of the broad values and behavior of people responsible for the inadvertent spread of aquatic invasive species can inform efforts to mitigate environmental impacts and enhance people’s experiences in the outdoors. However, there are polarized levels of trust held in regulatory communities that can influence how people draw on their core belief systems when making decisions. Therefore, we examined how values worked in conjunction with trust in the regulatory community to predict pro-environmental behavior using data from a state-wide survey of recreational boaters in Illinois, USA. Results from a latent variable structural equation model suggested that intentions were positively and negatively predicted by biospheric and altruistic values, respectively. We also observed that boaters instilled high levels of trust in the Illinois Department of Natural Resources and scientific communities; however, more pronounced relationships between values and behavior emerged for respondents who were more trusting. We contend that values serve as a dependable basis for the development of long-term management strategies given that they are unlikely to change over time, but the translation of these values to behavioral performance depends on trust as an intervening factor.
This study sought to understand how polarized levels of trust in regulatory institutions including resource managers and scientists affected outdoor recreationists' intentions to carry out environmentally beneficial behaviors. We found that water-based recreationists' intentions to mitigate the spread of aquatic invasive species were partially explained by their broad values that reflected their deeply seated perspectives on human-nature relationships. We also found that these values more effectively predicted behavior when trust was higher. These findings suggest that management agencies that are more trusted may have a greater ability to anticipate whether their constituents will perform prescribed behaviors to benefit the environment based on their deeply held values.
Development and validation of the “Instructional Leadership Behaviors Scale”: The Turkish urban sample
2024, Revue Europeenne de Psychologie Appliquee
The worldwide interest in the instructional leadership behaviors of school administrators has led to the prevalent use of the concept in domestic research. The increasing number of domestic research on instructional leadership research is a challenge but also an opportunity for the fields of educational leadership and management.
This study aimed to develop and validate the “Instructional Leadership Behaviors Scale” (ILBS) to describe the instructional leadership behaviors of school administrators in Turkey.
The data were collected online from 1100 teachers working in public schools in Turkey for exploratory and confirmatory factor analyses and reliability analyses.
The analyses showed that ILBS comprised 28 items and two factors: “instructional management” and “professional development.” The ILBS was a valid and reliable instrument developed according to teachers’ opinions.
It is expected that ILBS would be a reliable instrument sensitive to the domestic and legal aspects of “instructional management” and “professional development” in the instructional leadership behaviors of school administrators in Turkey.
L’intérêt mondial pour les comportements de leadership pédagogique des administrateurs scolaires a conduit à l’utilisation répandue du concept dans la recherche. Le nombre croissant de recherches en leadership pédagogique est un défi, mais aussi une opportunité pour les domaines du leadership et de la gestion de l’éducation.
Cette étude visait à développer et à valider l’« échelle des comportements de leadership pédagogique » (ILBS) pour décrire les comportements de leadership pédagogique des administrateurs scolaires en Turquie.
Les données ont été collectées en ligne auprès de 1100 enseignants travaillant dans des écoles publiques en Turquie en vue d’effectuer des analyses factorielles exploratoires et confirmatoires et des analyses de fiabilité.
Les analyses ont montré que l’ILBS comprenait 28 items répartis en deux facteurs : « gestion pédagogique » et « développement professionnel ». L’ILBS est un instrument valide et fiable développé selon les opinions des enseignants.
On s’attend à ce que l’ILBS soit un instrument fiable sensible aux aspects nationaux et juridiques de la « gestion pédagogique » et du « développement professionnel » dans les comportements de leadership pédagogique des administrateurs scolaires en Turquie.
Validation of the Spanish version of the body image acceptance and action questionnaire (BI-AAQ-Spanish): Measurement invariance across cultures
2024, Journal of Contextual Behavioral Science
The purpose of this study was to assess the psychometric properties of a Spanish version of the Body Image Acceptance and Action Questionnaire (BI-AAQ) and its recently developed 5-item version, the BI-AAQ-5. The BI-AAQ measures psychological flexibility/inflexibility regarding body image. A sample of Spanish adults (n = 938) completed the BI-AAQ and a battery of measures (including body mass index, psychological flexibility, exposure to and internalization of sociocultural body image expectations, body dissatisfaction, and eating disorder symptoms). Measurement invariance was tested against a U.S. American adult sample (n = 866) that completed the English version of the BI-AAQ. The unidimensional factor structure of the BI-AAQ and BI-AAQ-5 was replicated in both samples using confirmatory factor analysis, with model fit indexes ranging from adequate (e.g., CFI = 0.95) to excellent (e.g., CFI = 0.99). Internal consistency was good for both instruments across samples (α = 0.90 to 0.97). Measurement invariance analyses confirmed full configural and metric invariance and scalar partial invariance. The Spanish BI-AAQ and BI-AAQ-5 showed clear evidence of convergent and incremental construct validity. Both instruments’ scores correlated substantively with theoretically related variables. In addition, the results of a conditional process analysis showed that body-image psychological flexibility measured with either instrument moderated the mediated effect of pressure to conform to cultural ideals of body image on disordered eating through internalization of body image ideals and body dissatisfaction. We concluded both instruments are likely suitable for conducting cross-cultural research with Spanish and English-speaking samples.
Parent–adolescent discrepancies in positive parenting and adolescent problem behaviors in Chinese families
2024, Heliyon
Based on the discrepancy–maladaptive hypothesis and general strain theory, in this study, we examined two key aspects: first, the mediating role of self-control in the relationship between parent–adolescent discrepancies in positive parenting and adolescent internalizing and externalizing problems, and second, variations in problem behavior among subgroups with different parent–adolescent dyads reporting patterns. The participants were 349 intact Chinese families, with parents as the primary caregivers and teenagers aged 15–18 years who are attending secondary vocational schools. The results revealed that adolescents generally perceived lower levels of caring and behavioral control than parents. Compared to behavioral control, discrepancies in perceived levels of caring had more significant predicting levels of internalizing and externalizing problems, and the relationship between discrepancies of caring and internalizing and externalizing problems was mediated by self-control. Latent profile analysis revealed three parent–adolescent responding patterns (subgroups); compared to the other subgroups, only the subgroup characterized by adolescents perceiving lower caring and behavioral control than parents exhibited higher levels of internalizing and externalizing problems. The findings of this study provide insights on how parent–adolescent discrepancies may lead to adolescent problem behaviors and highlight the importance of self-control as a mediating mechanism.
Developing a short language classroom engagement scale (LCES) and linking it with needs satisfaction and achievement
2024, System
Although engagement is gaining attention in applied linguistics, the lack of a measurement tool with robust psychometric properties hinders progress. To address this gap, the study introduces a short language classroom engagement scale (LCES) based on Hiver et al.’s (2020) language engagement scale. Participants include Chinese high school EFL learners (n = 1578) and university learners majoring in Japanese (n = 535). In Study 1, a 9-item LCES was developed, comprising three factors: behavioral, emotional, and cognitive engagement, each including three items. The retained items are examined to ensure the scale's content and substantive validity. The LCES was subsequently validated in a separate sample of Chinese high school EFL learners, providing strong evidence for its structural validity and reliability. Study 2 further confirmed the LCES's validity and reliability among Chinese university learners of Japanese while also establishing generalizability by ensuring its invariance across gender and educational levels/language studied using the entire sample. Additionally, both Study 1 and Study 2 substantiated the external validity of the LCES by establishing associations between engagement, needs satisfaction, and achievement. The pedagogical implications of these findings are discussed.
Family functioning in students of health sciences in four Latin American countries: a study of the structure and factorial invariance of the FACES III scale. A cross-sectional study
2024, Psicologia: Reflexao e Critica

View all citing articles on Scopus

^☆: The authors would like to acknowledge the helpful comments of Robert Vandenberg and Eugene Stone-Romero on an earlier version of this paper.

View full text

Measurement invariance: Review of practice and implications☆

Abstract

Section snippets

Invariance defined

Assessments of factor invariance

Continued development of methods for the assessment of measurement invariance

Lack of measurement equivalence and corresponding differences in reliability and mean differences

Conclusions

Preventive Medicine

The Leadership Quarterly

Personality and Individual Differences

Intelligence

Intelligence

Intelligence

Personality and Individual Differences

Personality and Individual Differences

Intelligence

Personality and Individual Differences

Preventive Medicine

Intelligence

Appetite

Journal of Business Research

Journal of Business Research

Intelligence

A construct-driven investigation of gender differences in a leadership-role assessment center

Journal of Applied Psychology

Item parceling issues in structural equation modeling

Comparative fit indexes in structural models

Psychological Bulletin

When does measurement invariance matter?

Medical Care

Age-related invariance of abilities measured with the Wechsler Adult Intelligence Scale-III

Psychological Assessment

Measurement and structural invariance of parent ratings of ADHD and ODD symptoms across gender for American and Malaysian children

Psychological Assessment

The MACS approach to testing for multigroup invariance of a second-order factor structure: A walk through the process

Structural Equation Modeling

The issue of measurement invariance revisited

Journal of Cross-Cultural Psychology

Assessing the cross-cultural applicability of affective and cognitive components of attitude

Journal of Cross-Cultural Psychology

Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models

Journal of Applied Psychology

Age and gender invariance of self-concept factor structure: An investigation of a newly developed Chinese self-concept instrument

International Journal of Psychology

Testing measurement invariance of second-order factor models

Structural Equation Modeling

Attitude toward and propensity to engage in unethical behavior: Measurement invariance across major among university students

Journal of Business Ethics

Assessing extreme and acquiesence response sets in cross-cultural research using SEM

Journal of Cross-Cultural Psychology

Evaluating goodness-of-fit indexes for testing measurement invariance

Structural Equation Modeling

The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership: A multinational test

Organizational Research Methods

Measurement invariance of the center for epidemiological studies depression scale for Latino and Anglo adolescents: A national study

Journal of Consulting and Clinical Psychology

Confirmatory factor structure and measurement invariance of the Memory Compensation Questionnaire

Psychological Assessment

Factor structure invariance in the children's Big Five questionnaire

European Journal of Psychological Assessment

Group differences and measurement equivalence: Implications for command climate survey research and practice

Military Psychology

The meaning and measurement of user satisfaction: A multigroup invariance analysis of the end-user computing satisfaction instrument

Journal of Management Information Systems

Measurement invariance across gender and major: The love of money among university students in People's Republic of China

Journal of Business Ethics

Does vanity describe other cultures? A cross-cultural examination of the vanity scale

Journal of Consumer Affairs

Comparing typological structures across cultures by multigroup latent class analysis

Journal of Cross-Cultural Psychology

Structural invariance and stability of sense of coherence: A longitudinal analysis of two groups with different employment experiences

Work and Stress

Confirmatory factor-analytic procedures for the determination of measurement invariance

Structural Equation Modeling

Achievement emotions in Germany and China: A cross-cultural validation of the Academic Emotions Questionnaire-Mathematics

Journal of Cross-Cultural Psychology

Positive and negative affective states in a performance-related setting

European Journal of Psychological Assessment

A broad-bandwidth public-domain personality inventory measuring the lower-level facets of several five-factor models