On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model

Abstract

Investigating sources of within- and between-group differences and measurement invariance (MI) across groups is fundamental to any meaningful group comparison based on observed test scores. It is shown that by placing certain restrictions on the multigroup confirmatory factor model, it is possible to investigate the hypothesis that within- and between-group differences are due to the same factors. Moreover, the modeling approach clarifies that absence of measurement bias implies common sources of within- and between-group variation. It is shown how the influence of background variables can be incorporated in the model. The advantages of the modeling approach as compared with other commonly used methods for group comparisons is discussed and illustrated by means of an analysis of empirical data.

Introduction

Investigating within- and/or between-group differences on test scores is the focus of a large number of research studies. The variance of an item or subscale score within a group indicates the individual differences within the group. Individual differences with respect to multiple observed variables may be summarized in a within-group variance–covariance (or correlation) matrix. The structure of this matrix can be investigated using confirmatory (or exploratory) factor analysis. Confirmatory factor analysis (CFA) is concerned with “explaining” the common content of observed variables captured by their covariances with a smaller number of underlying latent variables called factors. As CFA is applied to the covariance matrix within a single group, the common factors can be regarded as the sources of systematic within-group differences. Differences between groups, on the other hand, are often tested by comparing the groups with respect to the means of the observed scores or with respect to the means of the factors underlying the observed scores. The latter may be viewed as an analysis of the sources of between-group differences and can be done by carrying out multigroup CFA.

To render the group comparisons meaningful, it is necessary to address the issue of measurement invariance (MI) and demonstrate that a given test measures the same underlying factors across groups. We use the expression “same factor” to indicate that a factor has exactly the same conceptual interpretation across groups. The interpretation of a factor depends on the content of the observed items or subscales that are related to the factor and the strength of those relations. Consequently, for a factor to have an identical interpretation across groups, it is necessary that the relations of the observed variables and the underlying factor are exactly the same across groups.

The present paper focuses on the relation between these three aspects of group comparisons, namely the relation between the sources of within-group differences (i.e., which factors explain individual differences within a group?), the sources of between-group differences (i.e., which factors explain the differences between groups?), and MI (i.e., does the test measure the same factors in all groups?). Although all three aspects have been extensively investigated separately, the relation between the three has not been clearly examined. In addition, we show that hypotheses concerning these three issues can be tested using multigroup confirmatory factor models.

Two areas, in which the relation between within- and between-group differences and MI is important, are ethnic group differences in IQ test scores and the seemingly linear increase over time in mean IQ test scores, termed the “Flynn effect” Flynn, 1987, Flynn, 1999. In these areas of research, it has been frequently noted that sources of within-group differences and sources of between-group differences are not necessarily identical (Lewontin, 1970). Differences between ethnic groups on an IQ test may be due to other factors than those which contribute to the individual differences within each of the groups. Although several studies focus explicitly on the issue of within- and between-group differences, both in a more general context Turkheimer, 1990, Turkheimer, 1991 and specifically with respect to the Flynn effect Flynn, 2000, Rodgers, 1998, it is common practice to investigate the two sources of variance separately. Examples are single group factor analysis and multiple regression, which are based on within-group differences, and (M)ANOVA, in which groups are compared with respect to their observed means. Another common strategy is to compare the test score means adjusted for the influence of some variable of interest (Phillips, Brooks-Gunn, Duncan, Klebanov, & Crane, 1998). As will be shown, this approach is based on the implicit assumption that sources of within-group variance are the same as sources of between-group variance. This assumption is usually not tested in practice. In order to show that within- and between-group differences are indeed due to the same factors, it is necessary to analyze the means and the covariances of the observed scores simultaneously.

As mentioned above, if comparisons of item or subscale scores are to be valid, the test has to measure the same underlying factors in all groups. The concept of MI provides a theoretical framework, which includes the necessary conditions to establish whether a given test measures the same factors in the groups under consideration. The definition of MI states that, conditional on the factor scores, observed scores do not depend on group membership. This means that members of different groups who have the same score on the factor (e.g., the same level of ability) have on average the same observed scores. The definition of MI implies that groups may differ only with respect to the means and covariances of the factors that are measured by the observed scores. In practice, MI can be investigated by fitting multigroup CFA models to a given data set. To represent MI, certain model parameters are restricted to be equal across groups. Both the restricted model and a less restricted model are fitted to the data. The models may be compared by means of a likelihood ratio test. The test can provide evidence that MI is tenable (for applications, see Dolan, 2000, Dolan & Hamaker, 2001).

The central issue of the present paper concerns the relation between MI on the one hand and within- and between-group differences on the other hand. Specifically, the definition of MI across groups implies that between-group differences cannot be due to factors with a different conceptual interpretation than the factors that account for the within-group differences. Although the importance of MI has been acknowledged Byrne et al., 1989, Dolan, 2000, Lubke et al., 2001, Marsh, 1994, McArdle, 1998, Oort, 1998, this implication is not well recognized. Hence, if in practice the hypothesis of MI is not rejected, one can conclude with some confidence that within- and between-group differences are attributable to the same factors.

Given the importance of conclusions in areas such as ethnic differences and/or the Flynn effect, it is surprising that, at least to our knowledge, few of the recent studies use multigroup CFA. A possible reason for the lack of using more state-of-the-art methods may lie in the rather technical character of publications discussing implications of MI Bloxom, 1972, Ellis, 1993, Meredith, 1993. Although some technical formulation is unavoidable, it is our aim to explain the relation between MI and common sources of within- and between-group in an accessible way and to discuss the advantages of using multigroup confirmatory factor models rather than other commonly used methods. The approach proposed in this paper is applicable to a wide range of research questions. The approach is adequate if groups are to be compared on tests that consist of a larger number of continuous items or subscale scores, which are assumed to measure a smaller number of underlying factors. This includes group comparisons on multidimensional test batteries (e.g., IQ test batteries) as well group comparisons on personality, mood, and attitude questionnaires or combinations of these.

The paper is organized as follows. First, the multigroup CFA model is presented. We show that observed scores are decomposed into common factor scores and a regression residual, which comprises measurement error and item specific error. This decomposition has the advantage that groups can be compared with respect to the means and covariances of the factors. Second, we explain the concept of MI on a theoretical level and on a more practical level in the context of the multigroup common factor model. The multigroup common factor model corresponding to MI is characterized by a set of invariance restrictions across groups. Third, we show that MI implies that between-group differences are unlikely to be due to other factors than those capturing systematic within-group differences. We discuss how this result can be used in practice. By comparing a model with the invariance restrictions across groups to a less restricted model in a likelihood ratio test, one can examine not only whether MI holds but also whether between-group differences are due to differences in the same factors as the within-group differences. Fourth, we discuss how the multigroup model can be extended to include background variables. The way in which background variables are integrated can be guided by the outcome of tests of MI Oort, 1992, Oort, 1998. Finally, we briefly discuss the advantages of multigroup CFA as compared with other commonly used methods and present, for the purpose of illustration, an analysis of scores of African and Caucasian Americans on an IQ test (Osborne, 1980).