A psychometric evaluation of the Mayer–Salovey–Caruso Emotional Intelligence Test Version 2.0

Abstract

There has been some debate recently over the scoring, reliability and factor structure of ability measures of emotional intelligence (EI). This study examined these three psychometric properties with the most recent ability test of EI, the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT V2.0; Mayer, Salovey, & Caruso, [Mayer, J. D., Salovey, & P., Caruso, (2000). Models of emotional intelligence. In R. J., Sternberg (Ed.). Handbook of intelligence (pp. 396–420). New York: Cambridge; Mayer, J. D., Salovey, P., & Caruso, D. R., (2000). The Mayer, Salovey, and Caruso emotional intelligence test: Technical manual. Toronto, ON: MHS]), with a sample (n=431) drawn from the general population. The reliability of the MSCEIT at the total scale, area and branch levels was found to be good, although the reliability of most of the subscales was relatively low. Consistent with previous findings, there was a high level of convergence between the alternative scoring methods (consensus and expert). However, unlike Mayer et al.'s [Mayer, J. D., Salovey, P., Caruso, D. R., & Sitarenios, G. (2003). Measuring emotional intelligence with the MSCEIT V2. 0. Emotion, 3, 97–105.] contentions, there was only partial support for their four-factor model of EI. A model with a general first-order factor of EI and a three first-order branch level factors was determined to be the best fitting model. There was no support for the Experiential Area level factor, nor was there support for the Facilitating Branch level factor. These results were replicated closely using the Mayer et al. [Mayer, J. D., Salovey, P., Caruso, D. R., & Sitarenios, G., (2003). Measuring emotional intelligence with the MSCEIT V2. 0. Emotion, 3, 97–105.] data. The results are discussed in light of the close comparability of the two scoring methods. Furthermore, the fundamental limitations of the MSCEIT V2.0, with respect to the inadequate number of subscales theorized to measure each branch level factor are identified and discussed.

Introduction

Very rarely do psychological constructs receive as widespread attention as the recently conceptualised construct of emotional intelligence (EI). EI has appeared on the cover of Time magazine (Gibbs, 1995), is the topic of the most widely read social science book in the world (Goleman, 1995) and many other popular books, magazine and newspaper articles (Mayer, Salovey, & Caruso, 2000). It has been argued to be a different construct to intellectual intelligence, and, thus, may add to knowledge relevant to individual differences (Mayer et al., 2000), and possibly offer unique predictive validity in wide variety of instances (Goleman, 1995).

Mayer and Salovey (1997) have conceptualised emotional intelligence (EI) as a set of mental abilities concerned with emotions and the processing of emotional information. With such it has been argued that the most valid assessment of EI will be gained from ability-based scales that involve (like other tests of mental ability), items for which there are more and less correct answers, that assess individuals' capacity to reason with and about emotions (Mayer, Caruso, & Salovey, 2000). Over a series of studies Mayer et al. have designed and examined the reliability and validity of a number of ability-based measures of EI (Mayer et al., 1999, Mayer et al., 1990, Mayer & Geher, 1996). This work has culminated in their most recent ability-based test of EI, the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT, Mayer et al., 2000). Independent psychometric evaluations of the MSCEIT are few in number as it has only been available for a short period. However, there are conceptual, developmental and correlational criteria inherent within the theoretical framework of the ability model from which it can be evaluated. Furthermore, there are research findings recently put forth by the authors (Mayer, Salovey, Caruso, & Sitarenios, 2003), and research findings with previous measures against which it can be compared (Ciarrochi et al., 2000, Mayer et al., 1999, Roberts et al., 2001).

Mayer and Salovey's (1997) ability model of EI comprises four conceptually related abilities arranged hierarchically from the most basic to the more psychologically complex, including: (1) the ability to perceive emotions; (2) the ability to utilise emotion to facilitate reasoning; (3) the capacity to understand the meaning of emotions and the information they convey; and (4) the ability to effectively regulate and manage emotion. Within this hierarchical organisation, the abilities are proposed to develop sequentially implying that they are a function of age and cognitive maturation (Mayer & Salovey, 1997). Consistent with this theoretical framework, measures of EI such as the MSCEIT are expected to; (a) show a positive manifold of correlations amongst the subscales designed to assess these four major areas; (b) a consistent factor structure that comprises a general factor of EI and four correlated primary factors; and (c) show age related differences that reflect the developmental perspective of the model (Mayer et al., 1999). As evidence that EI is an intelligence, in addition to the above, ability measures are expected to positively correlate with established measures of mental ability (as mental abilities typically do), such as those that index individuals verbal intelligence (Mayer et al., 1999). Finally, EI has been theoretically related to several important life criteria that ability measures are expected to predict (Mayer et al., 2000, Salovey & Mayer, 1990). These include variables such as psychological well-being, life satisfaction, empathy, the quality of interpersonal relationships, success in occupations that involve considerable reasoning with emotions (e.g., leadership, sales and psychotherapy) and scholastic and academic success. Mayer et al. (Mayer et al., 1990, Mayer et al., 1999, Mayer et al., 2003, Mayer & Geher, 1996) and others (e.g., Ciarrochi et al., 2000, Roberts et al., 2001), have assessed the validity of ability measures of EI according to these conceptual, developmental and correlational criteria.

Research with the predecessor measure to the MSCEIT, the Multi-factor Emotional Intelligence Scale (MEIS; Mayer et al., 1999), has provided preliminary evidence that EI meets some of the underlying conceptual, developmental and correlational criteria of Mayer and Salovey's (1997) ability model (Ciarrochi et al., 2000, Mayer et al., 1999, Roberts et al., 2001). This research has shown for example, that the four abilities measured form a positively interrelated set, and that they correlate with other measures of established ability e.g., verbal IQ (Mayer et al., 1999), and general intelligence (Roberts et al., 2001). The research by Mayer et al. (1999) also demonstrated age related differences with an adult criterion group scoring significantly higher on the MEIS than did an adolescent criterion group. Research with the MEIS has also demonstrated that scores on the test are meaningfully correlated with theoretically related criteria such as life satisfaction, empathy and parental warmth (Ciarrochi et al., 2000, Mayer et al., 1999). Importantly, the study by Ciarrochi et al. (2000), demonstrated that scores on the MEIS were related to criterion measures (e.g., life satisfaction) even after controlling for IQ and personality. Although these studies (Ciarrochi et al., 2000, Mayer et al., 1999, Roberts et al., 2001) provided promising evidence for the validity of the MEIS, the findings also revealed some psychometric problems with the test. Specifically, problems concerning the validity of the scoring methods (expert and consensus), the unacceptably low levels of internal consistency for some of the subscales, as well as the facture structure of the MEIS.

The MSCEIT (Mayer et al., 2000) has been designed to improve upon the MEIS in these three areas (scoring, reliability and factor structure). Mayer et al. (2000) have developed a more well-founded expert scoring criterion for the MSCEIT, have attempted to build upon the reliability of the test at the subscale level through item selection methods and to develop a test with a facture structure more consistent with the underlying theory of EI through a collection of new subscales.

While the expert scoring criterion for the MEIS was based on responses to the test set by Mayer and Caruso, the expert scoring criterion for the MSCEIT is based on responses to the test items from 21 members of the International Society of Research in Emotion (ISRE). Recent analyses by Mayer et al. (2003) of the MSCEIT standardization data (n=2112) demonstrated a higher level of convergence between expert and consensus scoring methods (r=.908) than that found with the MEIS. Mayer et al. also reported that there was higher inter-rater reliability in identifying correct alternatives to the test items amongst the expert group than a matched sample from the standardization group. Additionally, the standardization group as a whole obtained significantly higher scores on the Emotional Perception and Emotional Understanding subscales when scored with the expert scoring criterion than when scored with the consensus scoring criterion. Thus, Mayer et al. concluded that the expert scoring method may provide a more accurate criterion for identifying correct answers to the test items, particularly in the areas where the scientific study of emotion may have provided the expert group greater institutionalised knowledge concerning emotions. That is, emotional perception and emotional understanding, given that a great deal of emotion research has focused on coding emotional expressions (e.g., Ekman & Friesen, 1975, Scherer et al., 2001), and delineating emotional understanding (e.g., Ortony, Clore & Collins, 1988; cf. Mayer et al., 2003).

Thus, the high level of convergence between the expert and consensus scoring methods found with the MSCEIT (Mayer et al., 2003) need to be replicated. If the magnitude of correlation between the two methods is replicated, such findings may refocus, as stated by Mayer, Salovey, Caruso, and Sitarenios (2001), questions concerning the validity of the scoring protocols to “What does consensus mean?”, and “Is this form of determining a correct answer much different than that used in cognitive intelligence tests?” (p. 236).

Like the MEIS, the MSCEIT has been designed to assess the four conceptually related abilities of Mayer and Salovey's (1997) ability model of EI. Scores on the MSCEIT represent three categories; (1) an Overall EI score reflecting a general level of EI; (2) two area scores, Experiencing EI reflecting the ability to identify emotions and to assimilate emotions in thought, and Strategic EI reflecting the ability to understand and manage emotions; and (3) four branch scores (each measured by two subtests) that assess the four primary abilities of Mayer and Salovey's model. Reliability analyses of the MSCEIT with the standardization sample suggest that it has good internal consistency at the full-scale, area and branch level. Mayer et al. (2003) report split-half reliabilities ranging from r=.93 to r=.91 at the full-scale level, split-half reliabilities ranging from r=.90 to r=.86 at the area level, and split-half reliabilities ranging from r=.91 to r=.76 at the branch level (according to the consensus and expert scoring criteria). The reliability of the eight individual subscales were higher than those of the MEIS (ranging from α=.64 to α=.88), however, approximately half of the subscales have coefficient alpha's below the α=.7 criterion (Mayer et al., 2003).

Factor analyses of the MSCEIT suggest that its factor structure better represents the underlying theory of Mayer and Salovey's (1997) ability model. Mayer et al. (2003) assessed whether 1, 2 and 4 (oblique correlated) factor models of the MSCEIT provided a statistically significant fit with the standardization data via structural equation modeling. Mayer et al. report that the general factor model, two factor Experiential and Strategic models, and four primary factor models, were all found to exhibit reasonably good model fit statistics, suggesting that each model provides viable representations of the tests underlying factor structure.

However, it has been demonstrated recently that the close-fit statistics reported by Mayer et al. are inaccurate. Specifically, the NFI, TLI and RMSEA values all overestimate the degree of fit for each model. Based on the re-analyses of Gignac (in press), only the four-factor model was associated with good fit. However, to effect an acceptable CFA four-factor solution, Mayer et al. specified that “…the two within-area latent variable covariances (i.e., between Perceiving and Facilitating, and between Understanding and Managing) were additionally constrained to be equal so as to reduce a high covariance between the Perceiving and Facilitating Branch scores” (p. 103). The consequences of specifying such a constraint will be explored in this investigation, based both on the current study's data, as well as the expert scored correlation matrix reported in Mayer et al. (p. 102).¹

Published research findings with the MSCEIT suggest that its psychometric properties are considerably better than those of its predecessor the MEIS, particularly with respect to the scoring, reliability and the factor structure of the test. The level of convergence between the consensus and expert scoring methods further demonstrates that more and less correct answers to the test items may exist, and findings with the expert criterion suggest that more and less correct answers to the test may exist with respect to a more objective criterion (particularly for Emotional Perception and Understanding). Moreover, the overall quality of the MSCEIT appears to have adequate levels of reliability, and a factor structure more consistent with the underlying theory (Mayer et al., 2003; the CFA errors identified in Gignac (submitted) notwithstanding). Nonetheless, these findings need to be replicated, particularly given that the MSCEIT represents an entirely new collection of tasks and items (Mayer et al., 2003).

In their initial research study with the MSCEIT, Mayer et al. (2003) examined three of its fundamental psychometric properties; (1) the level of convergence between the consensus and expert scoring methods; (2) the reliability of the MSCEIT; and (3) its factor structure. The current study similarly examines the relationship between the consensus and expert scoring methods, and the reliability and factor structure of the MSCEIT with an Australian general population sample. The current study also expands upon Mayer et al.'s original work by examining the relationship between consensus scores determined with Mayer et al.'s. standardization data and consensus scores determined with the present sample, and their respective relationships with expert-based scores. In addition, the current study examines; (a) differences in MSCEIT scores according to gender; and (b) the relationship between scores on the MSCEIT and age. The objective of these analyses was to determine the replicability of Mayer et al.'s findings, and to examine the extent to which the consensual norms determined with Mayer et al.'s. standardization data provide a relevant scoring criterion for other Western societies, specifically the Australian population.

On the basis of Mayer et al.'s (2003) research findings it was hypothesised that; (1) there would be a strong relationship between consensus and expert-based scores on the MSCEIT; (2) that the sample would obtain somewhat higher test scores when scored with the expert scoring method on the Perceiving and Understanding branch scores reflecting the higher inter-rater reliability and superiority of the expert scoring criterion found by Mayer et al.; (3) that the MSCEIT would exhibit high internal consistency reliability at the full-scale and branch level; (4) that the factor analytic results obtained by Mayer et al. would be replicated; (5) that females would obtain significantly higher test scores than males as has been found with the MEIS (Ciarrochi et al., 2000, Mayer et al., 1999); and (6) that there would be positive relationships between scores on the MSCEIT and age, supporting previous research with the MEIS demonstrating age related differences (Mayer et al., 1999) and those reported in the MSCEIT technical manual (Mayer et al., 2000).