Investigating the Structure of the Children's Concentration and Empathy Scale Using Exploratory Graph Analysis
Abstract
Abstract. The current paper investigates the structural validity of the Children's Concentration and Empathy Scale (CCES), which was designed based on Montessori's developmental theory to assess, using teacher ratings, the coherence of attention-related characteristics (concentration, empathy, and normalization) in children from 1.5 to 12 years old. The dimensionality analysis was carried out using exploratory graph analysis (EGA), and the stability of the items and factors were checked using a bootstrap version of EGA, and the results contrasted to exploratory factor analysis. The results point to a four-factor structure (emotion regulation, fantasy, task engagement, and empathy) after eliminating items with low replicability across bootstrapped samples. Beyond pointing to ways to improve the CCES, our paper presents a number of data analytical strategies that can be useful for studies investigating the structural validity of measurement instruments and demonstrates how EGA can effectively be used in the scale construction and validation process. Our manuscript and results are fully reproducible and are available on the Open Science Framework.
Theory
What Is the Construct Being Measured?
Here, we investigate the structural validity of a new instrument, Children's Concentration and Empathy Scale (CCES), to assess the coherence of attention-related characteristics underlying cognitive performance in young children. The characteristics of the CCES are centered around the educational theory of Maria Montessori, a physician who developed an alternative education system in the early 1900s (Montessori, 1967b). Today, Montessori schooling is the world's most common and enduring alternative education system (Lillard, 2019). Montessori classrooms are characterized by 3-year age groupings, independent choice of activities, and specialized hands-on learning materials. Montessori outcomes are at least as good and often superior to those of non-Montessori schools (Lillard, 2019). Dr. Montessori noted that several cognitive, behavioral, and affective changes tend to occur when students are taught in this alternative program. Young children, for example, were often able to concentrate for extended periods of time. These heightened periods of focus lead to changes in their personalities: They became kinder, made better decisions, and seemed to grow in intelligence (Montessori, 1965, 1966).
The CCES instrument was designed to assess three main constructs of Montessori's framework: concentration, empathy, and “normalization” (a Montessori concept similar to equanimity or an unperturbed state) in children from 1.5 to 12 years old using teacher ratings. Concentration is defined as an intense focus of mental energy on an object or activity in the environment (Lillard, 1996; Montessori, 1965; Tregenza, 2008). Empathy is the capacity to discern or understand the thoughts and emotions of others (Dhiksha & Shivakumara, 2017). Specifically, where the observer reacts emotionally upon the realization that another person is experiencing or about to experience an emotion (Stotland, 1978), generating a state of resonance to the other (Ginot, 2009). Concentration was theorized to give rise to empathy and normalization with the latter being described as the state of being free of perturbation and therefore naturally empathetic, happy, and able to focus attention (Montessori, 1967a). The goal of the CCES is to identify behaviors related to these three overarching constructs, so that educators and researchers can monitor the development of these characteristics in children and relate them to academic and behavioral outcomes as well as biological changes (such as epigenetic markers). These constructs are not only interrelated in Montessori theory but also in much of psychology. Therefore, consideration of how these constructs are related to one another in early development could potentially shed light on their co-occurrence in adulthood.
The current paper investigates the structural (construct) validity of this new assessment instrument. To do so, we used a recently developed dimensionality assessment technique called exploratory graph analysis (EGA; Golino & Epskamp, 2017; Golino & Demetriou, 2017; Golino, Shi et al., 2020) and its bootstrap version called Bootstrap EGA (bootEGA; Christensen & Golino, 2019a). Together, these exploratory techniques can assess and evaluate the internal structure and the consistency of the new CCES. By using these exploratory approaches, we can verify whether the empirical structure (as assessed with these techniques) aligns with the theoretical structure (as developed from theory). Structural validation enables the empirical verification of the theory underlying the construction of the items, which can ensure the quality of future research using the CCES. Importantly, if certain constructs are not capturing what they intend to measure, then the interpretation of the scores will be unclear (Goldstein, 1980).
Our goals were thus twofold: (1) Estimate the optimal number of latent factors underlying the items of this new instrument and (2) improve on the structure of these factors by selecting items and dimensions that are consistent with the intended structure. Below, we briefly review research relevant to the purported changes noted by Dr. Montessori, which define the constructs being measured (i.e., concentration, empathy, and normalization). Then, we describe how the instrument was developed, the methods of data collection, and the recently developed approach, bootEGA, for assessing construct validity. Because validation is an iterative process, our goal is to first make sure that the scale is internally consistent before any convergent and discriminant validation was performed. Understanding what the scale measures is a prerequisite for any other validation because any issues of structural validity compound in convergent and discriminate validation analyses.
Constructs and Their Relations
Concentration, empathy, and normalization are related to several other psychological dimensions. Concentration is a construct that is positively linked to self-regulation and specific components of meta-cognition (e.g., self-monitoring; Zimmerman, 1995) and is a central component of psychological states that can lead to optimal performance such as flow (Csikszentmihalyi, 2014; Csikszentmihalyi & Csikzentmihaly, 1990). Empathy is positively associated with two personality traits, namely openness to experience (Costa et al., 2014) and agreeableness (Graziano et al., 2007), and is higher in people with greater inhibitory control (Hansen, 2011). Empathy also has a well-known relationship to prosocial behaviors (Mussen & Eisenberg-Berg, 1977). Silke et al. (2018) noted, however, that studies investigating both constructs should be more consistent in how they are conceptualized and measured. In contrast to the other two constructs, normalization has less obvious conceptual connections to traditional psychological constructs. Since the Montessori concept of normalization is similar to equanimity, it can be considered a central aspect of well-being (Desbordes et al., 2015) and related to positive characteristics such as happiness, optimism, and positive affect (Neff et al., 2007).
The psychological literature is full with examples of how concentration, empathy, and normalization are related. First, they are related in Buddhist literature, which draws on 2,500 years of human observation. In Buddhist teachings, states of deep concentration achieved in meditation lead to a characteristic called “equanimity,” meaning at equilibrium, or balanced (Goldstein, 2016, p. 273), which leads to happiness (Hanh, 2009, p. 77). In psychology, one place where the overlap of these constructs is observed is studies of the effects of mindfulness interventions. Although this research is still in its early phases (e.g., the level of methodological rigor is variable; Sedlmeier et al., 2012), there is growing consensus that mindfulness meditation practices have positive effects on people's attention, mood, empathy, and emotion regulation (e.g., Jain et al., 2007; Tang et al., 2015; Winning & Boag, 2015). One meta-analysis on mindfulness intervention studies suggests that the strongest effects of meditation interventions are on emotionality and relationship quality, with effect sizes ranging from medium to large (Sedlmeier et al., 2012). In one study, for example, a group of physicians who were given meditation experience reported a significant improvement in mood, well-being, and empathy compared to a group of physicians who did not (Krasner et al., 2009).
These links have been shown in children as well. Children's self-regulation is often measured by rating their attention, concentration, and inhibition – all of which have been related to prosocial behaviors (Eisenberg et al., 2011). In one study examining preschoolers, teachers rated their students' focused attention (e.g., “When drawing or coloring in a book, shows strong concentration”), attention shifting (e.g., “Can move on to a new task when asked”), inhibitory control (e.g., “Can lower their voice when asked to do so”), and impulsivity (e.g., “Sometimes interrupts others when they are speaking”; Cumberland-Li et al., 2004). In the study, other children and teachers also rated each child's level of agreeableness (a well-known prosocial trait; Caprara et al., 2010). Self-regulation, attention control, and agreeableness were all found to be strongly interrelated. This relation was found as early as elementary school where emotion regulation was shown to be positively related to psychological health, social competence, empathy, sympathy, and prosocial behaviors (Eisenberg et al., 1997, 2004, 2006, 2011). By contrast, low self-regulation was associated with conduct disorder, bullying, and delinquency (Riggs et al., 2006). As with adults, mindfulness interventions with children also lead to increased social competence and positive social behavior (Flook et al., 2015; Schonert-Reichl et al., 2015). These are among many studies showing a relation between emotion regulation, empathy, and prosocial behavior (Lockwood et al., 2014).
Concentration, Empathy, and Normalization in Montessori's Theory
The development of items started with Dr. Montessori's observations of these outcomes when children engaged in deep concentration in the classroom. She initially worked with atypically developing children who thrived with her sensorial and embodied methods of education, and eventually extended these techniques to typically developing low-income children. One day, she noticed a young child become deeply engrossed in a puzzle that involved placing 10 graduated cylinders inside their sockets (Montessori, 1967b). The child repeated the exercise of removing all 10 cylinders and replacing them in their proper holes over three dozen times, and when Montessori tried, she failed to break the child's concentration. Afterward, she noted that the child seemed calm and refreshed. Montessori speculated that deeply focused attention might be healthy for psychological development, and she began to incorporate conditions to encourage deep concentration into the education system she was developing. She did this by providing children with activities that most commonly inspired deep concentration in the classroom. Over the subsequent decades, she cataloged an array of personality changes that she claimed stemmed from deeply focused concentration. She considered the transformed personality to be normal, and the pretransformed one to be expressing deviations from normal. She therefore called the personality changes that stem from deep concentration “normalization” (Montessori, 1967a, p. 204), while others have described changes that come along with deep concentration as “achieving optimal experience” (Csikszentmihalyi & Csikzentmihaly, 1990).
Montessori identified a number of characteristics that emerged coincidentally when children began to concentrate more (Table 1). She observed that they become “completely transformed […] calmer, more intelligent, and more expansive” (Montessori, 1967a, p. 68). “After this phenomenon of concentration the children are really ‘new’ children. It is as though a connection has been made with an inner power […] and this brings about the construction of the personality” (Montessori, 1989, p. 21). Through concentration, children's misbehaviors cease because they are no longer “prey to all their little naughtinesses” (Montessori, 1989, p. 16). They become kinder, happier, and more interested in work (Montessori, 1994, p. 2). “When the children begin to be interested in the work and to develop themselves, […] have lively joy, […] mutual respect, and affection” become manifest (Montessori, 1967a, pp. 93–94). She noted that a child who concentrates becomes “precociously intelligent, one who has learned to overcome himself and to live in peace, and who prefers a disciplined task to futile idleness” (Montessori, 1966, p. 148).
“The child becomes capable of planning and carrying out his own actions” (Montessori, 1994, pp. 4–5). “There is a vital urge to completeness of action, and if the cycle of this urge is broken, it shows in deviations from normality and lack of purpose” (Montessori, 1948, p. 57). One reflection of such deviation is being lost in fantasy to the exclusion of reality: “The cure is to interest the mind in some work or activity which will put it back in relation to reality” (Montessori, 1961, p. 59). Montessori claimed that normalized children, those who have been allowed to deeply concentrate, are not so interested in fantasy. Instead, they show “spontaneous discipline, continuous and happy work, social sentiments of help and sympathy for others” (Montessori, 1967a, p. 207). “The child who concentrates is immensely happy; he ignores his neighbors of the visitors circulating about him. […] Love awakens in him for people and for things. He becomes friendly to everyone, ready to admire all that is beautiful” (Montessori, 1967a, p. 273).
Thus, in Montessori theory, bouts of deep concentration are believed to give rise over time to an enduring personality trait called normalization, meaning a return to normal, in the sense that life returns to normal following an illness or crisis. A normalized child is naturally emphatic and, in turn, better able to focus their attention (Montessori, 1967a). The goal of the CCES is to identify behaviors related to these three overarching concepts of concentration, empathy, and normalization.
Structural, Convergent, and Discriminant Validity
Our study does not contain data pertinent to convergent and discriminant validity because we have focused on first verifying the structural validity of the CCES. This is because the structural organization of a scale is one of the most important components of construct validity assessment (Christensen et al., 2020; Flake et al., 2017; Saccenti & Timmerman, 2017; Slocum-Gori & Zumbo, 2011). An instrument's structure determines how the scores are calculated. Therefore, incorrect or imprecise dimensionality estimation will affect every possible use of the scores – from inadequate theoretical inferences to inappropriate high-stakes decisions (Messick, 1987; Slocum-Gori & Zumbo, 2011). Similarly, accurate estimation of the number of latent factors and the allocation of items in each factor has important implications not only for the internal validity of an assessment instrument but also for the construction and evaluation of models and theory. This makes dimensionality assessment a central feature of psychological research and a paramount priority that is prerequisite to any other validation efforts.
Based on Montessori's theory, we would expect that the aforementioned characteristics would form dimensions of concentration, empathy, and normalization. In Table 1, characteristics of deep concentration (1) and realistic ventures (6) reflect concentration; showing empathetic concern (4) and appreciating beauty (5) reflect empathy; choosing challenging activities (2) and being happy (3) reflect normalization.
As a whole, the relations between these constructs follow from these characteristics. Deep concentration, for example, is required for engaging in challenging rather than easy activities. Even further, fantastical ventures may occur more during easy tasks rather than challenging ones. Indeed, task unrelated thoughts in mind wandering research typically occur during repetitive and easy tasks (Murray et al., 2020). Linking these together, the experience of flow is often described as being engaged or “lost in an activity” that is challenging, which leads to positive affect (Csikszentmihalyi & Csikzentmihaly, 1990). Moreover, positive affect and appreciating beauty are also often linked in research on esthetic experiences (Chatterjee, 2014). We expect that these interrelations will form a coherent structure of separate but related constructs. This makes the structure of the CCES best represented as a network because of the extensive cross-construct relations.
Item Construction
To assess concentration, empathy, Montessori's concept of normalization, and their associated characteristics referred to in the quotations above and summarized in points 1–6 (see Table 1), a new instrument was developed. The CCES was given to Montessori teachers (who through their training understood the concept of “normalization”). We developed 2–4 items aimed at each of the six characteristics named in Table 1, plus one item for each overarching concept of concentration, empathy, and normalization, for a total of 21 items. We administered the resulting scale to teachers at two large Montessori schools (n > 100 children) and asked them to fill the scale out for each child in their class.
Method
Data Collection
Data were collected from two suburban Montessori schools, one in the Midwestern United States and the other in the Southeastern United States. Both schools had a strong Montessori program and had children in the age range desired for this study (many Montessori schools do not go past age 6). Teachers filled out the instrument for the children in their classes. At the Midwestern school, eight teachers evaluated their students in the spring and fall; there were 74 students in the fall but three left over the year, so there were 71 students in the spring, for 145 total evaluations. At the time of fall data collection, students were 1.52–10.25 years old (M = 5.11 years). At the Southeastern school, data collection was shortened so only grade information could be provided. There, seven teachers evaluated a total of 128 students, ranging in age across prekindergarten through sixth grade (about 3–12 years old). In sum, we used data from 273 evaluations.
Measures
Children's Concentration and Empathy Scale
The CCES consisted of 18 specific items and 3 overarching items that aimed to elucidate the relations between various personality characteristics (Table 2). Concentration is believed to give rise to empathy and normalization. The specific items were aimed at these three core concepts. For example, “Shows concern for others” (CE12) is integral to empathy, whereas “Engages whole self in work” (CE1) captures concentration. These items cover a swath of personality characteristics and as such can also be connected to different categories outside of the ones they were designed under. The goal of our construct validation was to determine if they cohere as Montessori theory suggests, or in some other way. For example, some items could easily be categorized using the Big Five personality traits, such as “Is happy/content” (CE2) as positive affect of extraversion, or “Is careless/sloppy” (CE10_R) as disorganized of conscientiousness. The general items, one for each overarching concept, were included to investigate if the discrete behavior items were closely related to teacher judgments of the overarching concept, and to reveal how teachers perceived the relation between the substrate and overarching concepts.
Teachers rated each of the 21 items for each enrolled student in their class. The three overarching items were answered on a 6-point Likert scale with 0 reflecting an absence of the characteristic and 5 reflecting an abundance of the characteristic. Teachers never used the 0, basically making it a 5-point scale. For the 18 specific items, children were categorized into five columns, ranging from “Never/Not at all characteristic of child” to “Always/Very characteristic of child.” These were translated into 1 to 5 scores, and items CE3_R, CE5_R, CE10_R, CE11_R, CE13_R, and CE14_R were reversed (Table 2). When teachers placed a mark equidistant between two category levels, scores were adjusted accordingly (e.g., 2.5). The adjusted scores (e.g., 2.5) had very low frequencies, ranging from 0.36% to 0.72% of the scores.
Structural Validity
Before estimating the dimensional structure of the data, we employed a recent practice in measurement modeling to compare the fit of a simple structure theoretical confirmatory factor analysis (CFA) model to a more complex psychometric network model (Kan et al., 2020). The CFA model was fit using the diagonally weighted least squares estimator and the theoretical dimensional structure of the CCES.
Because item CE16 was the only item designed to reflect Montessori's characteristic five (appreciates beauty, demonstrating a strong esthetic sense), we could not estimate a factor of beauty appreciation. The closest characteristic of Montessori's theory to beauty appreciation was characteristic number two (choosing challenging rather than easy activities, and persisting in conquering challenges. Activity choices are constructive and purposeful, serving one's own development), so item CE16 was allocated to this factor.
Similarly, we included the normalization item in the factor representing Montessori's characteristic four (showing empathetic concern for others). The final adjusted hypothetical structure had five correlated factors (characteristics 1, 2, 3, 4, and 6 of Montessori's theory; see Table 2).
For the psychometric network model, we estimated the network following the EGA approach (Golino and Epskamp [2017]; Golino, Shi et al. [2020]; see ESM 1). The psychometric network model does not fit dimensions but rather implies the structure of a fully connected network.
We fit and compared these measurement models using the psychonetrics package (Epskamp, 2019) in R (R Core Team, 2018). We compared these models using fit indices of comparative fit index (CFI), Tucker–Lewis Index (TLI), and RMSEA. The χ2 difference was used as a test of significance for whether the more complex psychometric network model was preferred. Our results showed that the psychometric network model (CFI = .98, TLI = .97, RMSEA = .05, 95% CI = 0.04–0.06) fits better (
, p < .001) than the theoretical confirmatory factor model (
, p < .001, CFI = .97, TLI = .97, RMSEA = .08, 95% CI = 0.07–0.09).
These results confirm the notion that the constructs measured by the CCES are interrelated, making a simple structure confirmatory model inadequate for representing its structure. Thus, although we have an a priori theoretical structure, these results support the need to perform an exploratory analysis of the relations between the constructs to better understand an appropriate confirmatory model.
Exploratory Graph Analysis
To evaluate how the constructs are interrelated and the dimensions that arise from the empirical data, we employed EGA. EGA is a recently developed method to estimate the number of dimensions in multivariate data using undirected network models (Golino & Epskamp, 2017; Golino, Shi et al., 2020). EGA first applies a network estimation method followed by a community detection algorithm for weighted networks (Fortunato, 2010). EGA has been shown to be as accurate or more accurate than more traditional factor analytic methods such as parallel analysis (Christensen, Garrido, & Golino, 2020; Golino, Shi et al., 2020), that is, presenting higher correct estimations of the number of simulated factors. For the technical details, see SI 1. When data are generated by factor models, EGA is shown to accurately estimate the number of latent factors in the data (Golino, Shi et al., 2020). Because we hypothesize that the data are more likely to be generated from a network model, these dimensions represent the coherence of associated reciprocal processes across people. Said differently, these dimensions arise because some processes tend to co-vary more than others (Mõttus & Allerhand, 2018), and when aggregated across people, they reflect meaningful between person differences (Cramer et al., 2012). EGA was applied using the EGAnet package (version 0.9.8; Golino & Christensen, 2019) in R.
Bootstrap Exploratory Graph Analysis
In a recent paper, Christensen and Golino (2019b) developed a new bootstrap approach, called bootEGA, to estimate and evaluate the structural consistency of a dimensional structure estimated using EGA. Structural consistency is defined as the extent to which a dimension is interrelated (internal consistency) and homogeneous (test homogeneity) in the presence of other related dimensions (Christensen et al., 2020). Such a measure provides an alternative yet complementary approach to internal consistency measures in the factor analytic framework. The bootEGA approach has two main metrics for structural consistency: Investigate the integrity of the dimensionality structure and the robustness of each item's placement within those dimensions.
For the bootstrap, we implemented the parametric procedure, which begins by estimating a network using EGA and then generating new replicate data from a multivariate normal distribution (with the same number of cases as the original distribution). EGA is then applied to the replicate datasets, continuing iteratively until the desired number of samples is achieved (e.g., 500; Christensen & Golino, 2019b). The result is a large number of replicate EGA networks, which form a sampling distribution of EGA results. Two reasons to use a parametric bootstrap are (1) resampling (or nonparametric bootstrap) smaller samples increases the influences that outlier cases can have on the estimated sampling distribution and (2) its higher accuracy in detecting the correct dimensionality structure in the simulated populations (Christensen & Golino, 2019b).
There are several measures that can be derived from the sampling distribution of the EGA results. First, descriptive statistics – median number of dimensions, confidence intervals around the median, and the number of times the number of dimensions replicates – can be derived. Of these measures, we will focus on the number of times the number of dimensions replicate. Second, a median network structure can be computed by taking the median value of each edge across the replicate networks and building a single network using these values. The Walktrap algorithm (Pons & Latapy, 2006) is then applied, resulting in the dimensions that would be expected for a typical network from the EGA sampling distribution. Third, structural consistency or the proportion of times that each empirically derived dimension (i.e., the result from the initial EGA) was exactly recovered (i.e., identical item composition) from the replicate bootstrap samples (Christensen et al., 2020). Finally, item stability statistics or how often an item replicates in their respective empirically derived dimension but also in the other dimensions identified in the replicate networks. This provides information about which items may be causing the instability of an overarching dimension in the structural consistency analysis. Our goal for the CCES is to have separate but related constructs such that the dimensions were homogeneous (even in the presence of the other dimensions). Therefore, we used the item stability analysis to determine which items to retain to maximize each dimension's structural consistency.
For the structural consistency, values of what are “acceptable” have not been established; however, general rules of thumb are subject to the scale developer's intent (Christensen et al., 2020). In our case, the goal was to have separate but related constructs that were interrelated. This means that we expect that some dimensions may be less stable than others due to some of their interrelations. Because of this, we set a value of .75 or higher (i.e., the dimension replicates exactly across 75% of the bootstrap samples) as our benchmark for acceptable. We set a similar benchmark for the item stability where items were expected to replicate at least 75% of the time in their empirically derived (i.e., EGA) dimension.
Data Analysis
All analyses were implemented in R. To investigate the cross-construct relations present in the CCES structure, we employed parallel analysis combined with exploratory factor analysis (EFA) and EGA. For both empirically derived dimensionality structures, the fit of the CCES structure was verified using the total entropy fit index (TEFI). We note that performing exploratory and CFA on the same data leads to overestimates of fit measures – that is, the model can overfit the dataset yielding inflated estimates of model fit and parameter estimates (Fokkema & Greiff, 2017). Therefore, fitting traditional fit measures via CFA modeling is not justified. The TEFI provides estimation of fit without a confirmatory model, which makes it an appropriate choice for exploratory situations (Golino, Moulder, et al., 2020).
Importantly, the TEFI index has been shown to outperform the traditional fit indices used in CFA (Golino et al., 2020). In a recent simulation study, CFI, RMSEA, TFI, and standardized root mean square residual (SRMR) performed very poorly for the identification of the correct number of factors when used with the traditional cutoff values (≤ 65%). When taken as relative measures, these traditional fit indices performed much better (between 80% and 90%). Still, TEFI performed even better, identifying the correct number of factors above 93% in most of the conditions tested.
Parallel analysis with resampling (100 iterations and using the 95% quantile of simulated eigenvalues to determine the number of factors) and EFA (with oblimin rotation) were implemented using the psych package (Revelle, 2017). Both techniques used polychoric correlations and the generalized weighted least squares (GLS) factoring method. The number of empirical eigenvalues larger than the number of resampled eigenvalues obtained via parallel analysis were used to inform the number of factors to retain via EFA.
EGA was applied using the EGAnet package (Golino & Christensen, 2019). The networks were estimated using the Gaussian Graphical Model via the EBICglasso regularization technique in the qgraph package (Epskamp et al., 2012), and the dimensions were estimated via the Walktrap algorithm in the igraph package (Csardi and Nepusz [2006]; see ESM 1 for details). After estimating the dimensionality structure of the instrument, the parametric bootEGA algorithm (500 replicas) was used to investigate the stability of the estimates. A median network structure was computed as the median of the regularized partial correlations estimated in each of the 500 bootstraps, and the Walktrap algorithm was used to estimate the number of factors in this typical network. Finally, the structural consistency and item stability analyses were implemented using the dimStability function. The bootEGA algorithm and structural consistency analyses were both implemented using the EGAnet package (Golino & Christensen, 2019). The R code used in this paper, as well as the dataset and the RMarkdown file with the manuscript text, is available on https://osf.io/8sj6v/.
Results
Parallel Analysis and Exploratory Factor Analysis
Figure 1 shows the empirical and resampled eigenvalues obtained in the parallel analysis procedure, indicating five latent factors. Inspecting the pattern matrix (with loadings > .40) of the EFA with the GLS factoring method and oblimin rotation enables the interpretation of the five factor as follows. Factor 1 (willingness to do activities) is composed by the items CE1, CE4, CE9, CE14_R, CE17, and CE18, and factor 2 (sociability or care for others), by the items Empathy, CE8, CE12, and CE16, while factor 3 (self-regulation) is formed by the items CE2, CE5_R, CE6, and CE13_R. Factor 4 (carelessness and fantasy) is composed by items CE7_R, CE10_R, and CE11_R, while factor 5 (Concentration and good behavior) is formed by items Normalization, Concentration, CE3_R, and CE15. Notably, there were several variables that had sizable cross-loadings (≥ .30): CE6, CE10_R, CE3_R, Concentration, and CE15. Table 3 shows the standardized loadings per factor.
The fit of the five-factor structure identified via EFA was estimated using TEFI. The value of the TEFI for the five-factor solution was .19.
EGA and bootEGA
Figure 2 shows the estimated dimensionality of the Children's Concentration and Empathy Scale. Four dimensions were estimated: (1) a mix of items reflecting emotions and adequate/inadequate behaviors (CE2, CE5_R, CE6, CE10_R, CE13_R, and CE15), (2) fantasy (CE7_R and CE11_R), (3) willingness to do activities (Normalization, Concentration, CE3_R, CE4, CE9, CE14_R, CE17, and CE18), and (4) sociability or care for others (Empathy, CE8, CE12, and CE16).
The fit of the four-factor structure estimated via EGA was also verified using TEFI, which presented a value of −15.35. The TEFI value was lower than the value obtained by the hypothetical (or theoretical) five-factor model and the empirical five-factor model obtained via EFA. The TEFI result indicates a better fit of the dimensionality structure estimated via EGA compared to the structure estimated using EFA.
The right of Figure 2 depicts the dimensionality structure of the median network structure (i.e., the network computed as the median of the regularized partial correlations estimated across the bootstrap networks). Because the median network structure reflects the same factors estimated via EGA (Figure 2), the original four-factor solution represents a good portion of the dimensional solutions found in the EGA sampling distribution.
Digging deeper into the bootEGA results, we see that other dimensionality structures also occurred frequently, specifically a structure with five factors (about 28.00% of the time; Table 4).
The relatively high frequency of the five-factor structure suggests that some items may be forming a separate factor. To explore this further, we applied the item stability measure to examine how stable each item is within each dimension.
As shown in Figure 3, most items are replicating well within the original dimension designated by EGA. Three items, however, are showing relatively low item stability (≤ 75%). These items are CE6 (Willingly follows directions), CE10_R (Is careless, sloppy), and CE15 (Is well-behaved even when unsupervised) are not very stable. As shown in Table 5, factor 1 presented a low structural consistency (≤ .75). This is due to the fact that the three items mentioned above are replicating in the original factor (dimension 1 in EGA) about 75%, 54%, and 66% of the time, respectively.
Further examination of how often these items are replicating in the other dimensions reveals that these three items (CE6, CE10_R, and CE15) were forming their own separate factor about 20% of the time (Table 4). This provides direct evidence of why the four-factor structure was moderately stable and where a fifth factor was coming from in our dimension stability results (Table 5).
Increasing CCES's Dimensional Integrity
All three of these less stable items show a similarity of being related to internalized norms. Dr. Montessori described normalized children as enjoying following instructions and behaving well. Two of the three items match this characterization well (CE6 and CE15), whereas CE10_R (Is careless and sloppy) was conceptually tangential. Indeed, when reviewing each item's stability, CE10_R was the most unstable (.54) within dimension 1 and appeared somewhat regularly in dimension 2 (.14) and in its own dimension (.25) with the other unstable items.
When looking at the EFA loading matrix (Table 3), these items had large cross-loadings on other factors. CE6, for example, had a large cross-loading on factor 1 (.32) suggesting that this item represents more than one dimension. CE15 similarly had a large cross-loading on factor 4 (.34). Finally, CE10_R had a large loading on factor 4 (.42). These large cross-loadings corroborate why these items were leading to structural inconsistency. Notably, other items which had large cross-loadings (CE3_R and Concentration) did not affect the structural consistency as strongly (values ≥ .85; see Figure 3).
Given the instability of these items and our goal of creating structurally consistent dimensions (i.e., interrelated and homogeneous) that are separate but related, we took the approach to remove the three items with the lowest structural consistency (CE6, CE10_R, and CE15) to increase the integrity of the scale's structure and reanalyzed the data using EGA (Figures 4 and 5) and bootEGA (Christensen et al., 2020). By removing the items CE6, CE_10R, and CE15, the structural consistency of the CCES improved drastically (Figure 4): The dimensionality structure became much more stable (four factors were identified in about 89% of the bootstrap samples), which was also reflected in the item stability measure where all items were replicating in their designated EGA dimensions at least 90% of the time (see Table 6).
The final dimensionality structure had four dimensions. The first dimension was characterized by Self-Regulation (CE2, CE5_R, and CE13_R). The second dimension was characterized by Fantasy (CE7 and CE11_R). The third dimension was characterized by Task Engagement (Normalization, Concentration, CE1, CE4, CE9, CE14_R, CE17, and CE18). Finally, the fourth dimension represented Empathy (Empathy, CE8, CE12, and CE16). Each of these dimensions corresponded to one or more of Montessori's characteristics (Table 1; Self-Regulation [char. 3 and 4], Fantasy [char. 6], Task Engagement [char. 1 and 2], and Empathy [char. 4 and 5]).
Discussion
Montessori noted that once children begin to concentrate, an array of personality changes emerge that are related to emotion regulation, sociability, constructive work choices, and good behaviors. We constructed a scale designed to assess these characteristics in children via teacher report. Using EGA, we identified a four-factor structure that corresponded to these characteristics and verified that this structure was stable.
Importantly, we were interested in whether these dimensions are observed by teachers today as co-occurring with concentration. Empathy and concentration did not necessarily co-occur but rather were distinct dimensions. Concentration tended to co-occur more with normalization (being placed in the same dimension), suggesting that teachers see concentration and normalization as more similar to each other than empathy. Several of the characteristics Montessori described as occurring with normalization were also noted by teachers. More specifically, children who were said to concentrate and be normalized were also said to engage their whole self in activities and to choose and persist in challenging as well as purposeful and constructive activities. On the other hand, children who showed high concern for others and helped when others were hurt were considered empathetic, and interestingly, they also demonstrated the esthetic aspect that Montessori noted came with normalization: Teachers said the children appreciated beauty.
In general, the results demonstrate that EGA (and bootEGA) can be an effective tool for assessing and adjusting the dimensional integrity of an assessment instrument (Christensen et al., 2020). A clear example of this is provided by the contributions of items to structural consistency. By traditional factor analytic conventions, there were several items (CE6, CE10_R, CE3_R, Concentration, and CE15) that were multidimensional (cross-loadings ≥ .30). In most cases, researchers who are intent on measuring single attributes would remove these items because they are not reflected as homogeneous constructs (DeVellis, 2017). It is worth pointing that some might argue that a researcher analyzing the same data could use an iterative process of removing items with significant cross-loadings and using factor analysis again until a reasonably clean structure emerges. At the end, a structure somewhat similar to EGA could be found. However, EGA involves less work and fewer iterations and is less subjective. Furthermore, our structural consistency measures showed that items CE3_R and Concentration were relatively stable despite having high cross-loadings. This suggests that these items are structurally consistent with their dimension but may be strongly related to items outside of their dimension. Conversely, items CE6, CE10_R, and CE15 were often forming a separate factor, which may indicate the presence on a minor factor (i.e., correlated residual variance). In contrast to traditional internal consistency measures, bootEGA determines the consistency of a scale's dimensions in a multidimensional context. Such an approach captures the robustness of each dimension's cohesiveness, signaling that the dimension measures what it intends to measure (construct validity). Considering that some dimensions aligned better with Montessori's conceptualization of normalization than others, future development of this scale should consider examining different samples or shifting the target of the construct.
One notable limitation of our sample was that it was nested: We collapsed across schools and teachers. Although there are some approaches that could be used to evaluate these group differences (e.g., Costantini et al., 2019; Haslbeck et al., 2020), our sample sizes would have been too small to make meaningful inferences into a structure that is more likely to be reflected by the broader population. Such a limitation is likely to have effects on the observed structural consistency. Specifically, the structural consistency of the dimensions may be overestimated due to collapsing across group differences. An important future direction for assessing the reliability of the structural consistency results will be to examine how different group structures affect the results.
Our data suggest that empathy, normalization, and concentration are not simultaneously emergent, at least from the teacher's perspective and the current version of the scale. One possibility is that the characteristics emerge in sequence, such that the elements related to concentration and normalization emerge first followed by the elements related to empathy, positive emotion, and decreased fantasy. A second possibility is that the ages of the children examined exert an influence on the effects. Montessori's descriptions might have been particularly geared to children ages 3 to 6, when the phenomenon of concentration first emerged in her classrooms, whereas our sample also included older children. Future research should focus on two age groups (e.g., up to six and over six) to determine if the factor structure is more aligned with Montessori's observations at younger ages only. Conversely, our scale may not be tapping the aspects that relate “normalized” characteristics with empathy. Shifting the scale to include more items that associate between the Task Engagement and Empathy dimensions may make them more related. In either case, more research is needed to evaluate Montessori's theory on the emergence of these characteristics.
Montessori observed changes in children's personalities once they begin to concentrate, which are consistent with Buddhist descriptions regarding meditation's effect. In psychological research, such changes have been noted in studies of mindfulness meditation and with good self-regulation in children. Specifically, changes in emotion regulation, positive affect and mood, kindness and compassion, and concentrated attention co-occur. In this study, we found distinct factors related to these aspects of personality, rather than a single factor encompassing all aspects. The statistical technique employed allowed us to evaluate this structure and improve on it, demonstrating its utility for other scale development applications. Our analytic approach verified the existence of Montessori's characteristics in the CCES, which was shown to adequately measure her characteristics of deep concentration and normalization. Thus, these characteristics measured by the CCES enables researchers to further explore their emergence and allows future research to examine and validate Montessori's observations.
It is worth mentioning that although the structural validity (as well as the stability of the items and dimensions) of a measurement instrument is important in the validation process (Christensen et al., 2020; Saccenti & Timmerman, 2017; Slocum-Gori & Zumbo, 2011), it is just the first step. Future research should also investigate how the CCES relates to other constructs, such as agreeableness, conscientiousness, openness to experience, positive affect, emotional regulation, self-regulation, prosocial behaviors, and focused attention, which are closely related to the definitions of concentration, empathy, and normalization used in the development of the CCES items. It is important to also investigate the concurrent, discriminant, and criterion validity of this new scale. Moreover, verifying whether the dimensionality structure of the CCES is invariant for different sub-populations (e.g., gender, age groups, type of school, and others), especially if the data have a nested structure, should be a goal for future work.
One recommendation for future work is to continue using the 6-point rating scale. Although teachers in our study did not use the full rating scale, it does not mean that others will not. From a psychometric standpoint, we believe that this result was inconsequential. From an applied standpoint, the zero point of the rating scale (the absence of a characteristic) is important. It is possible that a child will not display a certain characteristic in front of a teacher or that the teacher may not believe that the child behaves in a way that is consistent with a characteristic. Keeping the zero point of the rating scale allows teachers to specify a characteristic is absent and may need further attention for development.
Our study has refined the CCES so that the scale's constructs are measuring what they intended to measure – that is, the dimensions of the scale are interrelated (internally consistent) and homogeneous (test homogeneity). Satisfying these structural validity concerns, researchers can interpret the dimension scores with confidence that they are not confounded by other factors (e.g., similar wording; Gerbing & Anderson, 1984; Goldstein, 1980).
Electronic Supplementary Material
The electronic supplementary material is available with the online version of the article at https://doi.org/10.1027/2698-1866/a000008
References
(2010). The contribution of agreeableness and self‐efficacy beliefs to prosociality. European Journal of Personality, 24(1), 36–55. 10.1002/per.739
(2014). The aesthetic brain: How we evolved to desire beauty and enjoy art. Oxford University Press.
(2020). Comparing community detection algorithms in psychological data: A Monte Carlo simulation. PsyArXiv Preprints. 10.31234/osf.io/hz89e
(2019). Estimating the stability of psychological dimensions via bootstrap exploratory graph analysis: A Monte Carlo simulation and tutorial. 10.31234/osf.io/9deay
(2020). A psychometric network perspective on the validity and validation of personality trait questionnaires. European Journal of Personality, 34(6), 1095–1108. 10.1002/per.2265
(2014). Associations between medical student empathy and personality: A multi-institutional study. PLoS One, 9(3), e89254. 10.1371/journal.pone.0089254
(2019). Stability and variability of personality networks. A tutorial on recent developments in network psychometrics. Personality and Individual Differences, 136, 68–78. 10.1016/j.paid.2017.06.011
(2012). Dimensions of normal personality as networks in search of equilibrium: You can't like parties if you don't like people. European Journal of Personality, 26(4), 414–431. 10.1002/per.1866
(2006). The igraph software package for complex network research. InterJournal, Complex Systems, 1695(5), 1–9. https://pdfs.semanticscholar.org/1d27/44b83519657f5f2610698a8ddd177ced4f5c.pdf
(2014). Flow and the foundations of positive psychology: The collected works of Mihaly Csikszentmihalyi. Springer.
(1990). Flow: The psychology of optimal experience (Vol. 1990). Harper & Row.
(2004). Relations of young children's agreeableness and resiliency to effortful control and impulsivity. Social Development, 13(2), 193–212. 10.1111/j.1467-9507.2004.000263.x
(2015). Moving beyond mindfulness: Defining equanimity as an outcome measure in meditation and contemplative research. Mindfulness, 6(2), 356–372. 10.1007/s12671-013-0269-8
(2017). Scale development: Theory and applications (4th ed.). Sage Publications.
(2017). The effect of Montessori and traditional methods of education on emotional intelligence of children. European Journal of Education Studies, 3(4), 367–382. 10.5281/zenodo.399050
(2006). Prosocial development. In W. DamonR. M. LernerN. EisenbergEds., Handbook of child psychology: Vol 3. Social, emotional, and personality development (6th ed., pp. 646–718). Wiley.
(1997). The relations of regulation and emotionality to resiliency and competent social functioning in elementary school children. Child Development, 68(2), 295–311. 10.2307/1131851
(2011). Effortful control: Relations with emotion regulation, adjustment, and socialization in childhood. In K. D. VohsR. F. BaumeisterEds., Handbook of self-regulation: Research, theory, and applications (pp. 263–283). Guilford.
(2004). The relations of effortful control and impulsivity to children's resiliency and adjustment. Child Development, 75(1), 25–46. 10.1111/j.1467-8624.2004.00652.x
(2019). psychonetrics: Structural equation modeling and confirmatory network analysis. R package verson 0.3.3. https://CRAN.R-project.org/package=psychonetrics.
(2012). qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software, 48(4), 1–18. 10.18637/jss.v048.i04
(2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. 10.1177/1948550617693063
(2015). Promoting prosocial behavior and self-regulatory skills in preschool children through a mindfulness-based kindness curriculum. Developmental Psychology, 51(1), 44–51. 10.1037/a0038256
(2017). How performing PCA and CFA on the same data equals trouble: Overfitting in the assessment of internal structure and some editorial thoughts on it. European Journal of Psychological Assessment, 33(6), 399–402. 10.1027/1015-5759/a000460
(2010). Community detection in graphs. Physics Reports, 486(3–5), 75–174. 10.1016/j.physrep.2009.11.002
(1984). On the meaning of within-factor correlated measurement errors. Journal of Consumer Research, 11(1), 572–580. 10.1086/208993
(2009). The empathic power of enactments: The link between neuropsychological processes and an expanded definition of empathy. Psychoanalytic Psychology, 26(3), 290–309. 10.1037/a0016449
(1980). Dimensionality, bias, independence and measurement scale problems in latent trait test score models. British Journal of Mathematical and Statistical Psychology, 33(2), 234–246. 10.1111/j.2044-8317.1980.tb00610.x
(2016). Mindfulness: A practical guide to awakening. Sounds True.
(2019). EGAnet: Exploratory graph analysis: A framework for estimating the number of dimensions in multivariate data using network psychometrics. https://CRAN.R-project.org/package=EGAnet
(2017). Estimating the dimensionality of intelligence like data using exploratory graph analysis. Intelligence, 62, 54–70. 10.1016/j.intell.2017.02.007
(2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLoS One, 12(6), e0174035. 10.1371/journal.pone.0174035
(2020). Investigating the structure of the Children's Concentration and Empathy Scale using exploratory graph analysis [Data set]. https://osf.io/8sj6v/
(2020). Entropy fit indices: New fit measures for assessing the structure and dimensionality of multiple latent variables. Multivariate Behavioral Research. Advance online publication. 10.1080/00273171.2020.1779642
(2020). Investigating the performance of exploratory graph analysis and traditional techniques to identify the number of latent factors: A simulation and tutorial. Psychological Methods, 25(2), 292–320. 10.1037/met0000255
(2007). Agreeableness, empathy, and helping: A person × situation perspective. Journal of Personality and Social Psychology, 93(4), 583–599. 10.1037/0022-3514.93.4.583
(2009). You are here: The power of the present moment. Shambhala.
(2011). Inhibitory control and empathy-related personality traits: Sex-linked associations. Brain and Cognition, 76(3), 364–368. 10.1016/j.bandc.2011.04.004
(2020). A tutorial on estimating time-varying vector autoregressive models. Multivariate Behavioral Research, 56(1), 120–149. 10.1080/00273171.2020.1743630
(2007). A randomized controlled trial of mindfulness meditation versus relaxation training: Effects on distress, positive states of mind, rumination, and distraction. Annals Behaviour Medicine, 33(1), 11–21. 10.1207/s15324796abm3301_2
(2020). How to compare psychometric factor and network models. Journal of Intelligence, 8(4), 35. 10.3390/jintelligence8040035
(2009). Association of an educational program in mindful communication with burnout, empathy, and attitudes among primary care physicians. Journal of the American Medical Association, 302(12), 1284. 10.1001/jama.2009.1384
(1996). Montessori today: A comprehensive approach to education from birth to adulthood. Random House LLC.
(2019). Shunned and admired: Montessori, self-determination, and a case for radical school reform. Educational Psychology Review, 31, 939–965. 10.1007/s10648-019-09483-3
(2014). Emotion regulation moderates the association between empathy and prosocial behavior. PLoS One, 9(5), e96555. 10.1371/journal.pone.0096555
(1987). Validity. ETS Research Report Series, 1987(2), i–208. 10.1002/j.2330-8516.1987.tb00244.x
(1948). To educate the human potential. Kalakshetra Publications.
(1961). What you should know about your child. Montessori-Pierson.
(1965). Spontaneous activity in education: The advanced montessori method. Schocken.
(1966). The secret of childhood. Ballantine.
(1967a). The absorbent mind. Henry Holt.
(1967b). The discovery of the child. Ballantine.
(1989). The child, society and the world: Unpublished speeches and writings (Vol. 7). Clio.
(1994). Creative development in the child II. Kalakshetra Press.
(2018). Why do traits come together? The underlying trait and network approaches. In V. Ziegler-HillT. K. ShackelfordEds., The SAGE handbook of personality and individual differences: The science of personality and individual differences (pp. 130–151). Sage Publications. 10.4135/9781526451163.n6
(2020). What's in a task? Complications in the study of the task-unrelated-thought variety of mind wandering. Perspectives on Psychological Science, 15(3), 572–588. 10.1177/1745691619897966
(1977). Roots of caring, sharing, and helping: The development of pro-social behavior in children. WH Freeman.
(2007). An examination of self-compassion in relation to positive psychological functioning and personality traits. Journal of Research in Personality, 41(4), 908–916. 10.1016/j.jrp.2006.08.002
(2006). Computing communities in large networks using random walks. Journal of Graph Algorithms and Applications, 10(2), 191–218. 10.7155/jgaa.0018510.7155/jgaa.00124
. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
(2017). psych: Procedures for psychological, psychometric, and personality research. Northwestern University. https://CRAN.R-project.org/package=psych
(2006). Executive function and the promotion of social–emotional competence. Journal of Applied Developmental Psychology, 27(4), 300–309. 10.1016/j.appdev.2006.04.002
(2017). Considering Horn's parallel analysis from a random matrix theory point of view. Psychometrika, 82(1), 186–209. 10.1007/s11336-016-9515-z
(2015). Enhancing cognitive and social–emotional development through a simple-to-administer mindfulness-based school program for elementary school children: A randomized controlled trial. Developmental Psychology, 51(1), 52–66. 10.1037/a0038454
(2012). The psychological effects of meditation: A meta-analysis. Psychological Bulletin, 138(6), 1139–1171. 10.1037/a0028168
(2018). Factors influencing the development of empathy and pro-social behaviour among adolescents: A systematic review. Children and Youth Services Review, 94, 421–436. 10.1016/j.childyouth.2018.07.027
(2011). Assessing the unidimensionality of psychological scales: Using multiple criteria from factor analysis. Social Indicators Research, 102(3), 443–461. 10.1007/s11205-010-9682-8
(1978). Empathy, fantasy and helping. Sage.
(2015). The neuroscience of mindfulness meditation. Nature Reviews Neuroscience, 16(4), 213–225. 10.1038/nrn3916
(2008). Looking back to the future: The current relevance of Maria Montessori's ideas about the spiritual well-being of young children. The Journal of Student Wellbeing, 2(2), 1–15. 10.21913/JSW.v2i2.392
(2015). Does brief mindfulness training increase empathy? The role of personality. Personality and Individual Differences, 86, 492–498. 10.1016/j.paid.2015.07.011
(1995). Self-regulation involves more than metacognition: A social cognitive perspective. Educational Psychologist, 30(4), 217–221. 10.1207/s15326985ep3004_8
