Elsevier

Early Childhood Research Quarterly

Volume 51, 2nd Quarter 2020, Pages 55-66
Early Childhood Research Quarterly

Computerized social-emotional assessment measures for early childhood settings

https://doi.org/10.1016/j.ecresq.2019.07.002Get rights and content

Highlights

  • Computerized emotion knowledge and social problem solving measures evaluated.

  • Cross-version and internal consistency reliability were adequate.

  • Concurrent validity with an in-person mode also was acceptable.

  • Measures also predicted, as theoretically expected, aspects of early school success.

  • Improvements suggested; also planning increased exportability, teacher access.

Abstract

Social-emotional competence (SEC) is increasingly acknowledged by parents, educators, and lawmakers as central to school success. Given the tremendous SEC gains made by preschoolers, early childhood educators need access to sensitive assessment tools that enable them to monitor and tailor instruction to individual children’s needs. Computerized direct assessment tools have several advantages to meet these needs, including inherent interest to children and ease of use for teachers. Thus, we evaluated the psychometric adequacy of computerized assessment tools measuring two key aspects of preschoolers’ SEC: emotion knowledge and social problem solving. Participants included 450 preschoolers from three regions. We used two versions each of two measures widely used in research: The Affect Knowledge Test, Shortened (AKT-S) and Challenging Situations Task (CST). Both were administered via in-person and computerized modes, in counterbalanced orders. For each computerized administration, observers rated children’s computer competence and interest in the assessment process. Analyses examined internal consistency reliability of the computerized measures. Interrelations and mean differences between computerized and in-person modes for each measure were used to demonstrate concurrent validity of the computerized measures. Because of the importance of SEC for early school success, associations of the computerized measures with aggregate teacher ratings of social-emotional behavior and learning behaviors/attitudes were used as indicators of predictive validity. Findings showed that the computerized AKT-S and CST appear reliable. Further, for concurrent validity, both are related to, and do not differ from, the in-person mode. Predictive validity relations were stronger for the AKT-S than the CST, therefore validity of the CST should be probed further. Discussion centers on advantages of using these computerized measures, and how teachers could be supported to use them.

Introduction

Ensuring that children are on positive school developmental trajectories at school entry is essential for later academic and social success. Previous research has highlighted social-emotional competence (SEC) as being especially important in establishing such success (Denham, Brown, & Domitrovich, 2010; Murray & Harrison, 2011; Romano, Babchishin, Pagani, & Kohen, 2010). In particular, the extant literature has identified educators as key promoters of Social Emotional Learning (SEL) across five important areas of competence: (a) self-management, the ability to regulate thoughts, emotions, and behaviors, (b) self-awareness, including the ability to recognize one’s emotions, (c) social awareness of culture, beliefs, and feelings of others, (d) relationship skills, the abilities to effectively communicate with and work well with peers, and build meaningful relationships, and (e) responsible decision-making, including the abilities to make plans for the future, and solving social problems (Oberle, Domitrovich, Meyers, & Weissberg, 2016; Payton et al., 2000). A preschooler who has attained such age-appropriate SEC skills is more likely than one who has not to become the kindergartner who can better plan and pay attention to academic tasks and devote more resources to learning. S/he can benefit more from teachers’ instructions, sharing academic information and resources with peers and modeling peers’ learning skills (Denham et al., 2010, Romano et al., 2010). In this study we focus on two SEC skills that are important during early childhood: emotion knowledge and social problem solving.

There is increasing stakeholder acknowledgement that SEC is key in the transition from preschool to elementary school. Both parents and early childhood educators point to an urgent need for SEC programming, especially for children who live in low income families and/or are members of minority groups who historically have faced challenges to academic achievement (Bridgeland, Bruce, & Hariharan, 2013; Piotrkowski, Botsko, & Matthews, 2000). Further, all state education systems and Head Start programs now incorporate SEC skills, including emotion knowledge and social problem solving, in prekindergarten learning standards (albeit with fewer indicators and implemented less systematically in comparison to cognitive skills; see Dusenbury et al., 2015). National legislation has also been introduced in the US authorizing funds for technical assistance, training, and programming on SEC (O’Connor, DeFeyter, Carr, Luo, & Romm, 2017). As a result, SEC is receiving increasing attention and incorporation in early childhood instruction.

Therefore, because SEC skills are so important and viewed as such not only by academicians, but vehemently by early childhood educators, parents, and even legislators, it would behoove us to assess them well. However, educators lack the requisite psychometrically sound tools to identify, track and assess SEC skills (McKown, 2017). Without sound measurement tools to facilitate early assessment of SEC, educators cannot best tailor instruction to each child, and stakeholders cannot ascertain progress or whether a given SEC curriculum is more or less fruitful (Denham, Bassett, & Zinsser, 2012; Denham, 2015). Hence, our goal in this study is to promote useful SEC assessments measuring preschoolers’ emotion knowledge and social problem solving.

There are, of course, definite requirements for such assessment tools (Denham, 2015; Kendziora, Weissberg, Ji, & Dusenbury, 2011). They should be developmentally appropriate, integrated with curricula, beneficial to all parties, primarily reliant on the child’s everyday activities in realistic situations, culturally and linguistically responsive, and reflect children’s actual performance (Copple & Bredekamp, 2009). Data emanating from such assessment should, moreover, not be used for high stakes decisions, such as retention in kindergarten. Instead, formative and summative functions of assessment should be undertaken to effectively identify children needing intervention or higher-level services, highlight specific needs of children and classrooms in terms of instruction, and show overall effects of programming (Denham, 2015; Denham, Ferrier, Howarth, Herndon, & Bassett, 2019).

With these stipulations in mind, our research team has developed several well-validated assessments for aspects of SEC, including direct in-person assessments of emotion knowledge with the Affect Knowledge Test-Shortened (AKT-S) and social problem solving with the Challenging Situations Task (CST). The AKT-S and CST measures have been widely used by researchers around the world (e.g., Leyva, Berrocal, & Nolivos, 2014; Kılıç & Aytar, 2016; Rebelo, Verissimo, Machado, & Silva, 2013; Sette, Bassett, Baumgartner, & Denham, 2015; Sheard, Ross, & Cheung, 2013; Upshur, Heyman, & Wenz-Gross, 2017). In fact, one recent meta-analysis found that over a third of all effect sizes in studies of children’s emotion socialization were measured with the AKT (Zinsser, Gordon, & Jiang, 2019).

As theoretically predicted, research has consistently shown that the AKT and CST predict social competence, classroom adjustment, and academic readiness (e.g., Denham, Bassett, Way, et al., 2012; Denham et al., 2013; Denham, Way, Kalb, Warren-Khot, & Bassett, 2013; Denham, Bassett, Zinsser, & Wyatt, 2014). The theoretical proposition underlying the AKT’s relations with these attributes is that emotion knowledge “frees up” personal resources, allowing young children to interact positively with others in the preschool environment, and focus more specifically on learning. Regarding social problem solving as assessed by the CST, being calmly “ok” or sad instead of angry during a peer provocation may allow the child to take time to think about a prosocial solution, whereas the angry child may merely lash out aggressively, such that sad or “just ok” self-emotion choices on the CST are related to their teacher-rated social competence and classroom adjustment (Denham et al., 2013; Izard, Stark, Trentacosta, & Schultz, 2008; Schultz et al., 2010). In contrast, choosing happy emotion responses connotes lack of understanding or denial that a provocation has occurred (or perhaps social desirability in the context of assessment). Choosing angry emotion responses would more directly relate to deficits of social competence and classroom adjustment (Orobio de Castro, Merk, Koops, Veerman, & Bosch, 2005). Regarding behavior response choices on the CST, children who choose socially competent behaviors as part of response decision-making, as opposed to aggressive or passive behaviors, are likely to demonstrate greater social competence and overall classroom adjustment.

Despite these theoretical foundations, the popularity of these tools among researchers, and the evidence base supporting their use, their administration and scoring by trained examiners makes them prohibitive for classroom teachers’ use. To provide educators with the most psychometrically rigorous tools, we have adapted these tools to maximize their potential utility and feasibility in preschool, Head Start, and childcare classrooms. Our goals are to (a) allow teachers to track students’ progress and inform instruction (formative assessment), (b) measure child outcomes (summative assessment), and (c) evaluate program outcomes. To accomplish these goals, we adapt these in-person measures for electronic administration, using tablet devices.

In sum, given the growing importance of SEC and its concomitant assessment in the early childhood classroom, the focus of the current study is to evaluate the psychometric efficacy of two direct assessment tools measuring emotion knowledge and social problem solving, adapted for computer administration. The goal is, with continued improvement based on these evaluations, to use these tools to inform early childhood instruction. In accessing what the child knows and thinks, direct assessment can complement and add to information emanating from the observable and ratable behavioral aspects of SEC (McKown, 2017).

Why computerize measures of early childhood SEC? In the busy early childhood classroom, teachers must have easy-to-administer assessment tools. Tablets have become very common in preschool classrooms in the last two decades (e.g., Fletcher, Whitaker, Marino, & Anderson, 2014; NAEYC, 2012), and meet this need. Other principal advantages of computer-based systems over conventional assessment methods are that: (a) stimuli are standardized, making assessment more precise; (b) significant savings can be made in both time and labor; (c) scoring can be immediately available, without error-prone optical scoring; and (d) training for assessment administrators (i.e., teachers or educational resource personnel) can be dramatically reduced and streamlined. In attempts to computerize our preschool SEC measures, we strove to meet these criteria, to help teachers move toward use of electronic portfolios of both formative and summative SEC assessment.

Such assessment also can take advantage of the capabilities of technology for animation, speech, and sound. In fact, young children engage with educational software as soon as they can manipulate a touchscreen (NAEYC, 2012; Vandewater et al., 2007). They show intense interest and pleasure, and surprising stamina, in interacting with computers (Ellis and Blashki, 2004, McCarrick and Li, 2007). Thus, computer use, whether for instruction or assessment, can be a highly motivating, positive experience for young children.

In summary, to facilitate teacher use of these SEC assessments in effective, efficient, and valid ways, it is crucial to develop appropriate means of standardizing and streamlining direct assessments emotion knowledge and social problem solving, via computer usage. Although our original research-based measures are valuable in predicting early school success, the training, coding, and administration requirements that attend them can be “deal breakers” for applied use in the early childhood classroom. If we are to make these means of assessment useful, to move toward both formative and summative assessment, we feel that they must be computerized, with much thought given to user support (for, e.g., preschool, Head Start, and childcare teachers, Head Start mental health consultants, and others). We also consider making the assessment attractive and fun to children to be an important goal, to maintain their engagement and enjoyment of the learning process.

In this study, we present findings from our direct assessments of emotion knowledge and social problem solving. A first step in creating usable assessment tools is to examine psychometric properties of the measures. To that end, data from in-person and computerized modes of each measure were collected. We used with two versions of each measure within each mode, to yield answers for four research questions. Research Question 1 (RQ1) asks whether internal consistency and between-version temporal stability for each version of the computerized mode are adequate. The adequacy of concurrent validity, via mean equivalence of in-person and computerized modes, is addressed in Research Question 2 (RQ2). For Research Question 3 (RQ3), we further examine the adequacy of the computerized versions’ concurrent validity via associations between corresponding versions across in-person and computerized modes. Finally, in Research Question Four (RQ4) we determine the adequacy of the computerized versions’ predictive validity via associations between the computerized mode and both teacher-rated SEC and learning behavior/attitudes (given the importance of emotion knowledge and social problem solving to early school success, and in accordance with the theoretical propositions related to these associations).

Section snippets

Participants

Participants included 450 preschoolers from urban, suburban, and more rural preschools (nsuburban= 337; nurban= 40; nrural= 73). Because we collected data in three locations, we examined regional differences. Participants’ average age across all regions was 51 months, but age differed across regions (F (2, 442) = 11.25, partial η2 = 0.05, p < 0.001); mean age in the suburban group was 52 months, but in both the urban and rural samples, the average age was 48 months (ranges 35–71 months,

Research question 1: reliability

First, all associations between versions of both measures were generally significant, exemplifying short-term test-retest reliability (see Table 1). Over one- to three-week intervals, correlations for the AKT ranged from weak in magnitude for negative recognition to moderate for other subscales. For CST emotion choices, all correlations were moderate in magnitude, and for CST behavior choices these ranged from weak for avoidant/passive choices to moderate for the other three choices. Thus, over

Discussion

For early childhood educators to make informed decisions about how best to support children’s social-emotional development, they must have access to rigorous formative assessments of children’s competencies that can easily be administered in classroom settings. This study set out, therefore, to develop and validate computer-based parallel versions of two highly regarded measures of children’s emotion knowledge (the AKT-S) and social problem solving (the CST). These measures are psychometrically

References (66)

  • K.L. Carson et al.

    Computer-administrated versus paper-based assessment of school-entry phonological awareness ability

    Asia Pacific Journal of Speech, Language and Hearing

    (2011)
  • L.A. Clark et al.

    Constructing validity: Basic issues in objective scale development

    Psychological Assessment

    (1995)
  • L. Clarke et al.

    Young pupils’, their teacher’s and classroom assistants’ experiences of iPads in a Northern Ireland school: “Four and five years old, who would have thought they could do that?”

    British Journal of Educational Technology

    (2016)
  • C. Copple et al.

    Developmentally appropriate practice in early childhood programs serving children from birth through age 8

    (2009)
  • B. Csapó et al.

    Computer-based assessment of school readiness and early reasoning

    Journal of Educational Psychology

    (2014)
  • S.A. Denham et al.

    “Plays nice with others”: Social-emotional learning and academic success

    Early Education and Development

    (2010)
  • S.A. Denham et al.

    Computerizing social-emotional assessment for school readiness: First steps toward an assessment battery for early childhood settings

    Journal of Applied Research on Children: Informing Policy for Children at Risk

    (2012)
  • S.A. Denham et al.

    Preschoolers’ emotion knowledge: Self-regulatory foundations, and predictions of early school success

    Cognition and Emotion

    (2012)
  • S.A. Denham et al.

    Social and emotional information processing in preschoolers: Indicator of early school success?

    Early Child Development and Care

    (2013)
  • S.A. Denham et al.

    Preschoolers' social information processing and early school success: The Challenging Situations Task

    British Journal of Developmental Psychology

    (2013)
  • S.A. Denham et al.

    “How would you feel? What would you do?” Development and underpinnings of preschoolers’ social information processing

    Journal of Research in Childhood Education

    (2014)
  • S.A. Denham et al.

    How preschoolers’ social-emotional learning (SEL) predicts their early school success: Developing theory-promoting, competency-based assessments

    Infant and Child Development

    (2014)
  • S.A. Denham

    Assessment of SEL in educational contexts

  • S.A. Denham et al.

    “I know how you feel”: Preschoolers’ emotion knowledge contributes to early school success

    Journal of Early Childhood Research

    (2015)
  • S.A. Denham et al.

    Key considerations in assessing young children’s emotional competence

  • L.A. Dusenbury et al.

    The case for preschool through high school state learning standards for SEL

    Handbook of social and emotional learning: Research and practice

    (2015)
  • K. Ellis et al.

    Toddler techies: A study of young children’s interaction with computers

    Information Technology in Childhood Education Annual

    (2004)
  • J. Fantuzzo et al.

    Preschool approaches to learning and their relationship to other relevant classroom competencies for low-income children

    School Psychology Quarterly

    (2004)
  • E.N. Fletcher et al.

    Screen time at home and school among low-income children attending head start

    Child Indicators Research

    (2014)
  • F.M. Ihmeideh

    Assessment of children’s digital courseware in light of developmentally appropriate courseware criteria

    British Journal of Educational Technology

    (2015)
  • C.E. Izard et al.

    Beyond emotion regulation: Emotion utilization and adaptive functioning

    Child Development Perspectives

    (2008)
  • U. Johnson

    A step in the right direction: Development of the computerized assessment for preschool social emotional learning (CAPSEL)

    Journal of Applied Research on Children: Informing Policy for Children at Risk

    (2012)
  • C. Kamler et al.

    Using multiple child assessments to inform practice in early childhood programs: Lessons from Milpitas Unified School District

    (2014)
  • Cited by (5)

    • Intervention response among preschoolers with ADHD: The role of emotion understanding

      2021, Journal of School Psychology
      Citation Excerpt :

      Total scores on the human and cartoon faces were combined, with higher scores indicative of better ERU (α's = 0.72–0.78 across cartoon and human faces, pre- and post-intervention). This task has shown good internal consistency and test-retest reliability, along with concurrent and predictive validity (Denham, 2006; Denham et al., 2012, 2020). Previous research using this task has shown ERU be associated with school readiness (Denham et al., 2012; Shields et al., 2001), EF (Denham et al., 2012), teacher and peer rating of prosocial behavior (Cassidy, Werner, Rourke, Zubernis, & Balaraman, 2003; Dunsmore & Karn, 2004), academic achievement (Leerkes et al., 2008), attention (Rhoades, Warren, Domitrovich, & Greenberg, 2011), and has demonstrated sensitivity to intervention effects (Graziano & Hart, 2016).

    • The PETaL approach to bilingual and intercultural education in early childhood education

      2022, Research Anthology on Early Childhood Development and School Transition in the Digital Era

    The present study was funded by NICHD grant #R21HD068744 and IES grant #IES R305A110730. We are grateful to the many children, families, and teachers who participated in this study, and the directors of the facilities who so cooperatively worked with us. Special thanks go to Andrew Steed, Adrian Grajdeanu, and Dustin Rhodes for programming our computerized measures. We also thank Claire Christensen, Candace DeVito, Charlotte Edwards, Nicole Fettig, Grace Howarth, Sarah Lim, Sara O’Malley, Tanvai Rawat, Caitlin Tarr, Ruvi Tsokodayi, and Naomi Watanabe for their major contributions to this work, as well as the numerous other graduate and undergraduate research assistants who also administered the measures.

    View full text