Assessing students' executive functions in the classroom: Validating a scalable group-based procedure

Abstract

We describe and validate a novel, scalable, group-based assessment of executive functions (EFs) in a classroom setting using tablet computers. Relative to the conventional method of a more controlled, one-on-one individual assessment (IA), the group assessment (GA) can be administered quickly to many students, requires less training for assessors, and measures performance in a naturalistic classroom setting. In a socioeconomically and ethnically diverse sample of 269 students in third through fifth grade, we show that IA and GA scores for the same tasks were highly inter-correlated, equally reliable, and showed analogous associations with known EF covariates. IA and GA scores independently predicted teacher-rated self-regulated classroom behavior and standardized test scores. Further, only the GA score emerged as a unique predictor of academic achievement when controlling for prior achievement. We are sharing the tablet apps, source code, and supporting materials for this GA procedure at no cost under an open-source license.

Introduction

Executive function (EF) skills have been linked to various educational outcomes, including specific academic skills, school engagement, and self-regulated classroom behaviors (Diamond, 2013, Obradović et al., 2012). However, the conventional approach to EF assessment is to measure children's performance on standard EF tasks in a highly controlled, laboratory-like setting, typically with a ratio of one child to one assessor. This approach lacks the ecological validity of assessment in a classroom setting—where children practice and apply EF skills daily—and does not scale well for collecting data from a large number of students. We developed a new procedure to simultaneously assess EF skills in all students in a classroom using standard EF tasks administered on tablet computers. The goals of the current study are to validate this new assessment by: (1) examining convergent validity with conventional individual assessment procedures; (2) comparing students' EF performance across group and individual assessment settings; (3) comparing associations of EFs with known demographic and educational covariates across the two assessment settings; and (4) investigating the predictive validity of EF skills assessed in group versus individual assessment settings for teachers' reports of students' self-regulated classroom behaviors and their academic achievement on standardized tests.

EFs are a set of higher-order cognitive skills that enable children to inhibit their impulses, control inappropriate behaviors, ignore distractions, hold and manipulate information in the mind, and shift between competing rules or attentional demands. As such, EF skills are implicated in many aspects of school success. Over the last decade, researchers have linked direct assessments of EF skills to teachers' reports of students' self-regulated classroom behaviors, such as their ability to follow instructions, stay focused on tasks, and work collaboratively with peers (Ciairano et al., 2007, Diamond, 2013, Obradović et al., 2012, Rimm-Kaufman et al., 2009). However, most of these studies have been conducted in early childhood. Researchers working with this age group often employ a composite of EF tasks tapping into multiple EF components (Fuhs et al., 2015, Neuenschwander et al., 2012, Sasser et al., 2015). However, more research is needed to better understand how similar direct assessments of EFs relate to self-regulated classroom behaviors in middle childhood.

In addition to their role in promoting self-regulated behaviors, EF skills also contribute directly to academic performance. For example, solving math problems requires children to flexibly shift attention between different strategies and to manipulate and update information in working memory (Blair, Ursache, Greenberg, Vernon-Feagans, & Family Life Project Investigators, 2015). Although empirical evidence is most robust for the association between working memory and math skills (Bull and Lee, 2014, Jacob and Parkinson, 2015), meta-analytic studies have demonstrated that direct assessments of inhibitory control (Allan, Hume, Allan, Farrington, & Lonigan, 2014), working memory (Friso-van den Bos, van der Ven, Kroesbergen, & van Luit, 2013) and cognitive flexibility (Yeniad, Malda, Mesman, van IJzendoorn, & Pieper, 2013) are all associated with children's performance on literacy and math achievement tests.

Researchers often assess children's EF skills in university-based laboratory settings using a battery of developmentally appropriate standard tasks administered by a highly trained research assistant (Carlson, 2005, Kochanska and Knaack, 2003). There are also many school-based studies, but these typically mimic a laboratory setting: researchers take children out of their classrooms to be assessed one-on-one in a quiet space such as a library room (Blair and Razza, 2007, Raver et al., 2011, Schmitt et al., 2015, Weiland et al., 2014). The assessor works closely with the child to explain the task instructions, provide guidance and feedback during practice trials, and ensure focused completion of the test trials. This context provides many external motivators (both intentional and inadvertent) for children to perform well on EF tasks — motivators that are not normally present in the classroom. Assessors are trained to establish good rapport with participants and express a caring and affirmative demeanor. They provide positive encouragement during practice trials, physical proximity during test trials, and praise after the task is completed. This individualized attention may motivate some children to (try harder to) perform well on the tasks and may contribute to artificially inflated EF performance that does not reflect the child's ability to engage EF skills in a more natural setting. Conversely, some children may be more comfortable in the classroom or better motivated by the presence of peers and teachers, and thus may underperform in a laboratory setting. Individual assessment minimizes the external distractions and interpersonal dynamics present in the classroom, and it provides controlled testing conditions that include constant monitoring and timed positive feedback (Silver, 2014), but it lacks ecological validity.

Ecological validity is an aspect of research design that refers to the similarity between the participants, materials, and settings used in a study and the real-world context under investigation (Shadish, Cook, & Campbell, 2002). By better aligning the assessment context with real-world conditions in which children employ their EF skills, researchers can improve the ecological validity of EF assessments (McCabe et al., 2000, Sbordone, 2001). Specifically, assessing EF skills in a classroom setting, with its naturally occurring distractors and motivators, will yield a more ecologically valid measure of EF skills. It may also improve the predictive validity of directly assessed EF skills for students' self-regulated classroom behavior and measures of academic achievement such as performance on standardized tests.

As educators and policymakers debate the merits of assessing student progress using measures of socioemotional learning (Campbell et al., 2016, Duckworth and Yeager, 2015, Ursache et al., 2012, West, 2016), researchers need to create valid, pragmatic, and cost-effective ways of assessing EFs at scale. Although teacher report on questionnaire measures of EF has been found consistently to predict children's academic achievement (Allan et al., 2014, McClelland et al., 2006), teacher report has several known limitations. First, teacher report of student behavior can be subject to a “halo” effect (Nisbett & Wilson, 1977), where the respondent's general impression of the child's overall functioning biases the report of specific skills. This may be exacerbated when teachers are required to rapidly evaluate and compare many students. There is also evidence of systematic racial and gender bias in teachers' reports (McKown and Weinstein, 2008, Ready and Wright, 2011). Moreover, when asked to consider students' self-regulation, teachers may find it difficult to differentiate between EFs and related constructs such as conscientiousness (Eisenberg, Duckworth, Spinrad, & Valiente, 2014). Further, questionnaire items tend to capture broad behavioral markers of self-regulation and composites tend to have positively skewed distributions, with many students scoring at or close to the scale maximum. As such, they are less sensitive than direct assessments in reflecting small differences in EF skills across students and incremental changes in EF skills over time. Finally, questionnaires require teachers to contribute considerable time and cognitive effort, which makes it difficult to gather information on all students in a classroom or to track changes throughout an academic year.

Direct assessment of EF skills addresses problems with objectivity and measurement precision (Silver, 2014) and is thus considered to be the “gold standard” of EF measurement. However, extant individual assessment procedures are prohibitively expensive for large-scale studies such as program evaluations. Further, taking children out of the classroom one at a time burdens teachers by reducing instructional time and disrupting students' attention and behavior. Understandably, teachers and district officials often object to this type of research design. In order to employ direct assessments of EF skills at scale, we need to develop a group-based assessment procedure that is pragmatic, cost-effective, and minimally disruptive.

Although researchers have recognized the need to measure EF skills in real-world settings (McCabe et al., 2000, Sbordone, 2001), they almost exclusively employ individual assessment procedures (Fuhs et al., 2013, Prager et al., 2016, Schmitt et al., 2015, Weiland et al., 2014). We were able to identify only one small pilot study (reported in a book chapter) in which EF data were collected in a group context. McCabe, Rebello-Britto, Hernandez, and Brooks-Gunn (2004) tested 44 preschoolers in a group administration procedure, where four familiar peers simultaneously completed modified laboratory-based tasks in a classroom setting with one administrator. The authors coded video recordings of group assessment and reported that children had a harder time controlling impulses during the Gift Wrap task when assessed in a peer group setting than during individual assessment, but otherwise did not compare children's EF performance across the two settings. Computerized tasks that automatically score accuracy and reaction time (RT) create an opportunity to extended this work and evaluate the feasibility of group assessment of EFs in middle childhood.

The main goal of the current study was to evaluate a new group assessment procedure that allows researchers to directly measure EF skills in all students at the same time. Our assessment procedures included a number of methodological innovations to obtain reliable and valid data while simultaneously reducing staff training requirements and the cost of data collection. We adapted developmentally appropriate, widely used EF tasks for administration on tablet computers. These tasks were selected to yield a broad measure of EFs, as represented by inhibitory control, working memory, and cognitive flexibility (see Measures for details). The computer-based tasks provided both accuracy and RT data, thus eliminating the need for video recording or coding of children's responses. Moreover, the portability of tablet devices and children's ease with the touch-screen interface enabled group assessment. Our procedure has the potential to significantly lower the costs and increase the widespread use of high quality direct assessments of EFs.

Our analyses compare the reliability and validity of this novel group assessment procedure with the reliability and validity of an analogous individual assessment procedure that was conducted in a quiet, highly controlled setting. We hypothesized that children's performance in the group assessment setting would show convergent validity with their performance in the more conventional individual assessment setting. Further, we expected that EF skills would show similar associations with known demographic and educational covariates across both assessment settings. Finally, since direct assessment of EFs in the classroom has greater ecological validity, we hypothesized that performance in the group assessment setting would have greater predictive validity for children's self-regulated classroom behaviors and academic achievement compared to performance in the individual assessment setting.