Elsevier

Computers & Education

Volume 78, September 2014, Pages 321-332
Computers & Education

The acceptability and efficacy of an intelligent social tutoring system

https://doi.org/10.1016/j.compedu.2014.06.013Get rights and content

Highlights

  • We assessed the efficacy of an innovative intelligent game-based social tutorial.

  • The software was easy to use and well-liked among this audience.

  • Program use was associated with lower psychosocial distress.

  • Program use was associated with behavioral and emotional strength.

  • The advantages of using this approach to teach social skills are discussed.

Abstract

This study tested the acceptability and efficacy of an innovative intelligent tutoring system (ITS), Adventures Aboard the S.S. GRIN, that translates the evidence-based in-person Social Skills Group Intervention (SSGRIN) into an interactive game-based social tutorial. This randomized controlled pilot trial tested the first half of the social tutorial software for children with social skills challenges. Participating children in grades 3–5 were randomly assigned to immediate treatment (n = 19) or wait-list control (n = 17). User ratings indicated the software was easy to use and well-liked for this audience. The program was also associated with observable changes in social skills and behavior; children who interacted with Adventures Aboard the S.S. GRIN exhibited lower psychosocial distress and higher behavioral and emotional strength at post-test compared to children who did not. This pilot study offers important insights about the feasibility and potential effectiveness of online social skills training and lays the groundwork for future full-scale effectiveness testing. The advantages of using this state-of-the-art approach and its implications for improving social, emotional, and academic outcomes are discussed.

Introduction

There is growing recognition that social skills are critical for success both in school and in life. Strong support for social skills education is shared by the public, educators, and policymakers alike (Camilli et al., 2010, Meier et al., 2006, National Association for the Education of Young Children, 2009, National Research Council, 2012). Gallup Poll results indicate that a majority of people think these skills should be taught to children (Bushaw & Lopez, 2013), and this opinion is increasingly being reflected in state and federal policies (Dusenbury et al., 2011, HR 1875, 2013). Parents, educators, mental health professionals, and others invested in the development of children and youth are seeking tools to promote positive social development.

There is good reason for this increasing emphasis on social competence. Children who struggle with social skills are at heightened risk for a wide array of externalizing and internalizing problems and serious negative outcomes, including aggression, anxiety, bullying, depression, and substance abuse, as well as academic failure and school dropout (DeRosier et al., 1994, Dodge and Crick, 1990, French and Conrad, 2001, Hawkins et al., 1992, Kupersmidt and DeRosier, 2004, Najaka et al., 2001, Olweus, 1993, Parker et al., 2006, Solberg et al., 2007, Spooner, 1999, Weiss and Hechtman, 1993). Unaddressed problems with peers tend to worsen over time, and as these problems become more chronic and severe the likelihood of serious negative consequences in adolescence significantly increases (DeRosier et al., 1994, Kupersmidt and DeRosier, 2004, Parker et al., 2006).

Fortunately, social skills training (SST) can improve outcomes for children who struggle. Scores of studies support the efficacy of these interventions for improving children's social skills and academic functioning, and for reducing aggression, mental health problems, and substance use (e.g., Beidel et al., 2005, DeRosier, 2004, Durlak et al., 2011, Foster and Bussman, 2007, Hawker and Boulton, 2000, Joseph and Strain, 2003, Losel and Beelmann, 2003, Zins et al., 2004). Effective universal or targeted interventions can help children learn and practice the social skills that increase positive and/or inhibit negative social interactions (Asher and Renshaw, 1981, Coie et al., 1990, Dodge and Feldman, 1990, Merrell and Gimpel, 1997). For example, SSTs directly teach cooperative skills that promote companionship and decrease domineering or impulsive behaviors. Similarly, SSTs enhance empathetic behaviors that promote communication and the ability to take another's perspective.

Unfortunately, the reach of traditional in-person SST is limited (Mueser & Bellack, 2007). Logistical barriers, such as the need for trained professionals to administer the program, financial costs to families, and time and travel requirements for children to participate, mean that many children who could benefit from SST never receive intervention. The families of children in need of social intervention may lack the means to access such treatment, or be unable to commit the time or resources required for in-person intervention. Even when individuals can afford and access traditional SST, in-person treatment delivery often does not provide the repeated opportunities for skill practice and concept exposure shown to result in greater intervention benefits (Foster and Bussman, 2007, Greenberg et al., 2001), including extending learning and practice opportunities beyond the group setting (Greenberg et al., 2001). The opportunity to not only address these barriers but also offer an unprecedented level of individualized, adaptive intervention is now available, as advances in emergent technologies are making it possible to translate evidence-based SST programs into broadly accessible engaging game-like platforms.

An Intelligent Tutoring System (ITS) is a computer-based learning system that attempts to adapt to the needs of learners (Self, 1999). ITSs foster and assess learning through adaptive interaction between the student and the system, and the instruction contains both domain-specific pedagogical knowledge and knowledge of the learner (Bordeau and Grandbastien, 2010, Collins and O'Brien, 2011). As the learner interacts with the software, the software responds by selecting the most appropriate content and pedagogical style for that learner; in this way, the software reacts in much the same way a human tutor would. The active engagement through interactivity between a child and the software can enhance learning and memory beyond more passive instructional methods (e.g., Corbett et al., 2001, Graesser et al., 2008).

ITSs offer an innovative way to advance SST, address barriers to effective implementation, and reach children on a much broader scale than traditional methodologies. In effect, ITSs for SST have the potential to lower cost, eliminate the need for travel, accommodate any schedule, and eliminate inconsistencies in treatment delivery across different providers. Beyond these practical benefits, ITSs can improve upon traditional SST models by providing repeated and unlimited practice opportunities within an intelligent, interactive learning environment that can individualize instruction at a level not typically possible with in-person SST. Specifically, ITS software has the capacity to adjust to a child's individual needs by varying difficulty based on performance and providing immediate personalized feedback and pedagogical assistance to guide the child towards target learning goals (Hutcherson et al., 2004, Mazzotti et al., 2009).

An additional advantage of interactive game-like ITSs is that they are inherently engaging and motivating for today's children, for whom technology has become a ubiquitous part of life (Fitzgerald, 2005, NCREL & Metiri Group, 2003). Learning through virtual or simulated environments can be a great motivator in the educational process and a valuable part of learning and development (Joyce et al., 2009, Ke, 2009). An interactive digital environment lets a child think, understand, prepare, execute actions, and adjust strategy as needed (Gee, 2003) by employing simulated situations analogous to those they experience in real life. Employing ITSs for SST offers the opportunity to improve technical capability and clinical practice in SST. It also provides a safe practice environment; practicing fledgling social skills with peers can be risky, but practicing in the virtual environment gives children a safe place to fail and then try again.

Although the use of ITSs for academic skills such as math and science is becoming more commonplace, an ITS for SST is highly innovative, and as a result the research base is just being established. Early results indicate that interactive software may be an effective tool for remediating social skills for children. To date, most of the available research (e.g., Golan and Baron-Cohen, 2006, Sansoti and Powell-Smith, 2008) has focused on children with autism spectrum disorders, who experience marked social skill deficits, and most of the existing programs are simplistic in design and implementation with little to no dynamic adjustment in difficulty or personalized feedback. One recent system designed to promote social skills and prevent bullying called Quest for the Golden Rule (Rubin-Vaughan, Pepler, Brown, & Craig, 2011) includes more complex social problem solving situations, and preliminary results look promising. However, because that research did not include a control group, it's not possible to draw conclusions about the effectiveness of Quest for the Golden Rule.

The primary goal of the interactive program is to enable broad delivery of an evidence-based SST and provide children with a safe and private learning environment in which to build and practice social skills through interactive social problem solving (SPS) scenes. The specific skills included were based on Social Skills Group Intervention (SSGRIN), an evidence-based in-person SST program. Developed at 3C Institute and currently used with thousands of children across the United States and abroad, SSGRIN has been repeatedly tested resulting in a strong empirical base for its effectiveness (e.g., DeRosier, 2004, DeRosier and Gilliom, 2007, DeRosier and Marcus, 2005).

SSGRIN is a 10 week in-person small group intervention for children with social skills deficits, led by a trained professional (e.g., psychologist or school counselor). SSGRIN is designed to be broadly applicable to children experiencing a wide range of social difficulties, including immature social skills relative to same-aged peers; social isolation with few close friends; rejection, teasing, or bullying by peers; social anxiety or awkwardness with peers; aggressive or bullying behavior with peers; and impulsive, disruptive behaviors. The program addresses the five social and emotional learning (SEL) competency clusters identified by CASEL (2013), which include self-awareness, self-management, social awareness, relationship skills, and responsible decision making. It focuses on increasing children's capacity to evaluate social situations and respond in an adaptive, prosocial manner to achieve social goals. Children who participate in SSGRIN show increased social skills knowledge, social problem solving, social acceptance, self-esteem, and emotional functioning, as well as lower depression, anxiety, and aggressive behavior (DeRosier, 2004, DeRosier, 2007, DeRosier and Marcus, 2005).

To develop the ITS Adventures Aboard the S.S. GRIN, we translated the SSGRIN curriculum into a game-like instructional program by designing in-game tasks that required children to apply specific social skills in order to solve each social problem encountered. The translation was done by a Developmental Psychologist (Ph.D.) with expertise in children's emotional development, and reviewed and approved by the developer of the original curriculum, Dr. Melissa DeRosier. Developed for children in grades 3–5 (aged 8–12 years) who struggle with social skills, the game-like program was designed to actively engage children in social problem solving tasks through interactive software technology, and reflects best practices for children's educational interactive software (e.g., Garris et al., 2002, Mathan and Koedinger, 2002, Roussou, 2005).

To develop the interactive game Adventures Aboard the S.S. GRIN, we designed the SPS scenes upon the foundations of general problem-solving models and cognitive-behavioral therapy (e.g., Bloomquist, 2005), as well as decades of theory and research in children's social development (e.g., Bandura, 1977, Crick and Dodge, 1994, Parker et al., 2006). We incorporated research-supported SST strategies such as increasing social knowledge, practice with personalized feedback, and positive reinforcement (Elliott & Gresham, 1993) and incorporating behavioral (i.e., what to do or not do in a social situation), cognitive (i.e., assumptions, evaluating consequences, goal setting), and emotional (i.e., self-awareness, emotion regulation) social skills into instruction (Asher et al., 1996, Graczyk et al., 2000). This integrative approach helps children think through a given social situation, consider alternative social problem solving approaches, and evaluate the success of a selected approach for achieving a social goal. Interventions that use these strategies to build children's capacity to actively engage in social problem solving have the most significant and lasting impact on the quality of their social relationships (Greenberg et al., 2004, Kam et al., 2004, Suveg et al., 2007). Examples for specific social skills addressed in each episode and examples of their in-game interactions are shown in Table 1.

An interdisciplinary team of psychologists, computer programmers, and artists collaborated to create an engaging learning environment to serve as the context for social problem solving. We decided to develop Adventures Aboard the S.S. GRIN as a single player game rather than a multiplayer game for a number of reasons. A single player game enabled individualized feedback and play paths based on player choices, allowed for control over the reactions of other characters in a game, created a safe environment in which to practice fledgling skills without social ramifications, and avoided the possible iatrogenic effects of participants reinforcing negative behaviors in other children.

The point-and-click problem solving tutorial provides a game-like feel due to the story arc, characters, and computer interaction. A compelling storyline and engaging challenges have been found to be critically important for engaging children in instructional software (Mott and Lester, 2006a, Mott and Lester, 2006b, Roussou, 2005), and this consideration was significant in the development of the program. The story world is based on a nautical adventure theme in which the player is a new recruit on the sailing ship S.S. GRIN and joins an established crew, each of whom specializes in a specific social skill (e.g., impulse control). The narrative purpose of the crew members is to be the child's companions, but the tutoring system also employs them as pedagogical agents to provide feedback and assistance as needed. When the child first uses the software, he/she creates a customized avatar to represent himself or herself in the game. Using this avatar, the child then navigates the virtual environment, interacting directly with non-player characters (NPCs) and learning to appropriately manage social challenges. Specifically, the child's avatar must join the crew and travel around an island engaging in social problem solving in order to address plot conflicts and save friendship on the island.

As children navigate the program, the software captures in-game behaviors. Some of these behaviors are dialog-driven menu choices, whereas others are game-based behavior (e.g., if a child is asked to help with a task, does the child comply or choose another action, such as walking away?). The difficulty level of the social tasks increases both within and across the nine episodes of the game. For example, an early episode addresses relatively simple choices that show self-respect and respect for others, whereas a later episode addresses the skills needed to initiate positive social interactions, including the complex interaction of respect, observation, communication, thinking out options, and considering consequences. Within the “initiation” episode participants move from a relatively easy scenario (initiating with one non-intimidating other) to a more difficult one (initiating with two characters who are already engaged with each other) to an even more difficult scenario (initiating with a group of friends who appear intimidating and reject initial attempts).

As an ITS, it was essential that the software track players' actions, tailor hints and feedback along the way, collect information on performance, make inferences about players' strengths and weaknesses, adapt to the needs of the learners, and contain the following components: knowledge of the domain, knowledge of the learner, knowledge of appropriate pedagogy, and interactive environment interface (Samuelis, 2007). Central to the development of the program, we ensured that the program adapted to the needs of the players by monitoring progress in real time and adjusting gameplay accordingly. The software monitored individual progress and responded with appropriate content, feedback, and pedagogical assistance based on player actions. We incorporated proven training strategies of scaffolding, coaching and reinforcement, and practice to create an effective social learning program for children. For example, in one task the player needs to complete a mission that initially appears possible to complete alone, but reveals itself to be a two-person job. Without explicit instruction, the player needs to seek the cooperation of an NPC. If the player does not perform this action in a given amount of time, an NPC walks through the scene as a hint that help is available. If the player still does not seek assistance after a given amount of time, the NPC reminds him or her that help is available. Finally, if the player is still attempting the impossible task, the NPC reappears to help and provides information about cooperation and help-seeking. At appropriate times, individualized feedback is provided in response to the child's performance, and when gameplay indicates that a skill has not been satisfactorily demonstrated the child is provided structured guidance, and in some cases is asked to continue to practice the skill in-game until he or she demonstrates an understanding of a good social choice. Achievements are provided for outstanding performance, and players are able to replay episodes to earn all achievements. All text-based dialog is accompanied by audio voiceover to account for differences in reading ability and to convey important social information, such as emphasis and tone of voice.

In order to test the potential promise of this innovative computerized method for SST, we examined the impact of participation in the first half of the software through this study. In particular, we were interested in examining how children and families use the program outside of a clinical environment in order to determine its feasibility, usability, and potential efficacy as a broad-scale SST tool. The tested portion of the ITS included four instructional episodes, each of which focused on a specific social skill (i.e., respect, social planning, impulse control, and communication). Children completed 38 rooms (i.e., discrete visual spaces), interacted with 42 unique characters, and completed 14 social problem-solving scenarios for this study, totaling approximately 90 min of gameplay. Sample screenshots from Adventures Aboard the S.S. GRIN are shown in Fig. 1 (further examples available at http://www.3ctechnologies.com/games).

Our research questions were:

  • 1.

    Will children with social skills deficits find the ITS likeable and usable?

  • 2.

    Will interaction with the ITS be associated with reduced pychosocial distress and greater behavioral and emotional strength?

We hypothesized that both research questions would be answered in the affirmative.

Section snippets

Participants

Participants were recruited in the spring of 2013 via postings on local parenting listservs, social media sites, and through contacts within area school systems and child service providers. Interested parents completed an online eligibility survey which included demographic information and the Behavior Assessment System for Children, Second Edition (BASC-2; Reynolds & Kamphaus, 2004), a behavior rating scale. To be eligible to participate, participants must have (1) been attending 3rd, 4th, or

Results

Our first research question was whether children with social skills deficits would find Adventures Aboard the S.S. GRIN likeable and useable. If children were unable or unwilling to independently and effectively use the program, its value as a social skills tutorial would be greatly reduced. We hypothesized that usability and likability would both be rated highly by participants, and these hypotheses were supported. Results are shown in Table 3 and described below.

Discussion

The goal of the current study was to investigate the viability of leveraging gaming technology to advance mental health service delivery by reaching beyond current in-person SST delivery models. We tested whether children with social skills challenges in grades 3–5 would like and be able to independently use the program Adventures Aboard the S.S. GRIN, and whether use of the program was associated with positive changes in psychosocial distress and behavioral and emotional strength. The

Conclusion

Developing evidence-based ITSs for SSTs involves the integration of developmental theory, research, computer science, gaming theory, and practical application to improve social programs and interventions for children. The results of this study provide evidence that ITSs can be an effective and welcomed strategy for addressing psychosocial and behavioral issues in the social domain. This methodology has the potential to overcome many of the barriers associated with traditionally-delivered SSTs,

Acknowledgments

This research was supported by a grant from the Institute for Education Science (Grant #R305A110583) to the Center for Research in Emotional and Social Health (CRESH). We gratefully acknowledge their financial support. We also thank our development team for their contributions to the program, particularly Chris Hehman, Jeremiah Weatherley, Matt Habel, Chrystin Garland, Jon Fabris, Charles Bevan, Marcelo Nader, Kim Smith, and Wes Sommer.

References (80)

  • L.A. Annetta et al.

    Investigating the impact of video games on high school students' engagement and learning about genetics

    Computers & Education

    (2009)
  • A. Rubin-Vaughan et al.

    Quest for the golden rule: an effective social skills promotion and bullying prevention program

    Computers & Education

    (2011)
  • S.R. Asher et al.

    Distinguishing friendship from acceptance: implications for intervention and assessment

  • S.R. Asher et al.

    Children without friends: social knowledge and social-skill training

    The Development of Children's Friendships

    (1981)
  • A. Bandura

    Self-efficacy: toward a unifying theory of behavioral change

    Psychological Review

    (1977)
  • D. Beidel et al.

    Social effectiveness therapy for children: three-year follow-up

    Journal of Consulting and Clinical Psychology

    (2005)
  • M.L. Bloomquist

    Skills training for children with behavior disorders, revised edition: A parent and therapist guidebook

    (2005)
  • J. Bordeau et al.

    Modeling tutoring knowledge

  • G.M. Burlingame et al.

    Youth Outcome Questionnaire

    (2005)
  • W.J. Bushaw et al.

    Which way do we go? The 45th annual PDK/Gallup Poll of the public's attitudes toward the public schools

    (2013)
  • G. Camilli et al.

    Meta-analysis of the effects of early education interventions on cognitive and social development

    Teachers College Record

    (2010)
  • V. Chandra et al.

    The methodological nettle: ICT and student achievement

    British Journal of Educational Technology

    (2008)
  • J. Cohen

    A power primer

    Psychological Bulletin

    (1992)
  • B.H. Cohen

    Explaining psychological statistics

    (2001)
  • J.D. Coie et al.

    2 Peer group behavior and social status

    Peer Rejection in Childhood

    (1990)
  • J.W. Collins et al.

    The Greenwood Dictionary of Education

    (2011)
  • A. Corbett et al.

    Cognitive tutors: from the research classroom to all classrooms

  • N. Crick et al.

    A review and reformulation of social information-processing mechanisms in children's social adjustment

    Psychological Bulletin

    (1994)
  • M.E. DeRosier

    Building relationships and combating bullying: effectiveness of a school-based social skills group intervention

    Journal of Clinical Child and Adolescent Psychology

    (2004)
  • M.E. DeRosier

    Peer-rejected and bullied children: a Safe Schools Initiative for elementary school students

  • M.E. DeRosier et al.

    Effectiveness of a parent training program for improving children's social behavior

    Journal of Child and Family Studies

    (2007)
  • M.E. DeRosier et al.

    Children's academic and behavioral adjustment as a function of the chronicity and proximity of peer rejection

    Child Development

    (1994)
  • M.E. DeRosier et al.

    Building friendships and combating bullying: effectiveness of S. S. GRIN at one-year follow-up

    Journal of Clinical Child and Adolescent Psychology

    (2005)
  • K.A. Dodge et al.

    Social information-processing bases of aggressive behavior in children

    Personality and Social Psychology Bulletin

    (1990)
  • K.A. Dodge et al.

    Issues in social cognition and sociometric status

    Peer Rejection in Childhood

    (1990)
  • J.A. Durlak et al.

    The impact of enhancing students' social and emotional learning: a meta-analysis of school-based universal interventions

    Child Development

    (2011)
  • L. Dusenbury et al.

    State learning standards to advance social and emotional learning

    (2011)
  • S. Elliott et al.

    Social skills interventions for children

    Behavior Modification

    (1993)
  • M.H. Epstein

    Behavioral and Emotional Rating Scale

    (2004)
  • A. Field

    Discovering statistics using SPSS

    (2005)
  • G.E. Fitzgerald

    Using technologies to meet the unique needs of students with emotional/behavioral disorders: findings and directions

  • S.L. Foster et al.

    Evidence-based approaches to social skills training with children and adolescents

  • D.C. French et al.

    School dropout as predicted by peer rejection and antisocial behavior

    Journal of Research on Adolescence

    (2001)
  • R. Garris et al.

    Games, motivation, and learning: a research and practice model

    Simulation & Gaming

    (2002)
  • J.P. Gee

    What video games have to teach us about learning and literacy?

    ACM Computers in Entertainment

    (2003)
  • O. Golan et al.

    Systemizing empathy: teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia

    Development and Psychopathology

    (2006)
  • R. Goldsworthy et al.

    The STAR Project: enhancing adolescents' social understanding through video-based, multimedia scenarios

    Journal of Special Education Technology

    (2000)
  • P. Graczyk et al.

    Criteria for evaluating the quality of school-based social and emotional learning programs

  • A. Graesser et al.

    Agent technologies designed to facilitate interactive knowledge construction

    Discourse Processes

    (2008)
  • M.T. Greenberg et al.

    The prevention of mental disorders in school-aged children: current state of the field

    Prevention & Treatment

    (2001)
  • Cited by (24)

    • Co-Op World: Adaptive computer game for supporting child psychotherapy

      2020, Computers in Human Behavior Reports
      Citation Excerpt :

      Social skills are the behaviors “that result in positive social interactions and encompass both verbal and nonverbal behaviors necessary for effective interpersonal communication” [29] (pp. 258). Children who struggle with social skills are at heightened risk for a wide array of serious negative outcomes, including aggression, anxiety, bullying, depression and substance abuse, as well as academic failure and school dropout [30]. Estimates indicate that prevalence rates of child social, emotional and behavioral difficulties are relatively consistent across countries and time, with clinical levels of difficulties reported to range from 10 to 20% [31].

    • Prosocial digital games for youth: A systematic review of interventions

      2020, Computers in Human Behavior Reports
      Citation Excerpt :

      Thus, future prosocial game evaluations should strive to include behaviour outcome measures to yield stronger and more precise results. Additionally, it is important to highlight studies that transformed existing Social Skills Training school-based programmes (SSTs) into digital games, as they yielded prevalent positive results in the sample (Craig et al., 2016; DeRosier & Thomas, 2019; Sanchez et al., 2014, 2017). The literature supports the benefits of SST programmes as a universal, integrated intervention approach for children to participate in social-emotional learning (SEL) (Durlak et al., 2011).

    • Generic versus disorder specific cognitive behavior therapy for social anxiety disorder in youth: A randomized controlled trial using internet delivery

      2017, Behaviour Research and Therapy
      Citation Excerpt :

      There is also a strong case for use of the internet to deliver social skills training as it can be used to present i) information about the nature and importance skills in an interactive manner, ii) videotaped illustrations specific skill usage by other young people, iii) instructions for practice in real world contexts, and iv) monitoring charts for homework completion. Although not specifically evaluated with children with SAD, computer or internet delivery of social skills training has been shown to produce significant improvements in social skills with children in regular classrooms and youth with autism spectrum disorders (Beaumont, Rotolone, & Sofronoff, 2015; Craig, Brown, Upright, & DeRosier, 2016; Sanchez, Bartel, Brown, & DeRosier, 2014; Tan, Mazzucchelli, & Beaumont, 2015). Specifically, we proposed that in comparison to generic CBT, treatment outcomes would be greater for an CBT intervention for youth anxiety that included social skills training, cognitive therapy components that focus specifically upon reducing self-focused attention, and greater focus on reducing maladaptive pre- and post-event processing of social tasks.

    View all citing articles on Scopus
    View full text