Abstract
Collaborative play, an educational tool for children on the Autism Spectrum Disorder (ASD) spectrum, has been demonstrated as having potential for increasing the engagement of children with ASD. Researchers in China and the USA have assessed three approaches for accessible interface design and learning by students with ASD. With the use of known tools and appropriate occupational therapy interventions, an educational protocol was designed to evaluate the two selected applications and a commercially available application. The pilot studies, including experimental design and outcomes, are presented in this paper in the context of prior ASD intervention research, correlated with child development studies, and provide a solid foundation for comparative usability assessment of mid-air finger gesture interaction as well as hand gesture interaction for the wider population of users. Early results in China are promising, based on experiences in the USA.
Similar content being viewed by others
Notes
For a more thorough review of the clinical and technical impact of Kinect, readers can refer to [56].
It is freely downloadable at: https://apps.leapmotion.com/apps/bongos/windows.
References
American Psychiatric Association. DSM-V: Diagnostic and Statistical Manual of Mental Disorders, 5th Edn. American Psychiatric Publication, Section 299.00 (2013)
Kasari, C., Freeman, S., Paparella, T.: Joint attention and symbolic play in young children with autism: a randomized controlled intervention study. J. Child Psychol. Psychiatry 47(6), 611–620 (2006)
Machalicek, W., Shogren, K., Lang, R., Rispoli, M., O’Reilly, M.F., Franco, J.H., Sigafoos, J.: Increasing play and decreasing the challenging behavior of children with autism during recess with activity schedules and task correspondence training. Res. Autism Spectr. Disord 3(2), 547–555 (2009)
Case-Smith, J., Arbesman, M.: Evidence-based review of interventions for autism used in or of relevance to occupational therapy. Am. J. Occup. Ther. 62, 416–429 (2008)
Azar, N.R., McKeen, P., Carr, K., Sutherland, C.A., Horton, S.: Impact of motor skills training in adults with autism spectrum disorder and an intellectual disability. J. Dev. Disabil. 22(1), 28–38 (2016)
Gal, E., Lamash, L., Bauminger-Zviely, N., Zancanaro, M., Weiss, P.L.: Using multitouch collaboration technology to enhance social interaction of children with high-functioning autism. Phys. Occup. Ther. Pediatr. 36(1), 46–58 (2016)
Biscaldi, M., Rauh, R., Irion, L., Jung, N., Mall, V., Fleischhaker, C., Klein, C.: Deficits in motor abilities and developmental fractionation of imitation performance in high-functioning autism spectrum disorders. Eur. Child Adolesc. Psychiatry 23(7), 599–610 (2014). doi:10.1007/s00787-013-0475-x
Sumner, E., Leonard, H.C., Hill, E.L.: Overlapping phenotypes in autism spectrum disorder and developmental coordination disorder: a cross-syndrome comparison of motor and social skills. J. Autism Dev. Disord. 46(8), 2609–2620 (2016)
Sacrey, L.-A.R., Germani, T., Bryson, S.E., Zwaigenbaum, L.: Reaching and grasping in autism spectrum disorder: a review of recent literature. Front. Neurol. 5(6), 1–12 (2014)
Mostofsky, S.H., Dubey, P., Jerath, V.K., Jansiewicz, E.M., Goldberg, M.C., Denckla, M.B.: Developmental dyspraxia is not limited to imitation in children with autism spectrum disorders. J. Int. Neuropsychol. Soc. 12, 314–326 (2006)
Leary, M.R., Hill, D.A.: Moving on: autism and movement disturbance. Ment. Retard. 34, 39–53 (1996)
Ming, X., Brimacombe, M., Wagner, G.C.: Prevalence of motor impairments in autism spectrum disorders. Brain Dev. 29, 565–570 (2007)
Fournier, K.A., Hass, C.J., Naik, S.K., Lodha, N., Cauraugh, J.H.: Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. J. Autism Dev. Disord. 10, 1227–1240 (2010)
Provost, B., Heimerl, S., Lopez, B.R.: Levels of gross and fine motor development in young children with autism spectrum disorder. Phys. Occup. Ther. Pediatr. 27(3), 21–36 (2007)
Jones, V., Prior, M.: Motor imitation abilities and neurological signs in autistic children. J. Autism Dev. Disord. 15, 37–46 (1985)
Haswell, C.C., Izawa, J., Dowell, L.R., Mostofsky, S.H., Shadmehr, R.: Representations of internal models of action in the autistic brain. Nat. Neurosci. 12, 970–972 (2009)
Molloy, C.A., Dietrich, K.N., Bhattacharya, A.: Postural stability in children with autism spectrum disorder. J. Autism Dev. Disord. 33(6), 643–652 (2003)
Vilensky, J.A., Damasio, A.R., Maurer, R.G.: Gait disturbances in patients with autistic behavior: a preliminary study. Arch. Neurol. 38(10), 646–649 (1981)
Ghaziuddin, M., Butler, E.: Clumsiness in autism and Asperger syndrome: a further report. J. Intell. Disabil. Res. 42, 43–48 (1998)
Morin, B., Reid, G.: A quantitative and qualitative assessment of autistic individuals on selected motor tasks. Adapt. Phys. Act. Q. 2, 43–55 (1985)
Noterdaeme, M., Mildenberger, K., Minow, F., Amorosa, H.: Evaluation of neuromotor deficits in children with autism and children with a specific speech and language disorder. Eur. Child Adolesc. Psychiatry 11(5), 219–225 (2002)
Grandin, T.: My Experiences with Visual Thinking Sensory Problems and Communication Difficulties. Center for the Study of Autism (1996). https://www.google.com/url?sa=t&source=web&rct=j&url=http://docshare01.docshare.tips/files/24278/242781820.pdf&ved=0ahUKEwjAhs_Y5PrUAhWGVz4KHRcBAqQQFghJMAM&usg=AFQjCNGoPz8moPt6PGT3xDzDeqdkYtkdiw
Provost, B., Lopez, B.R., Heimerl, S.: A comparison of motor delays in young children: autism spectrum disorder, developmental delay, and developmental concerns. J. Autism Dev. Disord. 37(2), 321–328 (2007)
Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., Ben Sasson, A., Gal, E., Weiss, P.L.: Collaborative puzzle game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ‘09), pp. 197–204. ACM Press (2009)
Grandin, T.: Visual abilities and sensory differences in a person with autism. Biol. Psychiatry 65(1), 15–16 (2009)
Simmons, D.R., Robertson, A.E., McKay, L.S., Toal, E., McAleer, P., Pollick, F.E.: Vision in autism spectrum disorders. Vis. Res 49(22), 2705–2739 (2009)
Behrmann, M., Thomas, C., Humphreys, K.: Seeing it differently: visual processing in autism. Trends Cogn. Sci. 10(6), 258–264 (2006)
Virnes, M., Kärnä, E., Vellonen, V.: Review of research on children with autism spectrum disorder and the use of technology. J. Spec. Educ. Technol. 30(1), 13–27 (2015)
Farr, W., Yuill, N., Hinske, S.: An augmented toy and social interaction in children with autism. Int. J. Arts Technol. 5(2–4), 104–125 (2012)
Chang, Y.-J., Chen, S.-F., Huang, J.-D.: A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Res. Dev. Disabil. 32(6), 2566–2570 (2011)
Wessels, R., Dijcks, B., Soede, M., Gelderblom, G.J., De Witte, L.: Non-use of provided assistive technology devices: a literature overview. Technol. Disabil. 15, 231–238 (2003)
Webster, D., Celik, O.: Systematic review of Kinect applications in elderly care and stroke rehabilitation. J. Neuro-Eng. Rehabil. 11, 108 (2014)
Palacios-Navarro, G., García-Magariño, I., Ramos-Lorente, P.: A Kinect-based system for lower limb rehabilitation in Parkinson’s disease patients: a pilot study. J. Med. Syst. 39, 103 (2015)
Freitas, D.Q., Da Gama, A.E.F., Figueiredo, L., Chaves, T.M., Marques-Oliveira, D., Teichrieb, V., Araujo, C.: Development and evaluation of a Kinect based motor rehabilitation game. In: Proceedings of SBGames, pp. 144–153. (2012)
Altanis, G., Boloudakis, M., Retalis, S., Nikou, N.: Children with motor impairments play a Kinect learning game: first findings from a pilot case in an authentic classroom environment. J. Interact Des. Archit. 19, 91–104 (2013)
Pompeu, J.E., Arduini, L.A., Botelho, A.R., Fonseca, M.B.F., Pompeu, S.M.A.A., Torriani-Pasin, C., Deutsch, J.E.: Feasibility, safety and outcomes of playing Kinect Adventures!™ for people with Parkinson’s disease: a pilot study. Physiotherapy 100(2), 162–168 (2014)
Lee, J.D., Hsieh, C.H., Lin, T.Y.: A Kinect-based Tai Chi exercises evaluation system for physical rehabilitation. In: Proceedings of the IEEE International Conference on Consumer Electronics (ICCE), pp. 177–178. (2014)
Chang, C.Y., Lange, B., Zhang, M., Koenig, S., Requejo, P., Somboon, N., Sawchuk, A. A., Rizzo, A.A.: Towards pervasive physical rehabilitation using Microsoft Kinect. In: Proceedings of 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health), pp. 159–162. (2012)
Mulroy, S.J., Thompson, L., Kemp, B., Hatchett, P.P., Newsam, C.J., Lupold, D.G., Haubert, L.L., Eberly, V., Ge, T.T., Azen, S.P., Winstein, C.J., Gordon, J.: Strengthening and optimal movements for painful shoulders (STOMPS) in chronic spinal cord injury: a randomized controlled trial. Phys. Ther. 91, 305–324 (2011)
Clark, R., Kraemer, T.: Clinical use of Nintendo Wii bowling simulation to decrease fall risk in an elderly resident of a nursing home: a case report. J. Geriatr. Phys. Ther. 32(4), 174–180 (2009)
Deutsch, J.E., Borbely, M., Filler, J., Huhn, K., Guarrera-Bowlby, P.: Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with cerebral palsy. Phys. Ther. 88(10), 1196–1207 (2008)
Deutsch, J.E., Brettler, A., Smith, C., Welsh, J., John, R., Guarrera-Bowlby, P., MichalKafri, M.: Nintendo Wii sports and Wii Fit game analysis, validation, and application to stroke rehabilitation. Clin. Appl. Technol. Pract. 18(6), 701–709 (2011)
Agarwal, R., Sampath, H.A., Indurkhya, B.: A usability study on natural interaction devices with ASD children. In: Stephanidis, C., Antona, M. (eds.) Universal Access in Human-Computer Interaction User and Context Diversity LNCS Volume 8010, pp. 447–453. Springer, Berlin, Heidelberg (2013)
Saiano, M., Pellegrino, L., Casadio, M., Summa, S., Garbarino, E., Rossi, V., Dall’Agata, D., Sanguineti, V.: Natural interfaces and virtual environments for the acquisition of street crossing and path following skills in adults with Autism Spectrum Disorders: a feasibility study. J. NeuroEng. Rehabil. 12, 17 (2015)
Bai, Z., Blackwell, A.F., Coulouris, G.: Using augmented reality to elicit pretend play for children with autism. IEEE Trans. Vis. Comput. Graph. 21(5), 598–610 (2015)
Bartoli, L., Corradi, C., Garzotto, F., Valoriani, M.: Exploring motion-based touchless games for autistic children’s learning. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC’2013), pp. 102–111. ACM Press (2013)
Bartoli, L., Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M.: Designing and evaluating touchless playful interaction for ASD children. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC’14), pp. 17–26. ACM Press (2014)
Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M: Motion-based touchless interaction for ASD children: a case study. In: Proceedings of AVI, pp. 117–120. ACM Press (2014)
Ringland, K.E., Zalapa, R. Neal, M, Escobedo, L. Tentori, M., Hayes, G.: SensoryPaint: a natural user interface supporting sensory integration in children with neurodevelopmental disorders. İn: CHI ‘14 Extended Abstracts, pp. 1681–1686. ACM Press (2014)
Casas, X., Herrera, G., Coma, I., Fernández, M.: A kinect-based augmented reality system for individuals with autism spectrum disorders. In: Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications (GRAPP/IVAPP ‘12). Pp. 440–446. SciTePress (2012)
Mora-Guiard, J., Crowell, C., Pares, N., Heaton, P.: Lands of fog: helping children with autism in social interaction through a full-body interactive experience. In: Proceedings of the 15th International Conference on Interaction Design and Children (ACM IDC’2016), pp. 262–274. (2016)
Van Zomeren, A.H., Brouwer, W.H.: Clinical Neuropsychology of Attention. Oxford University Press, USA (1994)
Brickenkamp, R., Zillmer, E.: The d2 Test of Attention. Hogrefe & Huber Publication, Göttingen (1998)
Bhattacharya, A., Gelsomini, M., Pérez-Fuster, P., Abowd, G.D., Rozga, A.: Designing motion-based activities to engage students with autism in classroom settings. In: Proceedings of the 14th International Conference on Interaction Design and Children (ACM IDC’2015), pp. 69–78. (2015)
Tung, J.Y., Lulic, T., Gonzalez, D.A., Tran, J., Dickerson, C.R., Roy, E.A.: Evaluation of a portable marker less finger position capture device: accuracy of the Leap Motion controller in healthy adults. Physiol. Meas. 36(5), 1025–1035 (2015)
Hondori, H.M., Khademi, M.: A review on technical and clinical impact of microsoft kinect on physical therapy and rehabilitation. J. Med. Eng. 2014, 846514 (2014)
Ebner, M., Spot, M.: Game-based learning with the leap motion controller. In: Russell, D., Laffey, J.M. (eds.) Handbook of Research on Gaming Trends in P-12 Education. IGI Global, pp 555–565. (2015)
Iosa, M., Morone, G., Fusco, A., Castagnoli, M., Fusco, F.R., Pratesi, L., Paolucci, S.: Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top. Stroke Rehabil. 22(4), 306–316 (2015)
Potter, L.E., Araullo, J., Carter, L.: The Leap Motion controller: a view on sign language. In: Proceedings of the 25th Australian Computer-Human Interaction Conference on Augmentation, Application, Innovation, Collaboration—OzCHI’13, pp. 175–178. (2013)
Gieser, S.N., Boisselle, A., Makedon, F.: Real-time static gesture recognition for upper extremity rehabilitation using the leap motion. In: Lecture Notes in Computer Science, pp. 144–154. (2015)
Tang, T., Winoto, P., Wang, R.: Having fun over a distance: supporting multiplayer online ball passing using multiple sets of Kinect. In: CHI ‘15 Extended Abstracts, pp. 1187–1192. ACM Press, New York (2015)
Zhu, G., Cai, S., Ma, Y., Liu, E.: A series of leap motion-based matching games for enhancing the fine motor skills of children with autism. In: 15th International IEEE Conference on Advanced Learning Technologies (ICALT’2015), IEEE, pp. 430–431. (2015)
Zwaigenbaum, L., Bauman, M.L., Choueiri, R., Fein, D., Kasari, C., Pierce, K., Stone, W.L., Yirmiya, N., Estes, A., Hansen, R.L., McPartland, J.C., Natowicz, M.R., Buie, T., Carter, A., Davis, P.A., Granpeesheh, D., Mailloux, Z., Newschaffer, C., Robins, D., Roley, S.S., Wagner, S., Wetherby, A.: Early identification and interventions for autism spectrum disorder: executive summary. Pediatrics 136(Suppl 1), S1–S9 (2015)
Greis, F., Silva, M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. Comput. Support. Coop. Work 24(2-3), 149–175 (2015)
Goh, W., Shou, W., Tan, J., Lum, G.T.J. Interaction design patterns for multi-touch tabletop collaborative games. In: CHI ‘12 Extended Abstracts on Human Factors in Computing Systems (CHI EA´12), pp. 141–150. New York, NY, USA (2012)
Chelsea La Valle. Chinese Cultural Factors Impacting the Educational Schooling of Children with Autism in China. DePaul Discoveries: Vol. 2: Issue 1, Article 10. Available at: http://via.library.depaul.edu/depaul-disc/vol2/iss1/10 (2013)
Wang, S.: In China, the making of an App for autism. Wall Street J. May 19, 2015. http://blogs.wsj.com/chinarealtime/2015/05/19/in-china-the-making-of-an-app-for-autism/ (2015)
Marco, E., Barett, L., Hinkley, N., Hill, S., Nagarajan, S.S.: Sensory processing in autism: a review of neurophysiologic findings. Pediatr. Res. 69(5 Pt 2), 48R–54R (2011)
Baron-Cohen, S., Golan, O., Ashwin, E.: Can emotion recognition be taught to children with autism spectrum conditions? Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci 364(1535), 3567–3574 (2009). doi:10.1098/rstb.2009.0191;10.1098/rstb.2009.0191
Gordon, I., Pierce, M., Bartlett, M., Tanaka, J.: Training facial expression production in children on the autism spectrum. J. Autism Dev. Disord. 44(10), 2486–2498 (2014). doi:10.1007/s10803-014-2118-6
Ploog, B., Scharf, A., Nelson, D., Brooks, P.: Use of computer-assisted technologies (CAT) to enhance social, communicative, and language development in children with autism spectrum disorders. J. Autism Dev. Disord. 43(2), 301–322 (2013). doi:10.1007/s10803-012-1571-3
Rice, L., Wall, C., Fogel, A., Shic, F.: Computer-assisted face processing instruction improves emotion recognition, mentalizing, and social skills in students with ASD. J. Autism Dev. Disord. 45(7), 2176–2186 (2015). doi:10.1007/s10803-015-2380-2
Tang, T.Y., Flatla, D.: Autism awareness and technology-based intervention research in China: the good, the bad, and the challenging. In: Proceedings of Workshop on Autism and Technology—Beyond Assistance and Intervention, in Conjunction with the 34th ACM International Conference on Human Factors in Computing Systems (CHI’2016) (2016)
Goldsmith, T.R., LeBlanc, L.A.: Use of technology in interventions for children with autism. J. Early Intensive Behav. Interv. 1(2), 166 (2004)
Hayes, G.R., Hirano, S., Marcu, G., Monibi, M., Nguyen, D.H., Yeganyan, M.: Interactive visual supports for children with autism. Pers. Ubiquitous Comput. 14(7), 663–680 (2010)
Silver, M., Oakes, P.: Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 5(3), 299–316 (2001)
Clampton, N.: China moves to tackle autism with first study of prevalence. South China Morning Post. (2013)
Acknowledgements
The authors would like to acknowledge Kean University’s financial support to Tiffany Tang under Wenzhou-Kean University’s Student Partnering with Faculty (SpF) Research Program. The authors are also grateful to children and their families at Wenzhou XingLe Children’s Educational Development Center and Wenzhou Orange Wheat Children's Educational center for participating in the experiments; thanks also go to Tina Xiaoting Fu and Esther Mingyue Tang for developing the two games and their efforts during both the experiment and interviews as well as Leila Zeqiang Huang, Relic Yongfu Wang and Pinata Winoto for their assistance in the experiments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tang, T.Y., Falzarano, M. & Morreale, P.A. Assessment of the utility of gesture-based applications for the engagement of Chinese children with autism. Univ Access Inf Soc 17, 275–290 (2018). https://doi.org/10.1007/s10209-017-0562-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10209-017-0562-8