Skip to main content
SpringerLink
Log in

Assessment of the utility of gesture-based applications for the engagement of Chinese children with autism

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Collaborative play, an educational tool for children on the Autism Spectrum Disorder (ASD) spectrum, has been demonstrated as having potential for increasing the engagement of children with ASD. Researchers in China and the USA have assessed three approaches for accessible interface design and learning by students with ASD. With the use of known tools and appropriate occupational therapy interventions, an educational protocol was designed to evaluate the two selected applications and a commercially available application. The pilot studies, including experimental design and outcomes, are presented in this paper in the context of prior ASD intervention research, correlated with child development studies, and provide a solid foundation for comparative usability assessment of mid-air finger gesture interaction as well as hand gesture interaction for the wider population of users. Early results in China are promising, based on experiences in the USA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. For a more thorough review of the clinical and technical impact of Kinect, readers can refer to [56].

  2. https://www.cdc.gov/ncbddd/autism/data.html.

  3. It is freely downloadable at: https://apps.leapmotion.com/apps/bongos/windows.

References

  1. American Psychiatric Association. DSM-V: Diagnostic and Statistical Manual of Mental Disorders, 5th Edn. American Psychiatric Publication, Section 299.00 (2013)

  2. Kasari, C., Freeman, S., Paparella, T.: Joint attention and symbolic play in young children with autism: a randomized controlled intervention study. J. Child Psychol. Psychiatry 47(6), 611–620 (2006)

    Article  Google Scholar 

  3. Machalicek, W., Shogren, K., Lang, R., Rispoli, M., O’Reilly, M.F., Franco, J.H., Sigafoos, J.: Increasing play and decreasing the challenging behavior of children with autism during recess with activity schedules and task correspondence training. Res. Autism Spectr. Disord 3(2), 547–555 (2009)

    Article  Google Scholar 

  4. Case-Smith, J., Arbesman, M.: Evidence-based review of interventions for autism used in or of relevance to occupational therapy. Am. J. Occup. Ther. 62, 416–429 (2008)

    Article  Google Scholar 

  5. Azar, N.R., McKeen, P., Carr, K., Sutherland, C.A., Horton, S.: Impact of motor skills training in adults with autism spectrum disorder and an intellectual disability. J. Dev. Disabil. 22(1), 28–38 (2016)

    Google Scholar 

  6. Gal, E., Lamash, L., Bauminger-Zviely, N., Zancanaro, M., Weiss, P.L.: Using multitouch collaboration technology to enhance social interaction of children with high-functioning autism. Phys. Occup. Ther. Pediatr. 36(1), 46–58 (2016)

    Article  Google Scholar 

  7. Biscaldi, M., Rauh, R., Irion, L., Jung, N., Mall, V., Fleischhaker, C., Klein, C.: Deficits in motor abilities and developmental fractionation of imitation performance in high-functioning autism spectrum disorders. Eur. Child Adolesc. Psychiatry 23(7), 599–610 (2014). doi:10.1007/s00787-013-0475-x

    Article  Google Scholar 

  8. Sumner, E., Leonard, H.C., Hill, E.L.: Overlapping phenotypes in autism spectrum disorder and developmental coordination disorder: a cross-syndrome comparison of motor and social skills. J. Autism Dev. Disord. 46(8), 2609–2620 (2016)

    Article  Google Scholar 

  9. Sacrey, L.-A.R., Germani, T., Bryson, S.E., Zwaigenbaum, L.: Reaching and grasping in autism spectrum disorder: a review of recent literature. Front. Neurol. 5(6), 1–12 (2014)

    Google Scholar 

  10. Mostofsky, S.H., Dubey, P., Jerath, V.K., Jansiewicz, E.M., Goldberg, M.C., Denckla, M.B.: Developmental dyspraxia is not limited to imitation in children with autism spectrum disorders. J. Int. Neuropsychol. Soc. 12, 314–326 (2006)

    Article  Google Scholar 

  11. Leary, M.R., Hill, D.A.: Moving on: autism and movement disturbance. Ment. Retard. 34, 39–53 (1996)

    Google Scholar 

  12. Ming, X., Brimacombe, M., Wagner, G.C.: Prevalence of motor impairments in autism spectrum disorders. Brain Dev. 29, 565–570 (2007)

    Article  Google Scholar 

  13. Fournier, K.A., Hass, C.J., Naik, S.K., Lodha, N., Cauraugh, J.H.: Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. J. Autism Dev. Disord. 10, 1227–1240 (2010)

    Article  Google Scholar 

  14. Provost, B., Heimerl, S., Lopez, B.R.: Levels of gross and fine motor development in young children with autism spectrum disorder. Phys. Occup. Ther. Pediatr. 27(3), 21–36 (2007)

    Article  Google Scholar 

  15. Jones, V., Prior, M.: Motor imitation abilities and neurological signs in autistic children. J. Autism Dev. Disord. 15, 37–46 (1985)

    Article  Google Scholar 

  16. Haswell, C.C., Izawa, J., Dowell, L.R., Mostofsky, S.H., Shadmehr, R.: Representations of internal models of action in the autistic brain. Nat. Neurosci. 12, 970–972 (2009)

    Article  Google Scholar 

  17. Molloy, C.A., Dietrich, K.N., Bhattacharya, A.: Postural stability in children with autism spectrum disorder. J. Autism Dev. Disord. 33(6), 643–652 (2003)

    Article  Google Scholar 

  18. Vilensky, J.A., Damasio, A.R., Maurer, R.G.: Gait disturbances in patients with autistic behavior: a preliminary study. Arch. Neurol. 38(10), 646–649 (1981)

    Article  Google Scholar 

  19. Ghaziuddin, M., Butler, E.: Clumsiness in autism and Asperger syndrome: a further report. J. Intell. Disabil. Res. 42, 43–48 (1998)

    Article  Google Scholar 

  20. Morin, B., Reid, G.: A quantitative and qualitative assessment of autistic individuals on selected motor tasks. Adapt. Phys. Act. Q. 2, 43–55 (1985)

    Article  Google Scholar 

  21. Noterdaeme, M., Mildenberger, K., Minow, F., Amorosa, H.: Evaluation of neuromotor deficits in children with autism and children with a specific speech and language disorder. Eur. Child Adolesc. Psychiatry 11(5), 219–225 (2002)

    Article  Google Scholar 

  22. Grandin, T.: My Experiences with Visual Thinking Sensory Problems and Communication Difficulties. Center for the Study of Autism (1996). https://www.google.com/url?sa=t&source=web&rct=j&url=http://docshare01.docshare.tips/files/24278/242781820.pdf&ved=0ahUKEwjAhs_Y5PrUAhWGVz4KHRcBAqQQFghJMAM&usg=AFQjCNGoPz8moPt6PGT3xDzDeqdkYtkdiw

  23. Provost, B., Lopez, B.R., Heimerl, S.: A comparison of motor delays in young children: autism spectrum disorder, developmental delay, and developmental concerns. J. Autism Dev. Disord. 37(2), 321–328 (2007)

    Article  Google Scholar 

  24. Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., Ben Sasson, A., Gal, E., Weiss, P.L.: Collaborative puzzle game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ‘09), pp. 197–204. ACM Press (2009)

  25. Grandin, T.: Visual abilities and sensory differences in a person with autism. Biol. Psychiatry 65(1), 15–16 (2009)

    Article  Google Scholar 

  26. Simmons, D.R., Robertson, A.E., McKay, L.S., Toal, E., McAleer, P., Pollick, F.E.: Vision in autism spectrum disorders. Vis. Res 49(22), 2705–2739 (2009)

    Article  Google Scholar 

  27. Behrmann, M., Thomas, C., Humphreys, K.: Seeing it differently: visual processing in autism. Trends Cogn. Sci. 10(6), 258–264 (2006)

    Article  Google Scholar 

  28. Virnes, M., Kärnä, E., Vellonen, V.: Review of research on children with autism spectrum disorder and the use of technology. J. Spec. Educ. Technol. 30(1), 13–27 (2015)

    Article  Google Scholar 

  29. Farr, W., Yuill, N., Hinske, S.: An augmented toy and social interaction in children with autism. Int. J. Arts Technol. 5(2–4), 104–125 (2012)

    Article  Google Scholar 

  30. Chang, Y.-J., Chen, S.-F., Huang, J.-D.: A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Res. Dev. Disabil. 32(6), 2566–2570 (2011)

    Article  Google Scholar 

  31. Wessels, R., Dijcks, B., Soede, M., Gelderblom, G.J., De Witte, L.: Non-use of provided assistive technology devices: a literature overview. Technol. Disabil. 15, 231–238 (2003)

    Google Scholar 

  32. Webster, D., Celik, O.: Systematic review of Kinect applications in elderly care and stroke rehabilitation. J. Neuro-Eng. Rehabil. 11, 108 (2014)

    Article  Google Scholar 

  33. Palacios-Navarro, G., García-Magariño, I., Ramos-Lorente, P.: A Kinect-based system for lower limb rehabilitation in Parkinson’s disease patients: a pilot study. J. Med. Syst. 39, 103 (2015)

    Article  Google Scholar 

  34. Freitas, D.Q., Da Gama, A.E.F., Figueiredo, L., Chaves, T.M., Marques-Oliveira, D., Teichrieb, V., Araujo, C.: Development and evaluation of a Kinect based motor rehabilitation game. In: Proceedings of SBGames, pp. 144–153. (2012)

  35. Altanis, G., Boloudakis, M., Retalis, S., Nikou, N.: Children with motor impairments play a Kinect learning game: first findings from a pilot case in an authentic classroom environment. J. Interact Des. Archit. 19, 91–104 (2013)

    Google Scholar 

  36. Pompeu, J.E., Arduini, L.A., Botelho, A.R., Fonseca, M.B.F., Pompeu, S.M.A.A., Torriani-Pasin, C., Deutsch, J.E.: Feasibility, safety and outcomes of playing Kinect Adventures!™ for people with Parkinson’s disease: a pilot study. Physiotherapy 100(2), 162–168 (2014)

    Article  Google Scholar 

  37. Lee, J.D., Hsieh, C.H., Lin, T.Y.: A Kinect-based Tai Chi exercises evaluation system for physical rehabilitation. In: Proceedings of the IEEE International Conference on Consumer Electronics (ICCE), pp. 177–178. (2014)

  38. Chang, C.Y., Lange, B., Zhang, M., Koenig, S., Requejo, P., Somboon, N., Sawchuk, A. A., Rizzo, A.A.: Towards pervasive physical rehabilitation using Microsoft Kinect. In: Proceedings of 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health), pp. 159–162. (2012)

  39. Mulroy, S.J., Thompson, L., Kemp, B., Hatchett, P.P., Newsam, C.J., Lupold, D.G., Haubert, L.L., Eberly, V., Ge, T.T., Azen, S.P., Winstein, C.J., Gordon, J.: Strengthening and optimal movements for painful shoulders (STOMPS) in chronic spinal cord injury: a randomized controlled trial. Phys. Ther. 91, 305–324 (2011)

    Article  Google Scholar 

  40. Clark, R., Kraemer, T.: Clinical use of Nintendo Wii bowling simulation to decrease fall risk in an elderly resident of a nursing home: a case report. J. Geriatr. Phys. Ther. 32(4), 174–180 (2009)

    Article  Google Scholar 

  41. Deutsch, J.E., Borbely, M., Filler, J., Huhn, K., Guarrera-Bowlby, P.: Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with cerebral palsy. Phys. Ther. 88(10), 1196–1207 (2008)

    Article  Google Scholar 

  42. Deutsch, J.E., Brettler, A., Smith, C., Welsh, J., John, R., Guarrera-Bowlby, P., MichalKafri, M.: Nintendo Wii sports and Wii Fit game analysis, validation, and application to stroke rehabilitation. Clin. Appl. Technol. Pract. 18(6), 701–709 (2011)

    Google Scholar 

  43. Agarwal, R., Sampath, H.A., Indurkhya, B.: A usability study on natural interaction devices with ASD children. In: Stephanidis, C., Antona, M. (eds.) Universal Access in Human-Computer Interaction User and Context Diversity LNCS Volume 8010, pp. 447–453. Springer, Berlin, Heidelberg (2013)

    Chapter  Google Scholar 

  44. Saiano, M., Pellegrino, L., Casadio, M., Summa, S., Garbarino, E., Rossi, V., Dall’Agata, D., Sanguineti, V.: Natural interfaces and virtual environments for the acquisition of street crossing and path following skills in adults with Autism Spectrum Disorders: a feasibility study. J. NeuroEng. Rehabil. 12, 17 (2015)

    Article  Google Scholar 

  45. Bai, Z., Blackwell, A.F., Coulouris, G.: Using augmented reality to elicit pretend play for children with autism. IEEE Trans. Vis. Comput. Graph. 21(5), 598–610 (2015)

    Article  Google Scholar 

  46. Bartoli, L., Corradi, C., Garzotto, F., Valoriani, M.: Exploring motion-based touchless games for autistic children’s learning. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC’2013), pp. 102–111. ACM Press (2013)

  47. Bartoli, L., Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M.: Designing and evaluating touchless playful interaction for ASD children. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC’14), pp. 17–26. ACM Press (2014)

  48. Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M: Motion-based touchless interaction for ASD children: a case study. In: Proceedings of AVI, pp. 117–120. ACM Press (2014)

  49. Ringland, K.E., Zalapa, R. Neal, M, Escobedo, L. Tentori, M., Hayes, G.: SensoryPaint: a natural user interface supporting sensory integration in children with neurodevelopmental disorders. İn: CHI ‘14 Extended Abstracts, pp. 1681–1686. ACM Press (2014)

  50. Casas, X., Herrera, G., Coma, I., Fernández, M.: A kinect-based augmented reality system for individuals with autism spectrum disorders. In: Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications (GRAPP/IVAPP ‘12). Pp. 440–446. SciTePress (2012)

  51. Mora-Guiard, J., Crowell, C., Pares, N., Heaton, P.: Lands of fog: helping children with autism in social interaction through a full-body interactive experience. In: Proceedings of the 15th International Conference on Interaction Design and Children (ACM IDC’2016), pp. 262–274. (2016)

  52. Van Zomeren, A.H., Brouwer, W.H.: Clinical Neuropsychology of Attention. Oxford University Press, USA (1994)

    Google Scholar 

  53. Brickenkamp, R., Zillmer, E.: The d2 Test of Attention. Hogrefe & Huber Publication, Göttingen (1998)

    Google Scholar 

  54. Bhattacharya, A., Gelsomini, M., Pérez-Fuster, P., Abowd, G.D., Rozga, A.: Designing motion-based activities to engage students with autism in classroom settings. In: Proceedings of the 14th International Conference on Interaction Design and Children (ACM IDC’2015), pp. 69–78. (2015)

  55. Tung, J.Y., Lulic, T., Gonzalez, D.A., Tran, J., Dickerson, C.R., Roy, E.A.: Evaluation of a portable marker less finger position capture device: accuracy of the Leap Motion controller in healthy adults. Physiol. Meas. 36(5), 1025–1035 (2015)

    Article  Google Scholar 

  56. Hondori, H.M., Khademi, M.: A review on technical and clinical impact of microsoft kinect on physical therapy and rehabilitation. J. Med. Eng. 2014, 846514 (2014)

    Google Scholar 

  57. Ebner, M., Spot, M.: Game-based learning with the leap motion controller. In: Russell, D., Laffey, J.M. (eds.) Handbook of Research on Gaming Trends in P-12 Education. IGI Global, pp 555–565. (2015)

  58. Iosa, M., Morone, G., Fusco, A., Castagnoli, M., Fusco, F.R., Pratesi, L., Paolucci, S.: Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top. Stroke Rehabil. 22(4), 306–316 (2015)

    Article  Google Scholar 

  59. Potter, L.E., Araullo, J., Carter, L.: The Leap Motion controller: a view on sign language. In: Proceedings of the 25th Australian Computer-Human Interaction Conference on Augmentation, Application, Innovation, Collaboration—OzCHI’13, pp. 175–178. (2013)

  60. Gieser, S.N., Boisselle, A., Makedon, F.: Real-time static gesture recognition for upper extremity rehabilitation using the leap motion. In: Lecture Notes in Computer Science, pp. 144–154. (2015)

  61. Tang, T., Winoto, P., Wang, R.: Having fun over a distance: supporting multiplayer online ball passing using multiple sets of Kinect. In: CHI ‘15 Extended Abstracts, pp. 1187–1192. ACM Press, New York (2015)

  62. Zhu, G., Cai, S., Ma, Y., Liu, E.: A series of leap motion-based matching games for enhancing the fine motor skills of children with autism. In: 15th International IEEE Conference on Advanced Learning Technologies (ICALT’2015), IEEE, pp. 430–431. (2015)

  63. Zwaigenbaum, L., Bauman, M.L., Choueiri, R., Fein, D., Kasari, C., Pierce, K., Stone, W.L., Yirmiya, N., Estes, A., Hansen, R.L., McPartland, J.C., Natowicz, M.R., Buie, T., Carter, A., Davis, P.A., Granpeesheh, D., Mailloux, Z., Newschaffer, C., Robins, D., Roley, S.S., Wagner, S., Wetherby, A.: Early identification and interventions for autism spectrum disorder: executive summary. Pediatrics 136(Suppl 1), S1–S9 (2015)

    Article  Google Scholar 

  64. Greis, F., Silva, M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. Comput. Support. Coop. Work 24(2-3), 149–175 (2015)

    Article  Google Scholar 

  65. Goh, W., Shou, W., Tan, J., Lum, G.T.J. Interaction design patterns for multi-touch tabletop collaborative games. In: CHI ‘12 Extended Abstracts on Human Factors in Computing Systems (CHI EA´12), pp. 141–150. New York, NY, USA (2012)

  66. Chelsea La Valle. Chinese Cultural Factors Impacting the Educational Schooling of Children with Autism in China. DePaul Discoveries: Vol. 2: Issue 1, Article 10. Available at: http://via.library.depaul.edu/depaul-disc/vol2/iss1/10 (2013)

  67. Wang, S.: In China, the making of an App for autism. Wall Street J. May 19, 2015. http://blogs.wsj.com/chinarealtime/2015/05/19/in-china-the-making-of-an-app-for-autism/ (2015)

  68. Marco, E., Barett, L., Hinkley, N., Hill, S., Nagarajan, S.S.: Sensory processing in autism: a review of neurophysiologic findings. Pediatr. Res. 69(5 Pt 2), 48R–54R (2011)

    Article  Google Scholar 

  69. Baron-Cohen, S., Golan, O., Ashwin, E.: Can emotion recognition be taught to children with autism spectrum conditions? Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci 364(1535), 3567–3574 (2009). doi:10.1098/rstb.2009.0191;10.1098/rstb.2009.0191

    Article  Google Scholar 

  70. Gordon, I., Pierce, M., Bartlett, M., Tanaka, J.: Training facial expression production in children on the autism spectrum. J. Autism Dev. Disord. 44(10), 2486–2498 (2014). doi:10.1007/s10803-014-2118-6

    Article  Google Scholar 

  71. Ploog, B., Scharf, A., Nelson, D., Brooks, P.: Use of computer-assisted technologies (CAT) to enhance social, communicative, and language development in children with autism spectrum disorders. J. Autism Dev. Disord. 43(2), 301–322 (2013). doi:10.1007/s10803-012-1571-3

    Article  Google Scholar 

  72. Rice, L., Wall, C., Fogel, A., Shic, F.: Computer-assisted face processing instruction improves emotion recognition, mentalizing, and social skills in students with ASD. J. Autism Dev. Disord. 45(7), 2176–2186 (2015). doi:10.1007/s10803-015-2380-2

    Article  Google Scholar 

  73. Tang, T.Y., Flatla, D.: Autism awareness and technology-based intervention research in China: the good, the bad, and the challenging. In: Proceedings of Workshop on Autism and Technology—Beyond Assistance and Intervention, in Conjunction with the 34th ACM International Conference on Human Factors in Computing Systems (CHI’2016) (2016)

  74. Goldsmith, T.R., LeBlanc, L.A.: Use of technology in interventions for children with autism. J. Early Intensive Behav. Interv. 1(2), 166 (2004)

    Article  Google Scholar 

  75. Hayes, G.R., Hirano, S., Marcu, G., Monibi, M., Nguyen, D.H., Yeganyan, M.: Interactive visual supports for children with autism. Pers. Ubiquitous Comput. 14(7), 663–680 (2010)

    Article  Google Scholar 

  76. Silver, M., Oakes, P.: Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 5(3), 299–316 (2001)

    Article  Google Scholar 

  77. Clampton, N.: China moves to tackle autism with first study of prevalence. South China Morning Post. (2013)

Download references

Acknowledgements

The authors would like to acknowledge Kean University’s financial support to Tiffany Tang under Wenzhou-Kean University’s Student Partnering with Faculty (SpF) Research Program. The authors are also grateful to children and their families at Wenzhou XingLe Children’s Educational Development Center and Wenzhou Orange Wheat Children's Educational center for participating in the experiments; thanks also go to Tina Xiaoting Fu and Esther Mingyue Tang for developing the two games and their efforts during both the experiment and interviews as well as Leila Zeqiang Huang, Relic Yongfu Wang and Pinata Winoto for their assistance in the experiments.

Author information

Authors and Affiliations

  1. Media Lab, Department of Computer Science, Wenzhou-Kean University, Zhejiang, China

    Tiffany Y. Tang

  2. Department of Occupational Therapy, Kean University, Union, NJ, USA

    Mary Falzarano

  3. Department of Computer Science, Kean University, Union, NJ, USA

    Patricia A. Morreale

Authors
  1. Tiffany Y. Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mary Falzarano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patricia A. Morreale
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tiffany Y. Tang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, T.Y., Falzarano, M. & Morreale, P.A. Assessment of the utility of gesture-based applications for the engagement of Chinese children with autism. Univ Access Inf Soc 17, 275–290 (2018). https://doi.org/10.1007/s10209-017-0562-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-017-0562-8

Keywords

Search

Navigation