Validation of the Chinese Handwriting Analysis System (CHAS) for primary school students in Hong Kong

https://doi.org/10.1016/j.ridd.2013.05.048Get rights and content

Highlights

  • Examined the validity, internal consistency, and test–retest reliability of CHAS.

  • Satisfactory and reasonable results were found.

  • CHAS can assist therapists in identifying students with handwriting problems.

Abstract

There are more children diagnosed with specific learning difficulties in recent years as people are more aware of these conditions. Diagnostic tool has been validated to screen out this condition from the population (SpLD test for Hong Kong children). However, for specific assessment on handwriting problem, there seems a lack of standardized and objective evaluation tool to look into the problems. The objective of this study was to validate the Chinese Handwriting Analysis System (CHAS), which is designed to measure both the process and production of handwriting. The construct validity, convergent validity, internal consistency and test–retest reliability of CHAS was analyzed using the data from 734 grade 1–6 students from 6 primary schools in Hong Kong. Principal Component Analysis revealed that measurements of CHAS loaded into 4 components which accounted for 77.73% of the variance. The correlation between the handwriting accuracy obtained from HAS and eyeballing was r = .73. Cronbach's alpha of all measurement items was .65. Except SD of writing time per character, all the measurement items regarding handwriting speed, handwriting accuracy and pen pressure showed good to excellent test–retest reliability (r = .72–.96), while measurement on the numbers of characters which exceeded grid showed moderate reliability (r = .48). Although there are still ergonomic, biomechanical or unspecified aspects which may not be determined by the system, the CHAS can definitely assist therapists in identifying primary school students with handwriting problems and implement interventions accordingly.

Introduction

Handwriting is a survival skill for students and they are expected to develop proficient handwriting skills in order to carry out their school work. In particular, handwriting is one of the most important means of determining the academic abilities of students in the Chinese educational system and students usually start their journey of writing since preschool age. When they are promoted to the primary school, over 50% of their time spent in school is involved in handwriting tasks (Tseng & Chow, 2000). Poor handwriting abilities may have adverse effect on students’ performance in tests and public examinations. Apart from academic success, failure to attain handwriting competency during the school-age years has long-term negative effects on students’ self-esteem (Feder & Majnemer, 2007). According to the Hong Kong Monthly Digest of Statistics (2009), there are about 9900 students with Specific Learning Difficulties (SpLD) in 2009. According to Chan's study in 2007, there are approximately 1 out of 10 students in Hong Kong having handwriting problems (Chan, Ho, Tsang, Lee, & Chung, 2007).

Unlike the alphabetic languages such as English that emphasize smoothness and continuity in their written forms, Chinese characters are logographic in nature and its complexity with multiple-stroke sequence and directions (Lam, Au, Leung, & Li-Tsang, 2011). Complex geometric figurations and stroke arrangements within a squared area are involved in Chinese handwriting (Chow et al., 2003, Tan et al., 2001). If a child writes “Iam ab oy”, the teacher can still guess the meaning (I am a boy) even though the spacing between the letters is incorrect. However, if a child writes the Chinese character with wrong strokes or radical positions and spacing, it can become another character with a very different meaning. Thus, writing proficiency in Chinese is more demanding than writing in English (Lam et al., 2011). Moreover, handwriting evaluation tools developed to assess English writing may not be applicable to evaluate Chinese handwriting.

The Chinese handwriting evaluation tools currently available to assess primary school students’ handwriting skills are the Tseng's Handwriting Speed Test (Tseng & Hsueh, 1997), the Handwriting Ability Checklist (HAC; Tam, 2008), and the Penmanship Objective Evaluation Tool (Rosenblum et al., 2001, Rosenblum et al., 2003a, Rosenblum et al., 2003b). However, these handwriting assessments are weak as they fail to measure handwriting accuracy and legibility.

The Tseng's Handwriting Speed Test (Tseng & Hsueh, 1997) is the most commonly used evaluation tools used to assess primary school students’ handwriting speed. It was developed in Taipei to assess Chinese handwriting speed of students from grade 2 to grade 6. Students were asked to copy a text with 475 characters on an A4-sized paper with pre-printed grids as fast and as legible as possible for 5 min. All the characters were studied in the second semester of the grade 1 in Taiwan primary school curriculum. The number of characters copied was counted manually and the writing speed was calculated. Normative study was conducted on 1525 subjects in Taipei area. Good test–retest reliability was found (r = .98). However, as the test was specifically validated in relevance to the population and school curriculum in Taipei, it may not be valid to be used in Hong Kong.

Handwriting Assessment Checklist (HAC) was a locally developed and validated handwriting evaluation checklist (Tam, 2008). It consisted of 10 questions concerning 3 domains of handwriting: writing process, writing product, physical and emotional well-being. Either teachers or parents could score it. The total score of the HAC ranged from 0 to 40. Increase in score represented decrease in handwriting proficiency. Test–retest reliability for both teachers and parents was good for the total score of HAC with high Intraclass Correlation Coefficient (ICC). For teacher, r = .92 and for parents, r = .84. Significant with moderate degree of relationship was found between the rating of parents and teachers (r = .61). Internal consistency of the total score of HAC was good with Cronbach's alpha = .94 for the rating of teacher and .80 for that of parents. The Content Validity Index (CVI) of relevance and clarity of all items ranged from .78 to .89. Principal Component Analysis showed that the question items loaded on the same construct of handwriting as proposed in the HAC. However, HAC can only primarily screen students’ handwriting skill and identify those at risk of handwriting difficulties. It cannot provide specific information on a student's handwriting skill such as the handwriting speed and the percentage of handwriting accuracy. Further handwriting evaluation is required to further examine the severity and the underlying problems of students with handwriting difficulties.

The Penmanship Objective Evaluation Tool (Rosenblum et al., 2001, Rosenblum et al., 2003a, Rosenblum et al., 2003b) was the only digital-based handwriting evaluation tool developed locally to assess students’ handwriting skill including total writing time, writing speed, and writing pressure exerted on the writing surface. Students were required to copy two Chinese templates, one with 6 characters and one with 20 characters on a WACOM digitizer in front of a laptop computer. It could capture the data generated from the WACOM tablet to assess the writing speed, the ground time, pen in air time, and pen pressure. Previous studies also adopted the POET to measure the handwriting process of children (Poon, Li-Tsang, Weiss, & Rosenblum, 2010). However, the POET could only measure the students’ handwriting speed, pen in air time, ground time and handwriting pen pressure. The handwriting legibility and accuracy remained difficult to be captured using the POET system. In addition, the POET system has not yet been commercialized for clinical application.

Apart from the above handwriting evaluation tools, there are also a few preliminary development of tools in recent years, such as the Chinese Handwriting Assessment Program (CHAP) which is a preliminary development of computer-assisted assessment of Chinese handwriting performance (Chang, Yu, & Shie, 2009). It was written in the MATLAB language and based on the Tseng Checklist and the Elementary Reading and Writing test (Hung et al., 2003, as cited in Chang et al., 2009) to evaluate the neatness and legibility of Chinese handwriting. The program adopted a template-matching approach and could measure size control, spacing, alignment, and the average resemblance between standard models and handwritten characters. By analyzing the data obtained from 20 grade 4 students, the test–retest reliability was found to be satisfactory (r = .81–.94). In addition, the data from 66 grade 3 students showed that the correlations between each CHAP and Tseng Checklist item were statistically significant. The CHAP was not meant for assessment of handwriting speed, handwriting errors, and the percentage of handwriting accuracy. There is also a lack of normative reference for students studying at different grades for comparison.

In order to develop a standardized, valid and reliable Chinese handwriting evaluation tool, the principles of developing a handwriting evaluation tool and approaches of measuring domains of handwriting will be reviewed.

According to Ziviani and Watson-Will (1998), a valid handwriting evaluation tool should be carefully constructed according to the constructs of handwriting and quantitative scores for different handwriting constructs should be provided. Besides, the assessment procedure, the handwriting assignment and the evaluation criteria should be similar to the writing tasks in school. Ergonomic factors such as environmental set-up and the use of writing accessories should also be standardized (Feder & Majnemer, 2007). All of these will be considered during the development of the handwriting evaluation tool.

Selection of measurement items is a critical step in the development of any evaluation tool. The measurement items should be valid to indicate the development of the targeted areas. Besides, the measurement scores should be able to reflect the performance of the individual and can be used for comparison with other individuals. Also, the scoring procedure should be reliable with minimized subjective interpretations (Stefansson & Karlsdottir, 2003).

The familiarity and complexity of the characters in the writing assignment will affect students’ performance in the evaluation directly. Thus, the writing assignment should be similar to what students frequently write at school with diverse complexity.

In the development of a handwriting assessment protocol, focus was put to connect the assessment procedure, the environment set-up and the writing materials with the real-life handwriting context (Sudsawad, Trombly, Henderson, & Tickle-Degnen, 2001). Ergonomic factors such as the placement of writing materials, lighting and noise will be carefully controlled (Feder & Majnemer, 2007).

Handwriting is a complex human activity interweaving cognitive, tactile and kinesthetic sensitivities and perceptual-motor components (Feder & Majnemer, 2007). When a student knows what to write, he or she has to retrieve the correct letters or words from memory, organize them into the right order, and convert phonemes into graphemes and plan and execute a motor program (Volman, Schendel, & Jongmans, 2006). It is commonly agreed that handwriting is constructed by 4 major domains, including handwriting speed, handwriting accuracy, handwriting legibility and ergonomic factors during handwriting (Feder and Majnemer, 2007, Stefansson and Karlsdottir, 2003). All of these domains should be evaluated in assessing students’ handwriting skill.

Handwriting speed represents either the amount of time required to write specific text or the amount of text reproduced within a specific time period. Handwriting speed is the easiest and common handwriting domain to be evaluated. Students are usually required to copy a number of characters, sentences or paragraphs (Tseng & Hsueh, 1997). The amount of handwriting produced per minute is then calculated.

Handwriting accuracy represents the deviation between the trajectory of the handwriting produced in comparison to the standard and the trajectory of the actual handwriting required (Stefansson & Karlsdottir, 2003). Handwriting accuracy is regarded as the most challenging and debatable handwriting domain to be evaluated and difficult to be defined. Teachers and clinicians evaluate students’ handwriting accuracy with different standards. For many years, large variations among different raters exist. Limited researchers attempted to define handwriting accuracy and few researches have been done in the past 20 years. With limited information, teachers and clinicians continue to perform their own evaluation with different criteria.

Handwriting legibility represents the degree of the handwriting produced in consideration of alignment and size on a line and spacing between letters and words in relation to each other as well as the organization of the whole page (Tomchek & Schneck, 2006, chap. 14). Two major approaches named the global holistic evaluation approach and the analytic evaluation approach (Rosenblum et al., 2003a, Rosenblum et al., 2003b) have been commonly adopted by teachers and clinicians to evaluate students’ handwriting legibility.

The global holistic evaluation approach focuses on judging how readable a written passage is in comparison to a group of standard writing samples that have been previously graded from readable to unreadable. The analytical evaluation approach, based on the assumption that there is a relationship between readability and the quality of specific features of handwriting, judges or grades different features for the entire written passage according to predetermined criteria. Example of those features includes letter size and slant, spacing between letters and words, the straightness of lines, letter forms and shapes (Rosenblum, Chevion, & Wiess, 2006). Both of these approaches are criticized as subjective, time consuming and unreliable in nature (Rosenblum, Weiss, & Parush, 2004).

Ergonomic factors include environment lighting and noise, heights of chair and table, the type of paper used and its placement on the desk, writing posture, type of pencil grasp, and stability of the writing hand (Feder and Majnemer, 2007, Summers and Catarro, 2003, Tomchek and Schneck, 2006). These factors should be carefully monitored and observed in a handwriting evaluation (Feder & Majnemer, 2007), and they will be standardized and controlled throughout the handwriting evaluation.

Based on the principles highlighted above, the Chinese Handwriting Analysis System (CHAS) is designed for Chinese students with handwriting difficulties. It is a computerized handwriting evaluation system developed to assess handwriting speed, pen pressure, handwriting accuracy, and character size. It consists of the Chinese Handwriting Assessment Tool (CHAT) and the Handwriting Analysis System (HAS). The CHAT was developed to evaluate the handwriting performance of each participant (Cheng, 2010, Lam, 2010, Li-Tsang et al., 2011), while the data as recorded in CHAT can be imported into HAS for further analysis. The CHAT consists of a digitized writing board (WACOM Intuos 3 digitizer) to be used with an ink pen, which can capture the handwriting data such as pressure exerted on the writing board while a user is writing on the grid paper. The rationales behind the development of the system and the definition of the variables can be found in Section 2. Each participant will be instructed to sit in front of the computer screen at a distance of 50 cm. The template consists of 90 common Chinese characters of primary 1 level selected from a list of Chinese Characters recommended for the subject of Chinese Language in primary schools in Hong Kong (Li-Tsang et al., 2011). During the assessment, the 90 Chinese characters with size 26, font type “Simsun” and triple-line spacing will be shown on the computer screen, and displayed in 9 columns of 10 characters. The display sequence of the columns will be randomized each time when the system was operated. Each participant will be instructed to copy the 90 characters as fast and accurately as possible on a piece of paper with a 9 × 10 grid pasted on the handwriting digitizer (Fig. 1). They will be asked to write the Chinese characters as quickly as possible but the characters should be readable and legible. Besides, they will be asked write just like what they used to in daily life, and write each character inside the grid accordingly and not to correct their writing if they write the character wrongly.

The objective of this study was to validate the Chinese Handwriting Analysis System (CHAS). The validation of CHAS was done by analyzing its construct validity, and convergent validity, while the reliability of CHAS was done by analyzing its internal consistency and test–retest reliability.

Section snippets

Selection of measurement items

An expert panel was formed to monitor and critique on the development of CHAT. It consisted of 5 experienced pediatric occupational therapists, 4 experienced primary school teachers teaching Chinese subject and 1 experienced educational psychologist. All of them had not less than 10 years of working experience in their expertise. 4 Expert panel meetings were held to ensure the constructs of CHAT were valid. In each meeting, progress on the selection of measurements items, the compilation of

Descriptive statistics and data selection

Among the data from 1136 subjects, 642 were males (56.51%) while 494 were females (43.49%), with age ranging from 6.06 to 14.20. The grades of the subjects are distributed evenly, ranging from 14.96% to 19.28%. In addition, apart from one school which contributed to 21.21% of the total number of subjects, all the other schools contributed to about 15%. Boxplots were performed for all the measurement items. 22 outliers were identified. Thus, only the data from 1114 subjects were included in

CHAT results obtained

Overall, the results were reasonable and supported the literature (e.g., Tseng & Hsueh, 1997) and clinical observations that students’ handwriting speed and pen pressure would improve with maturation.

Construct validity of CHAS

Principal Component Analysis revealed that the measurements items loaded into 4 components. Specifically, the measurement items of handwriting speed, pen pressure, handwriting accuracy, and character size were found to be in four different domains separately. The results are in line with the four

Conclusion

The objective of this study was to validate the Chinese Handwriting Analysis System (CHAS). The construct validity, and convergent validity, internal consistency and test–retest reliability of CHAS was analyzed using the data from 734 grade 1–6 students from 6 primary schools in Hong Kong. The CHAS showed reasonable and satisfactory results in construct validity, convergent validity, internal consistency, and test–retest reliability. To conclude, although there are still ergonomic,

References (34)

  • J.L. Hintze

    PASS power analysis

    (2000)
  • Hong Kong Monthly Digest of Statistics

    Persons with disabilities and chronic diseases in Hong Kong

    (2009)
  • A. Kaplan et al.

    Achievement goal orientations and self-regulation in writing: An integrative perspective

    Journal of Educational Psychology

    (2009)
  • S.T. Lam

    Chinese handwriting performance of primary school children with dyslexia

    (2010)
  • D. McCutchen

    Knowledge acquisition, processing efficiency, and working memory: Implications for a theory of writing

    Educational Psychologist

    (2000)
  • L.G. Portney et al.

    Foundations of clinical research: Application to practice

    (2009)
  • S. Rosenblum et al.

    Using data visualization and signal processing to characterize the handwriting process

    Pediatric Rehabilitation

    (2006)
  • Cited by (27)

    • Multi-criteria handwriting quality analysis with online fuzzy models

      2017, Pattern Recognition
      Citation Excerpt :

      In this paper, three main criteria are used to analyse the handwriting quality: shape, order and direction. Similarly to [2–5], the proposed approach evaluates the handwriting quality with regards to reference models using intra-class and inter-class scores (i.e. the similarity to the analysed model and the difference to other models, see Fig. 4). Compared to the previous approach [5], specialised features and classifiers are introduced for each criterion.

    • Persian Handwriting Assessment Tool: Reliability in Students with Specific Learning Disorders

      2024, Middle East Journal of Rehabilitation and Health Studies
    • DYPA: A Machine Learning Dyslexia Prescreening Mobile Application for Chinese Children

      2023, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    View all citing articles on Scopus
    View full text