Introduction

The American Spinal Injury Association (ASIA) first published a standard system for neurological assessment and classification of spinal cord injury (SCI) in 1982 that involves bilateral strength testing of 10 key muscles (five upper extremity and five lower extremity) and bilateral sensory testing (sharp/dull discrimination and light touch (LT)) of 28 dermatomes.1 Following a decade of use, major revisions to the standards were published2 and adopted by the International Medical Society of Paraplegia (IMSOP). Adoption by IMSOP reflected acceptance among the international SCI community as the standard for neurological assessment of persons following SCI. Most recently, Marino et al3 provided clarity on terminology related to the standards and, in their current form,4 the standards continue to represent the most common method for assessment of neurological consequence and classification of SCI.

The ISCSCI motor and sensory exam has been used as the basis for describing the adult and pediatric population with SCI,5, 6, 7 and have been the primary indicators to predict recovery of neurological function.8, 9, 10, 11, 12 Others have studied the relationship between motor and sensory scores and bladder recovery,13 the recovery patterns of muscle strength and sensation in vascular injuries,14 and the relationship between muscle strength/motor scores and functional recovery.15, 16, 17, 18, 19, 20

The ISCSCI motor and sensory exams have also been used as inclusion criteria for entry into drug and device trials.21, 22, 23, 24, 25, 26, 27, 28 While the work by Bracken21 added additional upper extremity muscles and variation in the scoring technique, the study used neurological motor level as the primary outcome in the study on high-dose methylprednisolone. Outcomes of activity-based rehabilitation, a program of intensive cycling, assisted treadmill training and swimming, also use the International standards as primary outcomes.22 Studies of functional electrical stimulation devices have extensive history using the International motor and sensory exams as both an entry criteria and outcome variable.23, 24, 25, 26, 27, 28

Despite the extensive use of the ISCSCI, few studies report the reliability of the motor and sensory data. The first reliability and validity studies29, 30 of both examination and classification skills were conducted following the 1992 revision31 and showed much improved reliability of the examination using the ISCSCI, but no change in the marginal reliability of classification. In a subsequent study by Cohen et al,32 reliability of classification improved after a training session, particularly for complex incomplete injuries. Additional revisions of the ISCSCI occurred in 1996 and 2002 to better define terminology with the intent to improve reliability.3 The only published report on the reliability of the current ISCSCI was by Marino et al33 in which inter-rater and intra-rater reliability was evaluated for summed motor and sensory scores. In this study, Marino reported that inter-rater reliability was high for total motor (TM) (0.97), pin prick (PP) (0.88) and LT (0.96) scores when the exam was performed by expert raters. Intra-rater reliability was good for the motor exam but poor for the two sensory exams. Based on this study, Marino et al concluded that the ISCSCI was appropriately reliable for use in clinical trials. None of the published reports or user's manual provides confidence intervals (CI) for the reported intraclass correlation coefficients (ICC), thus precision of the exams are not known.

Although it is a routine technique for assessing pediatric patients with SCI6, 7 and is considered the gold-standard assessment for both prognosis and outcomes, the reliability of the ISCSCI data has yet to be established with children and youths. The primary purpose of this research study was to begin to establish the reliability of the ISCSCI motor and sensory exam when applied to children and youths between birth and 21 years of age with stable SCI. Within this primary purpose, there were two secondary objectives of the study. The first objective was to define the lower age limit in which the ISCSCI neurological examination generates reliable data and the second objective was to obtain pilot data for a future research effort on pediatric modifications of the ISCSCI or development of a pediatric-specific neurological measurement and classification scheme for children and youths with SCI.

Methods

This reliability study investigated test–retest reliability of the ISCSCI summed motor scores and sensory scores obtained from 74 children or youths with SCI. Motor and sensory scores were obtained on children and youths according to the 2002 international standards.3, 4

Sample

A total of 84 children or youths were reviewed for participation in the study. Four children were not considered due to the presence of traumatic brain injury and/or other comorbidities that interfered with the cognitive ability to follow standard testing instructions. The remaining 80 children were invited to participate in the study. Two adolescents with tetraplegia in the oldest age group declined participation. Four other children participated in the first exam, but did not participate in the second exam. One subject in the 11–15 age group was withdrawn from the study by the investigator because he had normal sensory and motor function (ASIA E) and a repeat exam was felt to be inappropriate. Three other subjects (one in the 6–11 age group; two in the 12–15 age group) voluntarily withdrew by refusing to participate in the second exam. Therefore, the study sample consisted of 74 children or youths between 8 months and 21 years of age with stable SCI.

The study protocol, consent and assent were all approved by the research ethics/institutional review board. Written informed consent was obtained from all parents or legal guardians of all subjects under 18 years of age. As per guidelines of the Institutional Review Board, subjects between 7 and 18 years of age also provided written informed assent; subjects older than 18 years of age provided their own written consent. For those subjects who spoke Spanish as their primary language, an interpreter assisted in the informed consent process and testing procedures. The IRB-approved Health Insurance Privacy and Portability Act (HIPPA) forms were also reviewed with subjects for their consent. We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed throughout the duration of this study.

Data collection

Following consent, repeated ISCSCI motor and sensory exams were conducted by the primary author; the time between the two exams ranged between 24 and 48 h. Prior to data collection, the primary author conducted motor and sensory exams according to the standards for a 10-year period with approximately 150 patients for routine rehabilitation assessment and clinical research activity. In addition, she attended two formal training workshops on the motor and sensory exam as part of a research protocol. The time span between exams was chosen with existing clinical processes in mind – patients come to the hospital for a 2–3 day period for annual exams and ‘brush up’ rehabilitation.

As per standard the ISCSCI test protocol,4 the subjects were positioned on their backs in a bed or on a mat for the exam. Sensory appreciation to sharp/dull was tested starting in dermatomes with known impairment and proceeded upwardly towards the nonimpaired dermatomes. A standard sterile safety pin was used for testing (pointed end was used for sharp and rounded end was used for dull). Appreciation for LT proceeded similarly using a sterile cotton swab.

Muscle testing also occurred while supine, and began with testing of the C5 muscle on the subject's dominant extremity. Muscle testing of each key muscle on the dominant extremity followed. Once testing of the dominant extremity was complete, the strength of key muscles of the nondominant was tested.

Testing for severity of the injury was carried out as per the ISCSCI standard instructions including the anal exam. For younger children, particularly those injured prior to being toilet trained, test instructions for the anal contraction required multiple clarifications and several variations of the instruction. Often, parents were asked to restate the test instructions in a way they felt their child would comprehend the request. Often children's earnest attempts at an anal contraction caused spasticity of abdominal muscles. The researcher was diligent in making sure abdominal muscle spasticity was not mistaken for an anal contraction by requesting confirmation of the exam, when needed, by either an experienced physician or therapist. Data were de-identified and entered into an excel spreadsheet.

Data analysis

Data were transferred into SPSS for Windows release 11.0.1. Reliability was determined using ICC for TM scores, total PP scores and total LT scores. Sub-analysis involved the calculation of ICC and 95% confidence (CI) for four age groups (0–5, 6–11, 12–15, 16–21), severity of injury (complete, incomplete) and type of injury (paraplegia, tetraplegia). Coefficients above 0.90 were indicative of high reliability and values between 0.75 and 0.90 were indicative of moderate reliability. Values less than 0.75 were considered to have inadequate reliability. The 95% CI provides an indication of precision such that any given ICC can range between the lower CI limit and higher CI limit. Wide intervals indicate low precision and narrow intervals indicate high precision.

Results

As summarized in Table 1, a total of 74 children or youths between 8 months and 21 years of age with stable SCI provided written informed consent and/or assent, and attempted two repeated exams of motor and sensory function as described by the ISCSCI. Subjects were considered neurologically stable if there was no change documented over the 6-month period prior to study enrollment. Age groupings were defined based on previous publications, addressing the developmental considerations in pediatric SCI.6

Table 1 Characteristics of sample

Tables 2, 3, 4, 5, 6 and 7 summarize the results of reliability analyses. As shown in Table 2, when age, type of injury and severity of injury were combined, there was adequate reliability (0.888) for TM scores. High reliability was evidenced by ICC of 0.975 and 0.974 for total PP and total LT scores, respectively. The 95% CI indicate high precision of the coefficients for sensory testing and acceptable precision for motor testing. These coefficients and CI exceed the recommended reliability values for clinical tools.

Table 2 ICC coefficients for sample of subjects between 4 and 21 years of age
Table 3 ICC coefficients for TM scores for each age group
Table 4 ICC coefficients and 95% CI for total PP scores
Table 5 ICC coefficients and 95% CI for total LT scores
Table 6 ICC coefficients and 95% CI for complete and incomplete injuries
Table 7 ICC coefficients and 95% CI for tetraplegia and paraplegia

3, 4 and 5 show ICC and 95% CI for TM (Table 3), PP (Table 4) and LT (Table 5) for each age group. Shaded values denote poor reliability and wide CI. With the exception of TM scores for the 6–11-year-old age group, high reliability for TM, PP and LT scores was evidenced by ICC above 0.90. However, for the youngest age group, CI showed poor precision for TM scores (0.69–0.98) (Table 3), total PP (0.49–0.985) (Table 4) and total LT (0.63–0.993) (Table 5).

Tables 6 and 7 show ICC and 95% CI for TM, PP and LT scores based on severity of injury (complete, incomplete) and type of injury (paraplegia, tetraplegia), respectively. As shown in Table 6, when all age groups were combined, ICC evidenced high reliability for TM, PP and LT scores for complete and incomplete injuries with the exception of the motor score of complete injuries that showed moderate (0.808) reliability. The CI for motor scores for complete injuries was wide (0.664–0.89), suggesting poor precision. Regardless of the level of injury, ICC for TM, PP and LT scores evidenced high reliability and acceptable precision (Table 7).

Discussion

Lower age limit for ISCSCI

The primary purpose of this research study was to define the lower age limit in which the ISCSCI motor and sensory exams generate reliable data. In order to achieve this purpose, the reproducibility of the ISCSCI motor and sensory exam when applied to children and youths between birth and 21 years of age with stable SCI was examined. Within this effort, 74 children or youths were enrolled in the study and repeated exams of motor and sensory function as defined by the ISCSCI were attempted. Regardless of the type or severity of injury, children under 4 years of age were unable to comprehend and follow the standardized test instructions. With the exception of two who were unable to complete the sensory exams, children older than 4 years were able to participate in the testing, the results of which indicated acceptable to high reliability for TM and sensory scores. Thus, based on this study sample, 4 years of age may be the lower age limit in which the motor and sensory exams generate reliable data when summed scores are of interest.

Importantly however, even though 4 years was the age in which subjects successfully participated in the ISCSCI motor and sensory exams, the majority of them under 10 years were anxious during the PP exam as evidenced by crying, refusing to participate, requesting frequent breaks and/or parents having to encourage them to complete the test; one 6-year-old child withdrew following the first exam due to her anxiety during PP testing. It was clear from this study that elementary school-aged children dislike the PP exam and that their apprehension detracted from the testing procedures.

Another consideration related to the younger age limit concerns the test for completeness. While sensory testing at S4–5 was relatively straight forward, for children who were injured prior to being toilet trained, the instructions for the test for anal contraction were difficult to understand. The majority of children requested clarification on instructions multiple times and, despite several techniques to instruct children on the test, the majority did not consistently comprehend the request. Compounded by the inability to appreciate sensory feedback during attempts to contract, young age at the time of injury and at the time of exam and previous experience with volitional control of bowel movements may be factors that contribute to poor utility of the anal exam in youngsters. The anal exam was also difficult for older subjects due to presumed privacy, sexuality and self-esteem reasons. The two adolescents who declined to participate in the study cited the anal exam as the primary reason for not consenting to the study. The two subjects in the 12–15 year age group who refused the second exam also identified the anal exam as one of the primary reasons for their withdrawal. Adolescents' discomfort with the anal examination was not surprising and is often a challenging aspect of the examination in clinical practice.

Significant work remains in the modification of the ISCSCI for young children. In its current form, the neurological consequence of SCI in infants and toddlers may not be measurable by the ISCSCI and therefore, clinicians should be prudent when making statements about a child's neurological level. Formal testing should be delayed until 4 years of age, at which time standardized methods of assessment should be attempted. Clinicians may want to explain to parents how standardized testing for neurological classification is performed and, because of their child's young age, classification as complete or incomplete may not be reliably determined and only an estimated neurological level can be provided.

In parallel, alternative methods of assessment of neurological impairment in babies with SCI must be developed. Establishing the utility of infant motor scales may be beneficial towards the effort in assessing motor impairment of babies with SCI. Monitoring physiological variables such as heart rate and blood pressure during sensory testing or the use of electrodiagnostics such as nerve conduction studies may also be effective techniques in assessing the neurological consequence of SCI in children. Lastly, understanding the relationships between images of the injured spinal cord on CATscan, MRI or diffuse tensor imaging and the ISCSCI motor and sensory scores may establish a standard assessment technique useful for younger children or for older children who are unable to cognitively participate in the ISCSCI.

Test–retest reliability

With the exception of the 6- to 11-year-old subgroup, in this sample, the ISCSCI exam generated data that demonstrated good reliability for summed scores that meets or exceeds the recommended values for clinical measures regardless of age, type or severity of injury. Based on previous experience of motor and sensory examinations in children that suggest poor repeatability such as found with the 6–11 year olds, the high degree of reliability was an unanticipated finding and interpretation of the study results requires careful consideration and caution.

The first consideration when interpreting the positive results of this study is the CI. Reliability experts34, 35, 36, 37, 38, 39, 40 agree that construction of CI are more instructive and meaningful than the P-value, which simply tells if the coefficient is different from zero. A 95% CI for the ICC indicates the likely range of values containing the true population ICC. For example in this study, while the ICC was high for the TM score for the 4–5 years age group (0.917), the 95% CI indicates that the true reliability is likely to range between the lower limit of 0.69 (poor) to 0.98 (high). The lack of precision in these results may be due to the small number of subjects (N=11) in this subgroup. In this study, there were similar trends in other subgroups with low subject numbers. The ICC for TM score were 0.711 and 0.893 for groups in the 6–11 years age group (N=18) and 12–15 years age group (N=19), respectively. However, the 95% CI was 0.226–0.892 and 0.723–0.959, respectively. Similar trends in wide CI for PP and LT scores were noted in the youngest age group (Tables 4 and 5).

The large CI do not represent a study weakness, but rather provide parameters for interpretation of the results. Published reports of reliability studies of the ISCSCI with adults indicate high reliability for TM, PP and LT scores, but none, including the chapter on reliability of the ISCSCI from the test manual, report the CI. In the absence of CI reporting, ICC cannot be adequately interpreted and study results may be misleading.

The low number of subjects in the subgroups is a limitation of the study. Nunnally and Bernstein38 recommends the need for between 30 and 50 subjects for reliability studies. Interestingly, this study represents the largest test–retest reliability study of the ICSCI motor and sensory exam, including those studies with adults, with a total of 74 subjects enrolled and the only published study of the reliability of the ICSCI in which 95% CI are provided. It is clear that additional reliability studies of each subgroup in this study are warranted with larger numbers of subjects in each group. Due to the relatively low number of children with SCI, particularly those with incomplete SCI, it is likely that multicenter studies will be the only method to recruit a large number of subjects under 21 years of age with varying degrees and types of SCI.

There are additional considerations when interpreting the study results. Two of the primary hypotheses in this study were that motor and sensory exams would be less reliable for children with incomplete SCI (ASIA B, C, D) as compared to the exams of children with complete injuries (ASIA A), and that the reliability of exams from those with tetraplegia would be less than the reliability of exams from those with paraplegia. In this study, the distribution of severity and type of injury was not equal. Due to the relatively high incidence of lap belt injuries and SCI without radiographic abnormalities (SCIWORA) – both of which typically result in clinically complete injuires6 – there was an anticipated under-representation of incomplete injuries in children 11 years of age and younger; among 32 subjects in this age range, only two had incomplete injuries. However, the inability of children in the youngest age group who were not testable was not fully anticipated at the onset of this study. While a total of 13 subjects were recruited for the birth to 5-year-old age group, only one with tetraplegia was able to participate in formal testing (motor exam only). There were two other subjects in the youngest age grouping with paraplegia who were able to participate in the motor exam but not in the sensory exam; five other children, all with paraplegia, participated in motor and sensory testing.

Thus, in the youngest age group, motor and sensory data were obtained in a small number of subjects (motor only N=3; motor and sensory N=5), all of who had complete injuries. Importantly, while this study data showed strong reliability, the wide CI suggest poor precision and as such clinicians should not assume that data are reliable in all 4- and 5-year olds. Rather, it is recommended that clinicians establish reliability of data generated from their patients as routine research practice.

Finally, the inadequate representation of incomplete injuries in all age groups needs to be recognized when examining the study results. Among the 68 subjects who were testable, 49 (72%) had clinically complete (ASIA A) injuries. The disparity between the number of complete and incomplete injuries may be a pediatric phenomenon that may limit future studies that examine severity of injury as it relates to reliability of motor and sensory exams.

There are other limitations to the study that must be considered. The ISCSCI requires both an examination of sensory and motor function and classification of neurological impairment.4 The skills for accurate examination of motor and sensory function are different than the skills required for accurate classification.31 Reliability studies of both examination and classification are warranted. This study focused only on the reliability of the motor and sensory exam and did not address the reliability of neurological classification. Future work will explore classification of neurological impairment in children and will incorporate recent changes related to the summation of the upper extremity motor scores independent from the motor scores obtained from the lower limb.41 Also, the current study evaluated the reliability of the summed motor and sensory scores and did not evaluate reliability of scores at each myotome and dermatome. Greater variability in individual myotomes and dermatomes would be expected, particularly at and around the level of injury.

While recruitment of babies with SCI was achievable, six subjects between 8 months and 3 years of age were simply too young to participate in the motor and/or sensory exam and hence, were ‘not testable.’ Therefore, while attempted, reliability was not examined in children less than 4 years of age. Despite the obvious limitation of not having data from very young infants and toddlers for reliability analysis, a primary purpose of the study, determining the lower age limit for the ISCSCI, was accomplished.

The primary author has extensive experience in the examination of motor and sensory function using the international standards and may not represent the entire tester population. Therefore, the results of this study may not represent the results of other testers, especially those with less experience with the ISCSCI and without expertise in the evaluation of patients with tetraplegia. In addition to the researcher's experience with the exam, the majority of the subjects in this study have participated in repeated ISCSCI motor and sensory exams as part of their clinical care and thus have extensive experience with the test. While it is recommended that research involving reliability testing enroll subjects without previous experience in the test,38 the only individuals who have not been introduced to the ISCSCI are those with new injuries whose motor and sensory status is not stable and therefore, reproducibility would not be necessarily anticipated.

Similar to many test–retest situations, the selection of the second administration point is a limitation. For reliability testing, it is recommended to allow sufficient time between trials to minimize subject recall but short enough time to avoid any real change.39 Nunnally and Bernstein38 recommends 2–4 weeks between trials. In this study, there was a 24- to 48-h time interval between the first and second trial. This time interval was chosen because children with chronic SCI typically are admitted to the rehabilitation unit for a 2- to 3-day stay for annual evaluations. Since the subjects in this study travel significant distances to the rehabilitation hospital, it would be unrealistic to expect them to return within a 2-week period. In addition to the travel logistics, loss of work time for parents and school time for subjects would have been major barriers to subject recruitment.

The study generated several new questions and areas of inquiry. While classification was not a focus of this study, throughout the study as a basis for discussion with subjects and parents and to accurately document in the medical records, neurological level was classified using the summed motor and sensory scores from each trial. Paradoxically, despite the high ICC for TM, PP and LT scores, 28 subjects (38%) with complete injuries had a change in the highest neurological level between the first and second exam. This finding is interesting and speaks to the relationship between the reliability of the summed scores and the reliability of individual scores at each myotome and dermatome level. Future work will explore this relationship in children.

Conclusion

This study demonstrated that the ISCSCI motor and sensory exams most likely do not have utility in children less than 4 years of age and hence, may not be clinically appropriate methods to determine neurological consequence of SCI in infants and toddlers. Children injured at a young age and those with little or no prior experience with volitional bowel movements had difficulty with the anal motor exam. Also, despite completing the sensory exam, children under 10 years of age were anxious and stressed by the PP exam. While ICC, for the majority of the sample, met or exceeded recommended values for clinical tests, 95% CI were at times wide, suggesting poor precision. While type and severity of injury did not appear to influence reliability for the entire sample, there were two few subjects to explore reliability based on age, type of injury and severity of injury. Additional studies with a larger pool of subjects with incomplete and complete paraplegia and tetraplegia in each age group is needed. Also, modifications of the exam for children under 10 years may be warranted. It is recommended that the international standards committee, a committee established to provide expert and ongoing review of the ICSCI, incorporate pediatric considerations and modifications into future revisions of the ISCSCI by using findings from this study and from future studies that will follow.