Inter-assessor reliability of practice based biomechanical assessment of the foot and ankle

Twelve podiatrists (working in state funded and private health care settings, six male, mean age 42) specialising in foot and ankle biomechanics were invited to participate. All worked within a specialist biomechanics/musculoskeletal clinic and had at least 3 years clinical experience at this specialist level. Ethical approval was granted (University of Salford Institutional Committee) and all participants gave written consent. A Delphi method [22] was chosen to derive consensus on a foot biomechanics assessment protocol. The Delphi method [22] is a systematic and structured examination technique involving a panel of experts. The method combines use of questionnaires and group discussion to derive consensus [22].

There were three keys phases to the development of a consensus.

Phase 1: Questionnaire. All podiatrists answered a questionnaire (Additional file 1) anonymously and without discussion. The questionnaire (written by HJ and PB) investigated the use of static foot, leg and lower limb biomechanical examinations and gait analysis protocols by each podiatrist. Questions were derived from Root et al. [3], current undergraduate syllabus, information from Valmassey [23] and Michaud [24]. There was also space provided for podiatrists to report any additional examinations used.

Most questions required Yes/No answers and required information on how often each examination was used, the method and whether the information was used to classify foot type and/or to develop a treatment rationale.

Phase 2: Development of draft consensus from results of the questionnaire. From the completed questionnaires, PB and HJ identified where there was both agreement and disagreement amongst the expert panel. Agreement existed when there was an identifiable trend amongst podiatrists, for example the majority of podiatrists used the same measurement technique. Disagreement was where there was poor consensus between podiatrists, for example less than half used a particular examination. A separate adjudicator (CN) was present throughout. A draft assessment protocol was developed based on the questionnaire responses.

Phase 3: Group discussion. A group discussion (led by PB, HJ took notes) explored the validity of the questionnaire results and draft foot assessment protocol from Phase 2. Discussion orientated around whether it was true reflection of the current practice of the panel members but also related professional disciplines. The areas of agreement and disagreement from the questionnaire results were elaborated upon though open discussion. Podiatrists explained in more detail their assessment methodology, their conceptual understanding of the normative basis to which pathological cases are compared and the rationale for their assessment plan.

Eleven podiatrists (working in state funded and private health care settings, six male, mean age 46) specialising in foot and ankle biomechanics practice volunteered to participate. All worked within a specialist biomechanics/musculoskeletal clinic and had at least 5 years clinical experience at this specialist level.

Each podiatrist assessed six asymptomatic participants (three male, mean age 25, mean body mass index [BMI] 23), using a subset of the assessment protocol defined in Part 1 of the study. Ethical approval was granted from the University of Salford Institutional Committee and all participants gave written consent. This investigation was conducted nine months after Part 1.

Four of the eight biomechanical examination procedures identified in Part 1 were selected for the inter-assessor reliability study. These were selected primarily because the podiatrists identified them as essential rather than optional components of their clinical assessment. However, they also provided some assessment of the lower limb as well as the foot and could be completed within a reasonable time frame. The four assessments selected by podiatrists were used for all or the majority of patients and provided information critical to the development of treatment rationale and orthotic prescription. Thus the four selected contributed more to clinical practice than the four assessments omitted.

The assessments used in the inter-assessor study were 1) NCSP and RCSP, 2) ankle joint range of dorsiflexion, 3) first ray mobility and position, and 4) limb length examination. They were assessed quantitatively or qualitatively according to the preferences identified in Part 1.To help maintain consistency in how the 11 podiatrists implemented the assessment protocol, an information sheet and demonstration was provided. The participants whose feet were to be assessed were placed in six separate cubicles at the University clinic. Assessments were conducted as per the protocols described in Table 1 and podiatrists were allocated 30 minutes to assess each participant and at least 30 minutes rest between each assessment. In accordance with clinical practice, each assessment was completed once for each foot. Each podiatrist recorded their assessments in a booklet. No discussion was allowed between podiatrists or participants during the assessments. All pen marker lines on the participants were removed between podiatrists.

The researchers were blind to the data in each booklet. All data was collated into Microsoft Excel and then processed through Statistical Package Social Science Software (Version 17.0) (SPSS, Chicago, Illinois, USA). The mean, range, standard deviation (SD) and 95 % confidence intervals (95 % CI) were calculated for NCSP, RCSP and the range of ankle joint dorsiflexion.

Inter-assessor reliability for RCSP, NCSP and the range of ankle joint dorsiflexion were calculated using ICC (2,1) in accordance with Rankin and Stokes [25]. ICC values were chosen as they assess the consistency of quantitative measurements made by multiple testers (clinicians) measuring the same objects (participants) [7]. Bruton et al. [7] suggest that ICC values should not be interpreted clinically in isolation. Therefore a random effects ANOVA (analysis of variance, crossed random effects model) [26] was used to enable further evaluation of reliability. A random effects ANOVA models y as a constant, plus a random effect due to the assessor (clinician), a random effect due to the participant (e.g. moved their feet) and an overall random error of the examination itself. (E ¹ assessor error, E ² participant error, E random error).

This calculates the extent of between participant variability, between assessor variability and the amount of random error in the examination. This provides an indication of where the majority of error occurs. Therefore for each part of the assessment (e.g. NCSP, RCSP), the error variables have to be accounted for in addition to the true value of the feature being assessed:

Value provided by the assessor = actual value + (assessor error (E ¹) + participant error (E ² ) + random error (E)).

A particular advantage of the random effects ANOVA is that the outcomes are expressed in the same units as the measurement and thus are easily interpreted in terms of clinical practice. In addition, the three sources of error can be combined to provide an indication of the total error due to participant, assessor and random error:

Total error = √(assessor error (E ¹) + participant error (E ² ) + random error (E).

The assessment of first ray mobility and position and limb length involved categorical data, therefore the percentage agreement (%) [27] and a Fleiss Kappa [28, 29] were chosen.

Percentage agreement can lack sensitivity as it does not adjust for that agreement occurring by chance [27]. A Fleiss Kappa calculates the reliability of agreement between a fixed number of assessors [28‐30] and is a better representation of true inter-assessor reliability [27]. Fleiss Kappa values range from <0 for poor agreement to 1.00 for perfect agreement [28, 30, 31]. Both of these statistical measures are consistent with the available literature [17, 18].

Tables 1 and 2 represent the results of the questionnaire and the group discussion. Three key trends were derived from the questionnaire (Phases 1 and 2) and formed the basis to the subsequent discussion (Phase 3). These were:

The biomechanical assessment protocol identified through consensus comprised the following:

The results indicate poor inter-assessor reliability for the four examinations. Table 3 displays the reliability results for RCSP and NCSP. For RCSP an ICC of 0.23 (right), 0.14 (left) and 0.14 (right) and 0.11 (left) for NCSP suggest poor inter-assessor reliability. All mean 95 % CI were above 3.7° and the mean range of NCSP and RCSP values were greater than 8.8° (Table 4). The results of the random effects ANOVA indicate that the greatest error was random error (up to 4.9°), while the assessor error was up to 3.4°.

Table 3 demonstrates ICC values for the examination of the range of ankle joint dorsiflexion. There was moderate agreement with 0.44 (right) and 0.42 (left) for knee extended and 0.61 (right) and 0.51 (left) for knee flexed. All mean 95 % CI were above 9.0°, and the mean range of ankle dorsiflexion values was greater than 20.5° (Table 5). The results of the random effects ANOVA indicate that there were comparable contributions from the three sources of error, with values ranging from 4.3° to 5.8°.

The results for classification of first ray position and mobility are displayed in Table 3 and Table 6. There was greater consistency for the categorisation of mobility compared to first ray position. Fleiss Kappa values of −0.03 (right foot) and 0.01 (left foot) for categorisation of position and for the range of first ray motion (0.05 (right) and −0.01 (left)).

Table 3 and 7 demonstrates the results for examination of limb length. There was less agreement on the size of the difference in limb length than the identification of the longer limb when evaluating the percentage agreement values, however results were comparable according to Fleiss Kappa values (0.02 for both longer leg and the difference in leg length). Clinicians consistently reported differences in limb length of 5 mm or less (Table 3 and 7).

The assessment protocol developed in Part 1 of this investigation is largely a modified version of Root et al. [3]. The description provided by Root et al. [3, 4] is still very much at the forefront of clinical assessment of foot biomechanics and the basis for clinical descriptors of foot function during gait. This demonstrates the continued influence of Root et al. [3, 4] and the strong effect undergraduate education has on subsequent practice. However, the inclusion of the Foot Posture Index [31] and use of visual gait assessment signify that podiatrists have adopted new assessment approaches that they deem to add value. This did not extend as far as the use of potentially valuable instrumented gait assessment methods (For example video analysis, pressure plate).

Contrary to the specific instructions of Root et al. [3], podiatrists choose to estimate and classify joint position/motion rather than ascertain a directly measured numerical value. For example, when assessing the ankle joint, podiatrists choose to estimate the range of dorsiflexion rather than use a goniometer. Podiatrists felt that their experience was sufficient to accurately classify the range of motion as normal, excessive or restricted. All podiatrists stated that they were confident this approach was valid and cited time constraints as the primary barrier to use of objective measures. However, continuing to use assessments that have been shown to have low reliability is likely to be considered unsound practice. If reliability could be improved by an objective rather than subjective assessment, even if it takes longer to complete, then this could form a strong case to extend the time available for the assessment of patients.

These differentiations from the original description and instructions of Root et al. [3, 4] justify the consensus exercise in Part 1 and ensure that our investigation of inter-assessor reliability is relevant to current clinical practice.

There was poor inter-assessor reliability recorded for all of the static biomechanical examinations of the foot, leg and lower limb which questions their value in clinical practice. RCSP and NCSP produced poor inter-assessor reliability results (all ICC values were less than 0.23), and this concurs with the available literature. Picciano et al. [32] recorded ICC values for NCSP of 0.15 and 95 % confidence intervals of 0.87° to 8.65°. The results of the random effects ANOVA suggest that for RCSP and NCSP random error is the key issue. Differences between assessors of this scale would create different treatment plans and orthotic designs. Both Menz [8] and Elveru et al. [16] highlight that an overwhelming priority is placed upon the outcomes of these measurements in clinical assessment and orthotic prescription. However, the poor reliability and large variation in the results recorded here and elsewhere [8, 9, 12, 16] should be clinically unacceptable and we therefore question their continued use in clinical practice [8, 15, 16].

Although podiatrists reported some difficulty in using the goniometer [15], moderate reliability was observed for the examination of the range of ankle joint dorsiflexion. Elveru et al. [16] and Jonson and Gross [1] report similar ICC values of 0.50 and 0.65. In Part 1 of this study all podiatrists stated that they believed the examination of ankle joint dorsiflexion provided a good indication of dynamic foot function. However, the low reliability and large range of values recorded across assessors questions the clinical value of these examinations. Considering that 10° of dorsiflexion was stated as normal (results from Part 1, based on Root et al. [3], Table 2), clinical measures at either boundary of the 95 % CI (maximum 95 % CI were 5.6° to 15.5°) and the total error of up to 10.7°, could lead to false identification of the actual range of ankle dorsiflexion. This would directly affect the treatment rationale if the outcome suggested limited or adequate range of ankle motion. Moseley and Adams [33] suggest that such variation would make measurement of changes in range of motion due to interventions (e.g. stretching) unreliable. The results from the random effects ANOVA suggest that all three sources of error contribute to variation between assessors. Since random error was quite large (5.2°, left foot, knee flexed), reducing errors from participants and assessors (e.g. through training, use of measurement tools) might still not achieve an acceptable level of reliability.

Classification of first ray mobility demonstrated greater reliability than categorisation of first ray position. The Fleiss Kappa values of less than 0.05 for categorisation of first ray position and range of motion indicate only poor to slight agreement [28, 29]. For four of the 12 feet assessed there was greater than 90 % agreement for classification of first ray range of motion as flexible. However, percentage agreement can lack sensitivity as to the true level of agreement between assessors as it can over or under estimate the actual level of agreement and does not account for the possibility that the agreement observed occurred by chance [27]. High levels of agreement for assessment of flexibility might be expected as ‘rigidity’ suggests no motion at all and this is more easily identified than different grades of “some” motion [24]. However, taking into account the Fleiss Kappa and percentage agreement statistical values only poor to moderate reliability was observed. Classification of first ray position demonstrated poor agreement between assessors. There are significant identifiable differences between a plantarflexed and dorsiflexed first ray [24], something that experienced podiatrists would expect themselves to be able to identify. As with measures of rearfoot alignment, first ray position can influence orthotic prescription [24].

Identification of the longer limb provided marginally better agreement than classifying the actual amount of leg length difference, but still only suggests slight agreement [28, 29] with Fleiss Kappa values of 0.02 (longer leg) and 0.02 (difference in leg length). This level of reliability is similar to Woerman and Binder-MacLeod [21]. To be able to ascertain that there is a difference in limb length of less than 5 mm requires high precision and it is doubtful that through visual inspection and palpation a clinician could reliably work to such accuracy. If a clinician can identify a discrepancy this small then they will almost always identify a limb length difference because the skeleton is rarely truly symmetrical.

One purpose of clinical assessment is to decipher normal from pathological [1, 2, 23, 24] but the results from this investigation suggest that it would not be possible to accurately classify either. The protocol described by Root et al. [3] states precise measurements are required when undertaking a static biomechanical assessment of the foot. Results from this and prior research [11, 12, 16, 32] suggest that such accuracy is not achieved in clinical practice. For example, Root et al. [3, 4] states that RCSP and NCSP measurements will precisely dictate the inclination of a rearfoot wedge used in a foot orthoses. However the variability in the assessment of rearfoot position reported here would lead to very different orthotic prescriptions. This directly undermines the biomechanical rationale for intricate adjustments in the design of foot orthoses and the capture of static foot shape as a basis for foot orthosis design. This has profound implications for many areas of clinical practice and suggests a reappraisal of the theoretical and practical basis for orthotic practice is warranted. The low reliability of the assessments evaluated here questions their ability to accurately infer the behaviour of the foot during stance, which is the purpose of the static assessments in the model proposed by Root et al. [4]. Indeed, research investigating the validity of Root et al. [3, 4] is currently being undertaken by the authors. The results here also add weight to the case for a move toward objective assessment of dynamic foot behaviour in clinical practice, regardless of the practical challenges this raises.

There are several limitations to the work reported here. Four of the eight examinations used by assessors (from Part 1) were not included in Part 2 of this study. They were excluded because the podiatrists we worked with identified them as ‘optional’. Other clinicians might disagree with the ranking of the eight assessments, especially if their practice is different to that of the podiatrists involved in this current study. Using all eight examinations would have been logistically difficult with the number of assessors and participants in this study and time available for the assessments. The number of assessors used was relatively small and might not represent the true variation across the entire professional communities using the assessments evaluated in this work. All were podiatrists and whilst their professional networks are strongly multi-professional, practices could differ in other disciplines and countries. The literature indicates that the measures used by the assessors and those evaluated in the reliability study, are also used in the physical therapy profession [1, 16, 20]. The development of the foot assessment protocol occurred through just one iteration of the Delphi method, whereas two or more iterations are often employed [24]. Experience during the exercise suggested that consensus was already in place or very close from the outset. The number of feet assessed was quite small and all participants were free from pathology. The participants were young with an average BMI and may not represent feet that present in many clinical cases. Arguably, assessing these feet is easier than those of people in pain, feet with deformity or in cases of greater BMI, and thus our results might reflect a “best case” scenario in terms of reliability. This study recorded low ICC values, in particular for NCSP and RCSP. The large number of assessors and small number of participants would have increased the variability and therefore could have decreased the inter-assessor reliability. Finally, good reliability does not infer practical usefulness of the assessment. Good reliability may simply reflect low sensitivity and specificity in the measure, or highly repeatable errors by assessors. Thus, good reliability does not infer validity. However, measures cannot be valid unless reliable, and outcomes of this work indicate many of the assessments used in foot health practice are unreliable and thus invalid.