Skip to main content
Top
Gepubliceerd in: Journal of Foot and Ankle Research 1/2014

Open Access 01-12-2014 | Research

The Queensland high risk foot form (QHRFF) – is it a reliable and valid clinical research tool for foot disease?

Auteurs: Peter A Lazzarini, Vanessa Ng, Ewan M Kinnear, Maarten C Kamp, Suzanne S Kuys, Cameron Hurst, Lloyd F Reed

Gepubliceerd in: Journal of Foot and Ankle Research | Uitgave 1/2014

Abstract

Background

Foot disease complications, such as foot ulcers and infection, contribute to considerable morbidity and mortality. These complications are typically precipitated by “high-risk factors”, such as peripheral neuropathy and peripheral arterial disease. High-risk factors are more prevalent in specific “at risk” populations such as diabetes, kidney disease and cardiovascular disease. To the best of the authors’ knowledge a tool capturing multiple high-risk factors and foot disease complications in multiple at risk populations has yet to be tested. This study aimed to develop and test the validity and reliability of a Queensland High Risk Foot Form (QHRFF) tool.

Methods

The study was conducted in two phases. Phase one developed a QHRFF using an existing diabetes foot disease tool, literature searches, stakeholder groups and expert panel. Phase two tested the QHRFF for validity and reliability. Four clinicians, representing different levels of expertise, were recruited to test validity and reliability. Three cohorts of patients were recruited; one tested criterion measure reliability (n = 32), another tested criterion validity and inter-rater reliability (n = 43), and another tested intra-rater reliability (n = 19). Validity was determined using sensitivity, specificity and positive predictive values (PPV). Reliability was determined using Kappa, weighted Kappa and intra-class correlation (ICC) statistics.

Results

A QHRFF tool containing 46 items across seven domains was developed. Criterion measure reliability of at least moderate categories of agreement (Kappa > 0.4; ICC > 0.75) was seen in 91% (29 of 32) tested items. Criterion validity of at least moderate categories (PPV > 0.7) was seen in 83% (60 of 72) tested items. Inter- and intra-rater reliability of at least moderate categories (Kappa > 0.4; ICC > 0.75) was seen in 88% (84 of 96) and 87% (20 of 23) tested items respectively.

Conclusions

The QHRFF had acceptable validity and reliability across the majority of items; particularly items identifying relevant co-morbidities, high-risk factors and foot disease complications. Recommendations have been made to improve or remove identified weaker items for future QHRFF versions. Overall, the QHRFF possesses suitable practicality, validity and reliability to assess and capture relevant foot disease items across multiple at risk populations.
Opmerkingen

Electronic supplementary material

The online version of this article (doi:10.​1186/​1757-1146-7-7) contains supplementary material, which is available to authorized users.

Competing interests

The authors have no relevant conflict of interest to disclose.

Authors’ contributions

PAL conceived, designed, researched data, contributed to discussion, wrote and reviewed/edited the manuscript. VN provided research assistance, researched data, contributed to discussion and reviewed/edited the manuscript. EMK conceived, contributed to discussion and reviewed/edited the manuscript. MSK, SSK and LR designed, contributed to discussion, and reviewed/edited the manuscript. CH designed, researched data and reviewed/edited the manuscript. All authors read and approved the final manuscript.
Afkortingen
ANOVA
Analysis of variance
BGL
Blood glucose levels
CINAHL
Cumulative index to nursing and allied health literature
CKD
Chronic kidney disease
DFF
Diabetic foot form
ESRF
End stage renal failure
GP
General practitioner
HRFS
High risk foot service
ICC
Intra-class correlation
K
Kappa
wK
Weighted Kappa
MEDLINE
Medical literature analysis and retrieval system online
PPV
Positive predictive value
QHRFF
Queensland high risk foot form
SD
Standard deviation
SE
Standard error
UTWCS
University of Texas wound classification system.

Background

Foot disease contributes to considerable hospitalisation [14], amputation [58], institutionalisation [9, 10], and death [2, 11]; yet foot disease complications and these outcomes are largely preventable [5, 6, 12, 13]. “High risk factors” (such as peripheral neuropathy, peripheral arterial disease or foot deformity) significantly increase the risk of developing “foot disease complications” (such as foot ulcers, infection or ischaemia) [1, 1416]. Diabetes populations are frequently acknowledged as the leading “at risk population” for foot disease due to the increased risk of developing high risk factors, and in turn foot disease complications, from diabetes [58, 17]. However, other chronic disease populations (such as chronic kidney disease [15, 16], cardiovascular disease [5, 6, 18, 19] and some cancers [5, 6, 8]) have now been shown to cause comparable rates of high risk factors and foot disease complications to diabetes, and thus, are also becoming more readily identified as “at risk populations” for foot disease.
Best practice foot disease management has been shown to significantly reduce hospitalisation, amputation, mortality and overall costs within different at risk populations [5, 12, 13, 20]. These multi-faceted best practice interventions commonly include screening for high-risk factors, multi-disciplinary management of foot disease complications, clinical training, evidence-based clinical pathway utilisation and regular capture and analysis of foot disease clinical data [5, 6, 12, 13, 20]. In consideration of the growing problem of foot disease, and the potential future improvements observed with best practice clinical management and research, it is imperative that any clinical tools to assess, capture, measure or analyse patient outcomes in at risk populations are valid and reliable.
There are a multitude of studies that have investigated a specific high risk foot factor or foot disease complication within multiple at risk populations [3, 5, 6, 21]. Furthermore, numerous studies have investigated multiple high risk factors and foot disease complications in specific at risk populations such as diabetes [2, 15, 17]. However, very few studies have investigated multiple high risk foot factors and foot disease complications in multiple at risk populations. This situation appears to be mirrored by the clinical tools available to measure and report on foot disease. For example many validated single-item tools are available which measure specific high-risk factors or foot disease complications within multiple at risk populations [2224]; such as the ankle brachial index to capture and measure peripheral arterial disease within various at risk populations [22]. Furthermore, many validated multi-item tools measuring multiple high-risk factors and foot disease complications in specific at risk populations have been developed [15, 2528]; such as the University of Texas Diabetic Foot Classification System to capture and measure multiple high risk factors and foot disease complications in people with diabetes only [25]. However, to the best of the authors’ knowledge a multi-item tool designed to measure multiple high-risk factors and foot disease complications in multiple at risk populations is yet to be developed and tested for validity and reliability.
Recently a Diabetic Foot Form (DFF) tool was developed to measure multiple high-risk factors and foot disease complications in people with diabetes only in diverse Queensland (Australia) settings [29]. The DFF was a multi-item tool developed from a number of existing clinical tools recommended in the literature to reliably capture various high-risk factors and foot disease complications in the diabetes population [29]. The implementation of the DFF in diverse Queensland clinical sites, in conjunction with other multi-faceted strategies, resulted in improved capture, measurement and management of high risk factors and foot disease complications and a corresponding reduction in outcomes such as hospitalisation and amputation [29, 30]. However, the tool was not tested for validity or reliability and was designed only to measure multiple high-risk factors and foot disease complications in the specific at risk population of people with diabetes.
In this study, we sought to modify the DFF tool to enable the measurement of multiple high risk factors and foot disease complications in multiple at risk populations. It was intended that the new multi-item tool would align with best practice principles for clinical tools including being easily interpreted, practical to use and possessing high face, content and criterion validity, and inter- and intra-rater reliability [3133]. Thus, the aims of this study were firstly to develop a multi-item Queensland High Risk Foot Form (QHRFF) tool to capture multiple high risk factors and foot disease complications in multiple applicable at risk populations, and secondly, to investigate the validity and reliability of the tool’s individual items when used by clinicians representing different levels of foot disease expertise.

Methods

The study was conducted in two phases. Phase one involved development of the Queensland High Risk Foot Form (QHRFF) tool and phase two tested the validity and reliability of the QHRFF tool. Approval was granted from Institutional Ethics Committees and informed consent was obtained from all individual participants (patients and clinicians) for this study.

Phase one – development of the tool

Phase one primarily aimed to select items for the development of a QHRFF tool that were practical to collect in an Australian clinical setting, applicable to multiple 'at risk’ populations, and provided high face and content validity. The overarching procedures used for item selection included using the original DFF [29] as the starting tool to build the new QHRFF, searching the electronic literature for other recommended foot disease tools, establishing an expert panel to guide development and using several rounds of stakeholder consultation to refine the tool.
The original DFF contained 64 items pertaining to the construct of foot disease in diabetes populations [29] and was used as the starting tool to modify into the QHRFF. The DFF was originally developed using similar procedures to those used in this study and appeared to possess high practicality, face and content validity [29]. Furthermore, the DFF had been routinely used in over 25 High Risk Foot Service sites throughout Queensland for three years to collect standard clinical data on diabetes-related foot disease [30].
An electronic literature search was undertaken of relevant electronic databases including MEDLINE (all years to June 2011), CINAHL (all years to June 2011) and relevant professional websites to identify existing foot disease-related tools. The basic terms searched included tools to identify peripheral neuropathy, peripheral arterial disease, foot deformity, foot ulceration, foot infection and amputation or synonyms.
An expert panel was established from members of the Queensland Statewide Diabetes Foot Working Group; a working group of the Queensland Statewide Diabetes Clinical Network. The panel comprised 14 expert clinicians, with between five and 25 years of experience in the area of foot disease management, from the fields of endocrinology, vascular surgery, podiatry, nursing, public health, quality improvement and research. The panel had the responsibility to decide upon the final items for the QHRFF tool after each round of consultation.
Refinements of the tool were achieved through numerous rounds of consultation and trialling of draft QHRFF versions with external stakeholders in relation to practicality, face and content validity. Stakeholders included up to 200 multidisciplinary professionals from sites registered to use the existing DFF, members of the Queensland Health Statewide Diabetes Clinical Network, Statewide Renal Clinical Network and Statewide Podiatry Network.
At the conclusion of phase one the expert panel and aforementioned network’s management committees determined, via consensus endorsement, that each item of the QHRFF possessed high practicality, face and content validity, and was applicable across multiple 'at risk’ populations (particularly diabetes, cardiovascular disease and chronic kidney disease) in Australian clinical settings. See Phase One Results for further details regarding the final QHRFF.

Phase two – validity and reliability

The final endorsed QHRFF tool developed in phase one was then tested for validity and reliability. The general procedure for testing involved using podiatrists with different levels of foot disease expertise to assess patients with different levels and severity of high risk factors and foot disease. The authors decided to test only QHRFF items that directly related to the foot disease construct, thus, excluding items such as patients’ name, and facility. To test the validity of each item a general criterion measure (the agreement between two 'experts’) was initially assessed for reliability. Each item was then tested, using the podiatrists with different experience levels, against the criterion measure to evaluate magnitudes of concurrent criterion validity. The inter- and intra-rater reliability was also tested for the level of agreement on each item by podiatrists with different levels of experience.

Setting and participants

The testing was conducted within the High Risk Foot Services (HRFS), Metro North Hospital and Health Service, Brisbane, Australia. Eligible clinician participants were recruited from podiatrists practicing a minimum of one session per week in a HRFS. Four podiatrists were chosen as representative of the majority of clinicians managing foot disease within Queensland Health, and thus, potentially using the QHRFF in future. For the purposes of the study, levels of expertise were categorised using the Queensland Health 'Health Practitioner award’ [34]. Thus, expert clinicians were defined as either a 'consultant clinician’ (level 6) or 'specialist clinician’ (level 5) [34] and working in an acute hospital setting, plus, a member of the expert panel to ensure they understood the original intended construct for the QHRFF tool. General clinicians were defined as a 'senior clinician’ (level 4) or 'clinician’ (level 3) [34], working in a community setting, and thought to be representative of the general podiatry clinical workforce managing foot disease in Queensland. One of each level was recruited; one level 6, one level 5, one level 4 and one level 3 (however, the level 3 had been acting in a level 4 position at the time of the study). Written informed consent was obtained from all participants (patients and clinicians) prior to commencement of their study participation.
Eligible patient participants were consecutively recruited from patients already attending Community HRFS clinics for the care of high risk factors or foot disease complications; defined as a previous or current foot ulcer. Exclusion criteria included patients with a cognitive deficit, signs or symptoms of a systemic infection, younger than 18 years of age, or unwilling to provide written consent to participate. The authors considered that patients with previous or current foot ulcers would ensure that the majority of high risk factors and foot disease items had the realistic possibility of being present or absent, and thus, could be suitably tested on each participant. Furthermore, it was thought this population should possess the moderate prevalence rates, yet unpredictable mixes and severities of each item, of different high risk factors and foot disease complications, that are suggested in the literature to improve statistical robustness for validity and reliability studies [31].
Three different patient cohorts were used; one cohort to test the reliability of the criterion measures (agreement between 'experts’) (n = 32), another cohort to simultaneously test the criterion validity (an 'expert’ diagnosis against general clinicians) and inter-rater reliability (n = 43), and the last cohort to test the intra-rater reliability of a general clinician (n = 19). The recruitment of consecutive community patients did mean that patients may have been familiar to the level 3 or level 4 clinicians; however, not to the expert 'gold standard’ clinicians working in the hospital setting. To minimise the risk of patients being familiar to the level 3 or level 4 clinicians’ seven different Community HRFS clinics were used to recruit patient participants. The level 3 and level 4 clinicians had only worked at two of the seven clinics recruiting patients.

Procedures

A designated research assistant coordinated all procedures. Training of each clinician consisted of being provided with a QHRFF manual that gave a definition of each item [35], a 1–2 hour training session on instructions and tips to use the QHRFF tool, and each was encouraged to trial the tool on their existing patients and clarify any queries with the research assistant prior to testing.
The general assessment procedure for each validity or reliability test consisted of patients having their feet examined by at least two different clinicians, blinded from each other’s assessment, within the one clinical visit (validity and inter-rater reliability). However, to ensure all clinicians had an equal opportunity to determine the patient’s debridement and wound management needs, all were permitted to visually inspect the patient’s feet together for up to five minutes prior to any clinician ratings. In this initial inspection period, the clinicians were instructed to only visually inspect the need for debridement and previous wound dressings whilst not conversing with or touching the patient or each other. The order of clinician assessment after this inspection period was then at the discretion of the research assistant based on clinician availability. The first clinician would have the additional task of debriding the wound or callus if they deemed necessary and the last clinician the additional task of completing any clinical management.
The research assistant ensured all clinicians were blinded from each other’s assessments in separate rooms and that all examination records were de-identified. Each clinician conducted the assessment of patient’s feet using the QHRFF to record their assessment and management recommendations. Demographic, medical history and medication information were available from the patients’ medical records or by direct communication with the patients themselves. To minimise assessment bias all historical foot-related records (including progress notes, DFF tools or QHRFFs tools) were removed from the medical record prior to each clinician’s assessment.
Criterion measure
The criterion measure was tested on the first cohort of patients. The criterion measure (i.e. 'gold standard’ diagnosis) for each QHRFF item in this study was defined by the authors as the diagnosis made by an expert clinician. Expert clinicians were pragmatically chosen as a general criterion measure for all items, instead of using multiple resource and time intensive individual gold standard-recognised criterion measures (such as nerve conduction studies for neuropathy), due to the resource and time constraints of the study. Best practice dictates that any criterion measure should be reliable, free from bias and measure the same item as the new tool [31]. Thus, the study’s criterion measure, of an expert clinician, required testing to determine its reliability, independence (free from bias) and applicability, prior to its use as a 'gold standard’ criterion measure in the criterion-related validity tests. The authors tested the criterion measure by testing the reliability of the agreement between two independent (blinded from one another) experts (one level 6 and one level 5) to determine the magnitude of reliability of agreement on their 'gold standard’ diagnoses for each item on the same patient cohort. The “general assessment procedure” as described above was utilised for each patient. Once the criterion measure for each item was determined to be reliable, the criterion validity of each QHRFF item was tested.
Criterion validity
The criterion validity and inter-rated reliability was then tested simultaneously on a second cohort of patients. Criterion validity was tested using the concurrent criterion validity method by comparing the reliable criterion measure (one of the expert clinicians) against each of the two representative 'general’ clinicians (one level 3 and one level 4). Thus, two separate criterion assessments were carried out for each item; one using the level 3 clinician against the criterion measure, and another the level 4 clinician against the criterion measure. Again the “general assessment procedure” as described above was utilised for each patient when testing for criterion validity.
Inter-rater reliability
Inter-rater reliability tests were performed simultaneously with the criterion validity tests on the same second cohort of patients and were tested using the expert clinician (level 5), senior clinician (level 4) and clinician (level 3). Thus, three inter-rater reliability measures of agreements were carried out for each item; one testing the agreement between the level 5 and level 4 clinician, a second between the level 5 and level 3 clinician, and a third between the level 4 and level 3 clinician. “General assessment procedures” as described above were again utilised for each patient when testing for inter-rater reliability.
Intra-rater reliability
Intra-rater reliability was tested on a third cohort of patients. The clinician with the least expertise (level 3) was used to test intra-rater reliability as it was hypothesised that the clinician with the least expertise would have the most variability of the tested clinicians. The two different time points to test intra-rater reliability were between one and four weeks apart. This time period was chosen as it was considered that a minimum of one week (of full clinical load) would be necessary to adequately reduce recall bias in a clinician, plus, a maximum of four weeks would not be sufficient time for the majority of items to markedly change and thus items would remain stable between ratings. Any items that did not fit this criteria were excluded from intra-rater testing. Furthermore, this time period aligned with any necessary follow up treatment time period for patients, and thus, was convenient to patient participants. To control for any potential changes in foot ulcer characteristics over time (for example ulcer combined surface area and clinical signs of infection) digital images were used. Digital photographic images taken of foot ulcers at the first rating were used at the second rating to standardise the foot ulcer characteristics across two time points [36]. Images were taken perpendicular and 30-60 cm away from ulcers at the first rating, and incorporated two calibrated measures across the length and width of the ulcer in the image. These images were reviewed by the clinician at the second rating to determine the foot ulcer characteristics only.

Statistical analysis

All data was analysed using SPSS 19.0 for Windows (SPSS Inc., Chicago, IL, USA) or GraphPad Software. Descriptive statistics were used to display the single demographic, medical co-morbidity, high-risk factors and foot disease variables for each cohort of patient participants; using means and standard deviations (SD) for continuous variables (which were normally distributed) or proportions for categorical variables. Each above descriptive cohort variable was collected from the clinician with most expertise or on the first rating of the intra-rater reliability cohort. Chi-squared test of independence and ANOVA were used to test for differences between the three cohorts’ characteristics. A significance level of p < 0.05 was used throughout.
All QHRFF items were tested for validity and reliability; except those stated items considered not to impact on the foot disease construct. Criterion validity was tested by calculating sensitivity, specificity and positive predictive values. Ordinal variables were collapsed into dichotomous data to enable calculations of sensitivity, specificity and positive predictive values. All measures of agreements between clinicians were tested using Kappa (K) for dichotomous variables, weighted Kappa (wK) for ordinal variables and intra-class correlation (ICCs) (model 2, 1) for continuous variables [31, 37, 38]. Kappa and wK value (Standard Errors (SE)) strengths were categorised as: no agreement < 0; slight agreement = 0 – 0.20; fair agreement = 0.21 - 0.40; moderate agreement = 0.41 – 0.6; substantial agreement = 0.61 – 0.8; and near perfect agreement = 0.81 – 1.0 [31, 37, 38]. ICC (SD) strengths were categorised as: weak-moderate agreement < 0.75 and strong agreement > 0.75 [31].

Results

Phase one – development of the tool

The literature search identified 174 papers that reported on tools to identify or measure foot disease complications and/or high-risk factors, with the majority of papers specifically relating to diabetes populations (90 (52%)). Tools reported in other populations were in conditions commonly identified in the literature to be associated with lower limb amputation [58]. These included cardiovascular disease (peripheral arterial disease), chronic kidney disease, malignancy, infection and other neurological conditions [58]. No tool was identified that was specifically designed to identify multiple high-risk factors and foot disease complications in multiple at risk populations.
Fifty-nine individual tools were identified from the literature search; 23 were considered to have adequate practicality, face validity and applicability to an Australian clinical context to be considered for the QHRFF. These tools and the original DFF tool [29] were considered by the expert panel for QHRFF item selection. At this point, the expert panel determined that the tool should be divided into a clinical assessment record section that informed a separate data collection section, and thus, only the data collection section would require testing.
Overall 87 items were initially identified from the considered tools. At the conclusion of phase one procedures the final endorsed QHRFF data collection tool was made up of 46-items (excluding general identification items) covering seven domains (Figure 1). The seven content domains included identifying general demographics, different health professionals attending, medical co-morbidity history, high-risk factor history, clinical diagnosis of high-risk factors, clinical diagnoses of foot disease complications, and clinical management principles performed. A QHRFF manual was developed to provide definitions and instructions on each item contained in the tool [35]. Table 1 outlines the tools or literature used to support each QHRFF item. At the conclusion of phase one, the expert panel concluded that the QHRFF tool’s items had the required high clinical practicality, applicability to multiple at risk populations, and good face and content validity. Subsequently, the tool was endorsed for use by the Queensland Health Statewide Diabetes, Renal and Podiatry Clinical Networks. Thus, the tool was ready for validity and reliability testing.
Table 1
Supporting tools or literature for QHRFF items
QHRFF item
Supporting tool or literature
General demographics*
[35]
Indigenous status
[14]
Health professionals attending*
[14, 39]
Medical co-morbidity history*
[40, 41]
Medical (diabetes) history
[58]
Diabetes year diagnosed (duration)
[17, 42]
Recent BGLs > 15 mmol/L
[41, 42]
HbA1c result
[41, 42]
CKD
[41, 43]
ESRF
[41, 43]
High-risk factor history*
[14, 35, 39]
Clinical diagnoses of high-risk factors
Neuropathy
Monofilament test [1, 14, 23, 39, 44, 45]
PAD
Foot pulses, ankle brachial index and/or toe systolic pressure [1, 14, 21, 22, 39, 4649]
Acute Charcot
[14, 39]
Foot deformity
Six-point foot deformity scale [14, 50]
Risk classification
[14, 39, 51, 52]
Clinical diagnoses of foot disease
(Ulcer) type
[14, 53, 54]
Combined surface area mm2
[5456]
Combined surface area (change since last visit)
[5559]
Clinical signs of infection
[1, 54, 60, 61]
UTWCS grade
UTWCS tool [14, 62, 63]
Ulcer depth#
[1, 14, 54]
Clinical management principles performed*
[14, 39]
*All items in this Domain cite the same references, unless otherwise stated.
#Ulcer depth was not specifically an item recorded on the QHRFF, but can be directly extrapolated from the UTWCS Grade item.

Phase two – validity and reliability

Forty items were tested for validity and reliability unless otherwise stated. The items not tested were those considered not to impact on the foot disease construct; i.e. 'facility’, 'visit type’, 'todays visit to HRFS’, 'separation status’, and 'date of referral’. Table 2 displays the general demographic, medical co-morbidity history, high-risk factor history, clinical diagnoses of high-risk factors and foot disease complications variable prevalence for the three patient cohorts used. No significant differences were noted for these variables, except for diabetes duration and any other co-morbidity (p < 0.05). All cohorts contained variables with moderate prevalence rates (> 15%) [31], except for different health professionals attending previously, co-morbidity of end stage renal failure (ESRF), acute Charcot and ischaemic ulcers.
Table 2
Demographic, co-morbidity and high-risk foot complication data for each patient cohort (n (%) unless otherwise stated)
 
Criterion measure
Criterion validity* & inter-rater reliability*
Intra-rater reliability
p Value
General demographics
    
Numbers
32
43
19
 
Male
25 (78%)
37 (86%)
17 (90%)
0.501
Age, mean (SD) (years)#
69 (13)
68 (13)
70 (15)
0.826
Age range (years)
48 – 90
36 – 89
42 – 90
 
Indigenous
1 (3%)
0 (0%)
1 (5%)
NA
Health professionals attending (in the past week)
Podiatrist
32 (100%)
43 (100%)
19 (100%)
1.000
GP
2 (6%)
10 (23%)
2 (11%)
0.103
Nurse
3 (9%)
12 (28%)
4 (21%)
0.141
Orthotist
0
0
0
NA
Physician
2 (6%)
1 (2%)
1 (5%)
NA
Surgeon
1 (3%)
3 (7%)
0
NA
Other
0
0
0
NA
Medical co-morbidity history
    
Diabetes (type 1 or 2)
28 (88%)
35 (81%)
16 (84%)
0.384
Type 2 diabetes
28 (88%)
31 (72%)
14 (74%)
0.256
Diabetes duration (years)#
12 (7)
23 (11)
20 (14)
0.001**
Recent BGLs > 15 mmol/L
5 (16%)
4 (9%)
5 (26%)
0.220
HbA1c (SD)#
9.2 (3.5)
8.1 (1.6)
8.4 (2.7)
0.450
Hypertension
28 (88%)
30 (70%)
11 (58%)
0.053
Dyslipidaemia
23 (72%)
25 (58%)
9 (47%)
0.201
Smoker
5 (16%)
11 (26%)
3 (16%)
0.492
CVD
21 (66%)
21 (49%)
9 (47%)
0.281
CKD
5 (16%)
13 (30%)
5 (26%)
0.339
ESRF
0
1 (2%)
0
NA
Other (non-listed)
29 (91%)
36 (84%)
11 (58%)
0.013**
High-risk history
  
Neuropathy
31 (97%)
37 (86%)
18 (95%)
0.214
PAD
14 (44%)
21 (49%)
8 (42%)
0.853
Previous foot ulcer
27 (84%)
37 (86%)
17 (90%)
0.878
Current foot ulcer
32 (100%)
42 (98%)
19 (100%)
0.549
New foot ulcer
21 (66%)
42 (98%)
19 (100%)
≤ 0.001**
Previous amputation
16 (50%)
17 (40%)
7 (37%)
0.566
Clinical diagnoses of high-risk factors
Neuropathy
31 (97%)
38 (88%)
19 (100%)
0.146
Any PAD
14 (44%)
22 (51%)
8 (42%)
0.735
Acute Charcot
1 (3%)
1 (2%)
0
NA
Foot deformity
26 (81%)
33 (77%)
17 (90%)
0.501
Acute risk classification
32 (100%)
42 (98%)
19 (100%)
0.549
Clinical diagnoses of foot disease
Ulcer type: neuropathic
14 (44%)
14 (33%)
8 (42%)
0.572
Ulcer type: neuroischaemic
9 (28%)
16 (37%)
8 (42%)
0.555
Ulcer type: ischaemic
0
3 (7%)
1 (5%)
NA
Ulcer type: other
9 (28%)
9 (21%)
2 (11%)
0.339
Combined surface area (mm2)#
122 (191)
352 (899)
379 (1436)
0.469
Any clinical infection
6 (19%)
14 (33%)
2 (11%)
0.125
UTWCS (Grade 1A)
13 (41%)
14 (33%)
7 (37%)
0.770
Deep ulcer (Grade 2 or 3)^
8 (25%)
6 (14%)
2 (11%)
0.394
* = Same participants used for three tests.
** = Statistically significant (p < 0.05).
# = Measure in mean (standard deviation).
^ = Deep ulcer determined from the UTWCS Grade item.
NA = Not applicable to test as the assumption of Chi-squared test is violated as 2 cells had expected counts of < 5.
GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound Classification System.

Criterion measure

Table 3 displays the reliability results for the criterion measure of expert clinicians. Thirty-two items were able to be statistically tested. Nine items (28%) recorded near perfect categories of agreement, nine (28%) substantial/strong categories, eleven (34%) moderate categories and three (9%) weak/fair categories. Thus, overall 29 (91%) of these items recorded at least a moderate category of reliability (K > 0.4; ICC > 0.75). The items recording weaker categories of reliability included other (non-listed) condition, University of Texas Wound Classification System (UTWCS) grade and optimum footwear.
Table 3
Criterion measure reliability – measure of agreement between gold standard experts
QHRFF item
%
K(SE)
Strength of agreement [[37]]
General demographics
  
Indigenous status*
77.4
0.45 (0.17)
Moderate
Health professionals attending (in the past week)
Podiatrist
96.9
--
--
GP
81.2
0.78 (0.21)
Substantial
Nurse
84.4
0.47 (0.18)
Moderate
Orthotist
100
--
--
Physician
96.9
0.78 (0.21)
Substantial
Surgeon
100
1.00 (0.01)
Near perfect
Other
100
--
--
Medical co-morbidity history
  
Medical (diabetes) history
100
1.00 (0.01)
Near perfect
Diabetes duration#
 
0.94 (0.88–0.98)
Strong
Recent BGLs > 15 mmol/L*
90.6
0.81 (0.11)
Near perfect
HbA1c result#
 
0.93 (0.79–0.98)
Strong
Hypertension
87.5
0.60 (0.17)
Moderate
Dyslipidaemia
75.0
0.46 (0.16)
Moderate
Smoker
96.9
0.89 (0.11)
Near perfect
CVD
93.8
0.86 (0.10)
Near perfect
CKD
100
1.00 (0.01)
Near perfect
ESRF
96.9
--
--
Other (non-listed)
46.9
0.04 (0.09)
Slight
High-risk history
 
Neuropathy
93.8
0.48 (0.31)
Moderate
PAD
84.4
0.68 (0.13)
Substantial
Previous foot ulcer
84.4
0.45 (0.21)
Moderate
Current foot ulcer
96.9
--
--
New foot ulcer
83.9
0.58 (0.15)
Moderate
Previous amputation
90.6
0.81 (0.10)
Near perfect
Clinical diagnoses of high-risk factors
 
Neuropathy
96.9
0.65 (0.32)
Substantial
PAD*
77.4
0.61 (0.12)
Substantial
Acute Charcot
100
1.00 (0.01)
Near perfect
Foot deformity
81.3
0.46 (0.19)
Moderate
Risk classification*
96.8
--
--
Clinical diagnoses of foot disease
 
Ulcer type*
87.5
0.85 (0.08)
Near perfect
Combined surface area mm2#
 
0.73 (0.52-0.86)
Moderate
Clinical signs of infection*
84.3
0.68 (0.15)
Substantial
UTWCS grade#
 
0.56 (0.28–0.76)
Weak-moderate
Ulcer depth^
78.1
0.67 (0.13)
Substantial
Clinical management principles performed
 
Debrided ulcer/callus*
93.8
--
--
Dressing optimum*
96.9
--
--
Antibiotics required*
84.4
0.47 (0.19)
Moderate
Off-loading optimum*
68.8
0.41 (0.14)
Moderate
Footwear optimum*
65.6
0.28 (0.16)
Fair
Educated patient*
100
--
--
K = Kappa; (SE) = Standard Error; % = Percentage agreement.
-- = Measure of agreement was unable to be calculated because at least one variable was a constant.
GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound Classification System.
* = Weighted Kappa used for ordinal data.
# = Intraclass correlation (ICC) (Standard Deviation (SD)) for continuous data.
^ = Depth was determined from the UTWCS score.

Criterion validity

Tables 4 and 5 display the criterion validity results for both the senior (level 4) clinician and clinician (level 3) respectively tested against the criterion measure for each item. Thirty-six items were able to be statistically tested for sensitivity, specificity or positive predictive values (PPV) on both clinicians. Thus, 72 different tests were each performed for sensitivity, specificity and PPV. Sixty-one (85%), 59 (82%) and 60 (83%) items recorded at least moderate categories (> 0.7) for sensitivity, specificity and PPV respectively. Conversely, three (8%), five (14%) and four (11%) out of the 36 items recorded weak categories (< 0.7) for sensitivity, specificity and PPV respectively on both clinicians tested. The items registering weak categories of validity (sensitivity, specificity or PPV on both clinicians) included identifying a patient that had attended a GP, physician or surgeon; chronic kidney disease (CKD) or other (non-listed) condition; previous, current and new foot ulcer all had poor specificity in particular; and optimum footwear had poor PPVs.
Table 4
Criterion validity summary statistics for senior clinician (Sensitivity, specificity and positive predictive value) & inter-rater reliability between expert and senior clinicians (Kappa and strength of agreement)
QHRFF field
%
K(SE)
Strength of agreement [[37]]
Sensitivity (95% CI)
Specificity (95% CI)
PPV (95% CI)
General demographics
      
Any indigenous status
100
--
--
--
1.00 (0.90-1.00)
--
Health professionals attending (in the past week)
   
Podiatrist
100
--
--
1.00 (0.90-1.00)
--
1.00 (0.90-1.00)
GP
86.0
0.64 (0.14)
Substantial
0.80 (0.44-0.96)
0.88 (0.71-0.96)
0.67 (0.35-0.88)
Nurse
93.0
0.83 (0.09)
Near perfect
0.92 (0.60-1.00)
0.94 (0.77-0.99)
0.85 (0.54-0.97)
Orthotist
100
--
--
--
1.00 (0.90-1.00)
--
Physician
93.0
--
--
0 (0 – 0.95)
0.95 (0.83-0.99)
0 (0 – 0.80)
Surgeon
95.3
0.64 (0.24)
Substantial
0.67 (0.13-0.98)
0.98 (0.85-1.00)
0.67 (0.13-0.98)
Other
100
--
--
--
1.00 (0.90-1.00)
--
Medical co-morbidity history
    
Any medical (diabetes) history
100
1.00 (0.01)
Near perfect
1.00 (0.88-1.00)
1.00 (0.60-1.00)
1.00 (0.88-1.00)
Diabetes duration#
 
0.90 (0.80– 0.95)
Strong
   
Recent BGLs > 15 mmol/L
93.0
0.69 (0.16)
Substantial
0.57 (0.20-0.88)
1.00 (0.88-1.00)
1.00 (0.40-1.00)
HbA1c result#
 
1.00 (0.99–1.00)
Strong
   
Hypertension
93.0
0.83 (0.09)
Near perfect
0.97 (0.81-1.00)
0.85 (0.54-0.97)
0.94 (0.77-0.99)
Dyslipidaemia
95.3
0.90 (0.07)
Near perfect
0.96 (0.78-1.00)
0.94 (0.71-1.00)
0.96 (0.78-1.00)
Smoker
90.7
0.72 (0.13)
Substantial
0.64 (0.32-0.88)
1.00 (0.87-1.00)
1.00 (0.56-1.00)
CVD
90.7
0.81 (0.09)
Near perfect
0.86 (0.63-0.96)
0.95 (0.75-1.00)
0.95 (0.72-1.00)
CKD
81.4
0.49 (0.15)
Moderate
0.46 (0.20-0.74)
0.97 (0.81-1.00)
0.86 (0.42-0.99)
ESRF
1.00
1.00 (0.01)
Near perfect
1.00 (0.05-1.00)
1.00 (0.90-1.00)
1.00 (0.05-1.00)
Other (non-listed)
57.1
0.09 (0.12)
Slight
0.57 (0.40-0.73)
0.57 (0.20-0.88)
0.87 (0.65-0.97)
High-risk history
    
Neuropathy
93.0
0.73 (0.15)
Substantial
0.95 (0.80-0.99)
0.83 (0.36-0.99)
0.97 (0.84-1.00)
PAD
88.4
0.77 (0.10)
Substantial
0.86 (0.63-0.96)
0.91 (0.69-0.98)
0.90 (0.67-0.98)
Previous foot ulcer
79.1
0.28 (0.18)
Fair
0.84 (0.67-0.93)
0.50 (0.14-0.86)
0.91 (0.75-0.98)
Current foot ulcer
95.3
--
--
0.98 (0.86-1.00)
0 (0 – 0.95)
0.98 (0.86-1.00)
New foot ulcer
93.0
--
--
0.95 (0.82-0.99)
0 (0 – 0.95)
0.98 (0.86-1.00)
Previous amputation
97.7
0.95 (0.05)
Near perfect
0.94 (0.69-1.00)
1.00 (0.84-1.00)
1.00 (0.76-1.00)
Clinical diagnoses of high-risk factors
   
Neuropathy
97.7
0.90 (0.10)
Near perfect
0.97 (0.85-1.00)
1.00 (0.46-1.00)
1.00 (0.88-1.00)
Any PAD
93.0
0.86 (0.8)
Near perfect
0.91 (0.69-0.98)
0.95 (0.74-1.00)
0.95 (0.74-1.00)
Acute Charcot
100
1.00 (0.01)
Near perfect
1.00 (0.05-1.00)
1.00 (0.90-1.00)
1.00 (0.05-1.00)
Foot deformity
86.0
0.51 (0.16)
Moderate
1.00 (0.87-1.00)
0.4 (0.14-0.73)
0.85 (0.69-0.94)
Acute risk classification
97.7
--
--
1.00 (0.90-1.00)
0 (0 – 0.95)
0.98 (0.86-1.00)
Clinical diagnoses of foot disease
    
Ulcer type: neuropathic (neuropathic or neuroisch.)
90.7
0.76 (0.11)
Substantial
0.88 (0.72-0.96)
1.0 (0.63-1.00)
1.00 (0.86-1.00)
Ulcer type: ischaemic
79.1
0.58 (0.13)
Moderate
0.75 (0.51-0.90)
0.83 (0.60-0.94)
0.79 (0.54-0.93)
(Neuroisch. or ischaemic)
Combined surface area mm2#
 
0.93 (0.87–0.96)
Strong
   
Any clinical infection
88.4
0.72 (0.12)
Substantial
0.71 (0.42-0.90)
0.97 (0.80-1.00)
0.91 (0.57-1.00)
UTWCS (Grade 1A)
83.7
0.65 (12)
Substantial
0.85 (0.56-0.97)
0.83 (0.64-0.93)
0.71 (0.44-0.89)
Deep ulcer
95.3
0.81 (0.13)
Near perfect
0.71 (0.30-0.95)
0.97 (0.84-1.00)
0.83 (0.36-0.99)
(Grade 2 or 3)^
Clinical management principles performed
   
Ulcer debridement is required
95.3
0.66 (0.01)
Substantial
1.00 (0.89-1.00)
1.00 (0.20-1.00)
1.00 (0.89-1.00)
Dressing is optimum
100
--
--
1.00 (0.90-1.00)
--
1.00 (0.90-1.00)
Antibiotics are required
93.0
0.82 (0.10)
Near perfect
0.83 (0.51-0.97)
0.94 (0.77-0.99)
0.83 (0.51-0.97)
Off-loading is optimum
93.0
0.80 (0.11)
Substantial
0.80 (0.44-0.96)
0.97 (0.82-1.00)
0.89 (0.51-0.99)
Footwear is optimum
72.1
0.31 (0.16)
Fair
0.50 (0.20-0.80)
0.82 (0.64-0.92)
0.45 (0.18-0.75)
Educated patient
97.7
--
--
0.98 (0.86-1.00)
--
1.00 (0.90-1.00)
K = Kappa; (SE) = Standard Error; % = Percentage agreement; PPV = Positive Predictive Value.
-- = Measure of agreement was unable to be calculated because at least one variable was a constant.
GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound Classification System.
# = Intraclass correlation (ICC) (Standard Deviation (SD)) for continuous data.
^ = Depth was determined from the UTWCS score.
Table 5
Criterion validity summary statistics for clinician (Sensitivity, specificity and positive predictive value) & inter-rater reliability between expert and clinician (Kappa and strength of agreement)
QHRFF field
%
K(SE)
Strength of agreement [[37]]
Sensitivity (95% CI)
Specificity (95% CI)
PPV (95% CI)
General demographics
      
Any indigenous status
100
--
--
--
1.00 (0.90-1.00)
--
Health professionals attending (in the past week)
   
Podiatrist
88.4
--
--
0.88 (0.74-0.96)
--
0.88 (0.74-0.96)
GP
81.4
0.39 (0.17)
Fair
0.40 (0.14-0.73)
0.94 (0.78-0.99)
0.67 (0.24-0.94)
Nurse
79.1
0.37 (0.16)
Fair
0.33 (0.11-0.65)
0.97 (0.81-1.00)
0.80 (0.30-0.99)
Orthotist
100
--
--
--
1.00 (0.90–1.00)
--
Physician
97.6
0.66 (0.32)
Substantial
1.00 (0.05-1.00)
0.98 (0.86-1.00)
0.50 (0.03-0.97)
Surgeon
90.7
0.45 (0.23)
Moderate
0.67 (0.13-0.98)
0.93 (0.79-0.98)
0.40 (0.07-0.83)
Other
100
--
--
--
0.98 (0.86-1.00)
--
Medical co-morbidity history
   
Any medical (diabetes) history
100
100 (0.01)
Near perfect
1.00 (0.88-1.00)
1.00 (0.60-1.00)
1.00 (0.88-1.00)
Diabetes duration#
 
0.95 (0.90–0.97)
Strong
   
Recent BGLs > 15 mmol/L
90.7
0.62 (0.17)
Substantial
1.00 (0.40-1.00)
0.90 (0.75-0.97)
0.50 (0.17-0.83)
HbA1c result#
 
1.00 (0.99–1.00)
Strong
   
Hypertension
86.0
0.68 (0.12)
Substantial
0.87 (0.68-0.96)
0.85 (0.54-0.97)
0.93 (0.75-0.99)
Dyslipidaemia
79.1
0.58 (0.12)
Moderate
0.76 (0.54-0.90)
0.83 (0.58-0.96)
0.86 (0.64-0.96)
Smoker
88.4
0.69 (0.13)
Substantial
0.73 (0.39-0.93)
0.94 (0.78-0.99)
0.80 (0.44-0.96)
CVD
79.1
0.58 (0.12)
Moderate
0.71 (0.48-0.88)
0.86 (0.64-0.96)
0.83 (0.58-0.96)
CKD
86.0
0.65 (0.13)
Substantial
0.69 (0.39-0.90)
0.93 (0.76-0.99)
0.82 (0.48-0.97)
ESRF
97.7
0.66 (0.32)
Substantial
1.00 (0.05-1.00)
0.98 (0.86-1.00)
0.50 (0.03-0.97)
Other (non-listed)
79.1
0.19 (0.19)
Slight
0.89 (0.73-0.96)
0.29 (0.05-0.7)
0.86 (0.70-0.95)
High-risk history
   
Neuropathy
93.0
0.73 (0.15)
Substantial
0.95 (0.80-0.99)
0.83 (0.36-0.99)
0.97 (0.84-1.00)
PAD
86.0
0.72 (0.10)
Substantial
0.76 (0.52-0.91)
0.95 (0.75-1.00)
0.94 (0.69-1.00)
Previous foot ulcer
79.1
0.28 (0.18)
Fair
0.84 (0.67-0.93)
0.50 (0.14-0.86)
0.91 (0.75-0.98)
Current foot ulcer
97.7
--
--
1.00 (0.90-1.00)
0 (0 – 0.95)
0.98 (0.86-1.00)
New foot ulcer
97.4
--
--
0.90 (0.76-0.97)
0 (0 – 0.95)
0.97 (0.85-0.99)
Previous amputation
95.3
0.90 (0.07)
Near perfect
0.88 (0.62-0.98)
1.00 (0.84-1.00)
1.00 (0.75-1.00)
Clinical diagnoses of high-risk factors
   
Neuropathy
100
1.00 (0.01)
Near perfect
1.00 (0.89-1.00)
1.00 (0.46-1.00)
1.00 (0.89-1.00)
Any PAD
95.3
0.91 (0.06)
Near perfect
0.91 (0.69-0.98)
1.00 (0.81-1.00)
1.00 (0.80-1.00)
Acute Charcot
100
1.00 (0.01)
Near perfect
1.00 (0.05-1.00)
1.00 (0.90-1.00)
1.00 (0.05-1.00)
Foot deformity
93.0
0.80 (0.11)
Substantial
0.97 (0.82-1.00)
0.80 (0.44-0.96)
0.94 (0.79-0.99)
Acute risk classify
97.7
--
--
1.00 (0.90-1.00)
0 (0 – 0.95)
0.98 (0.86-1.00)
Clinical diagnoses of foot disease
   
Ulcer type: neuropathic
83.7
0.55 (0.14)
Moderate
1.00 (0.86-1.00)
0.46 (0.20-0.74)
0.81 (0.64-0.91)
(Neuropathic or neuroisch.)
Ulcer type is ischaemic
79.1
0.57 (0.13)
Moderate
0.74 (0.48-0.90)
0.83 (0.62-0.95)
0.78 (0.52-0.93)
(Neuroisch. or ischaemic)
Combined surface area mm2#
 
0.99 (0.99–1.00)
Strong
   
Any clinical infection
81.4
0.58 (0.13)
Moderate
0.71 (0.42-0.90)
0.86 (0.67-0.95)
0.71 (0.42-0.90)
UTWCS (Grade 1A)
81.4
0.61 (0.12)
Substantial
0.86 (0.56-0.97)
0.79 (0.60-0.91)
0.69 (0.41-0.86)
Deep ulcer
93.0
0.73 (0.15)
Substantial
0.71 (0.30-0.95)
0.97 (0.84-1.00)
0.83 (0.36-0.99)
(Grade 2 or 3)^
Clinical management principles performed
   
Ulcer debridement is required
95.3
0.39 (0.22)
Fair
1.00 (0.89-1.00)
0.50 (0.03-0.97)
0.98 (0.86-1.00)
Dressing is optimum
97.7
--
--
0.98 (0.86-1.00)
--
1.00 (0.90-1.00)
Antibiotics are required
90.7
0.76 (0.11)
Substantial
0.75 (0.43-0.93)
0.97 (0.81-1.00)
0.90 (0.54-0.99)
Off-loading is optimum
79.1
0.50 (0.14)
Moderate
0.80 (0.44-0.96)
0.79 (0.60-0.90)
0.53 (0.27-0.78)
Footwear is optimum
79.1
0.47 (0.15)
Moderate
0.70 (0.35-0.92)
0.82 (0.64-0.92)
0.54 (0.26-0.80)
Educated patient
100
--
--
1.00 (0.90-1.00)
--
1.00 (0.90-1.00)
K = Kappa; (SE) = Standard Error; % = Percentage agreement; PPV = Positive Predictive Value.
-- = Measure of agreement was unable to be calculated because at least one variable was a constant.
GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound Classification System.
*= Weighted Kappa used for ordinal data.
# = Intraclass correlation (ICC) (Standard Deviation (SD)) for continuous data.
^ = Depth was determined from the UTWCS score.

Inter-rater reliability

Tables 4, 5 and 6 display the inter-rater reliability results for all three tests. Ninety-six different tests were able to be statistically tested for inter-rater reliability across the three different pairs of clinician agreement. Twenty four items (25%) recorded near perfect categories of agreement, 43 (45%) substantial/strong categories, 17 (18%) moderate categories and 12 (12%) weak/fair categories. Thus, overall 84 (88%) of these items recorded at least moderate categories of reliability (K > 0.4; ICC > 0.75). The items recording weak categories of reliability across two or more agreements tested included other (non-listed) condition, previous foot ulcer and optimum footwear.
Table 6
Reliability measure of agreement summary statistics: Inter-rater (Senior clinician & clinician) & intra-rater (Clinician)
 
Inter-rater reliability
Intra-rater reliability
QHRFF field
%
K(SE)
Strength of agreement [[37]]
%
K(SE)
Strength of agreement [[37]]
General demographics
      
Indigenous status*
83.7
--
--
100
1.00 (0.01)
Near Perfect
Health professionals attending (in the past week)
   
Podiatrist
88.4
--
--
N/A
N/A
N/A
GP
81.4
0.45 (0.15)
Moderate
N/A
N/A
N/A
Nurse
81.4
0.47 (0.14)
Moderate
N/A
N/A
N/A
Orthotist
100
--
--
N/A
N/A
N/A
Physician
90.7
--
--
N/A
N/A
N/A
Surgeon
90.7
0.45 (0.23)
Moderate
N/A
N/A
N/A
Other
97.7
--
--
N/A
N/A
N/A
Medical co-morbidity history
    
Medical (diabetes) history
95.3
0.94 (0.05)
Near perfect
100
1.00 (0.01)
Near Perfect
Diabetes duration#
 
0.94 (0.87–0.97)
Strong
 
1.00 (0.99–1.00)
Strong
Recent BGLs >15 mmol/L*
83.7
0.67 (0.12)
Substantial
N/A
N/A
N/A
HbA1c result#
 
1.00 (1.00–1.00)
Strong
 
1.00 (1.00–1.00)
 
Hypertension
88.4
0.73 (0.11)
Substantial
89.5
0.78 (0.15)
Substantial
Dyslipidaemia
83.7
0.67 (0.11)
Substantial
94.7
0.90 (0.10)
Near Perfect
Smoker
88.4
0.64 (0.15)
Substantial
94.7
0.83 (0.17)
Near Perfect
CVD
88.4
0.76 (0.10)
Substantial
84.2
0.69 (0.17)
Substantial
CKD
86.1
0.58 (0.15)
Moderate
94.7
0.86 (0.14)
Near Perfect
ESRF
97.7
0.66 (0.32)
Substantial
100
--
--
Other (non-listed)
69.1
0.34 (0.11)
Fair
78.9
0.57 (0.19)
Moderate
High-risk history
    
Neuropathy
90.7
0.66 (0.16)
Substantial
94.7
--
--
PAD
83.7
0.67 (0.11)
Substantial
78.9
0.60 (0.16)
Moderate
Previous foot ulcer
90.7
0.72 (0.13)
Substantial
94.7
0.64 (0.33)
Substantial
Current foot ulcer
97.7
--
--
100
--
--
New foot ulcer
97.4
--
--
100
--
--
Previous amputation
97.7
0.95 (0.05)
Near perfect
100
1.00 (0.01)
Near Perfect
Clinical diagnoses of high-risk factors
    
Neuropathy
97.7
0.90 (0.10)
Near perfect
94.7
--
--
PAD*
97.9
0.97 (0.03)
Near perfect
84.2
0.69 (0.16)
Substantial
Acute charcot
100
1.00 (0.01)
Near perfect
100
--
--
Foot deformity
83.7
0.38 (0.18)
Fair
84.2
0.31 (0.30)
Fair
Risk classification
100
--
--
100
--
--
Clinical diagnoses of foot disease
    
Ulcer type*
88.4
0.81 (0.08)
Near perfect
84.2
0.77 (0.14)
Substantial
Combined surface area mm2#
 
0.94 (0.89–0.97)
Strong
 
1.00 (1.00–1.00)
Strong
Clinical signs infection*
76.7
0.60 (0.12)
Moderate
89.5
0.46 (0.32)
Moderate
UTWCS grade#
 
0.90 (0.83–0.95)
Strong
 
0.98 (0.94–0.99)
Strong
Ulcer depth^
95.3
0.90 (0.10)
Near perfect
94.7
0.79 (0.23)
Substantial
Clinical management principles performed
    
Debrided ulcer/callus*
97.7
0.66 (0.32)
Substantial
94.7
--
--
Dressing optimum*
97.7
--
--
100
--
--
Antibiotics required*
79.1
0.54 (0.12)
Moderate
89.5
0.44 (0.33)
Moderate
Off-loading optimum*
72.1
0.36 (0.13)
Fair
68.4
0.32 (0.23)
Fair
Footwear optimum*
67.4
0.33 (0.14)
Fair
68.4
0.37 (0.21)
Fair
Educated patient*
97.7
--
--
89.5
--
--
K = Kappa; (SE) = Standard Error; -- = Measure of agreement was unable to be calculated because at least one variable was a constant.
GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; N/A = Not applicable to test due to unstable rating environment; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound Classification System.
*= Weighted Kappa used for ordinal data;
# = Intraclass correlation (ICC) (Standard Deviation (SD)) for continuous data.
^ = Depth was determined from the UTWCS score.

Intra-rater reliability

Table 6 also displays the intra-rater reliability results for the level 3 clinician. The median (interquartile range) period between the first and second ratings for the intra-rater reliability testing was 2(1–2) weeks. Twenty-three items were able to be statistically tested. Six items (26%) recorded near perfect categories of agreement, ten (43%) substantial/strong categories, four (17%) moderate categories and three (13%) weak/fair categories. Thus, overall 20 (87%) of these items recorded at least a moderate category of reliability (K > 0.4; ICC > 0.75). Those items scoring weak categories of agreement were foot deformity, optimal offloading and optimal footwear.

Discussion

The QHHRF appears to be the first multi-item tool developed and tested to identify multiple high-risk factors and foot disease complications in multiple at risk populations. Our findings indicate that the majority of the tool’s items demonstrates at least moderate categories of validity (face, content and criterion validity) and reliability (inter-rater and intra-rater); particularly those in the domains of identifying relevant medical co-morbidity history, and clinical diagnoses of high-risk factors and foot disease complications. However, some items in the domains of identifying different health professionals previously attending the patient and general clinical management principles performed appear to have weaker categories of validity and reliability that need addressing in future versions of the tool.
The QHRFF went through a number of developmental, validity and reliability tests to determine its status as a valid and reliable tool. The magnitude of validity (or accuracy) in this study was evaluated via the methods of face, content and importantly criterion validity. Face and content validity are considered the least robust of the validity tests due to their inherent subjectivity [31, 33]. However, they are considered important factors in the development phase to ensure the tool can actually measure the general identified construct [31, 33]; in this case foot disease. At the completion of phase one, the expert panel and network stakeholders endorsements implied high practicality, face and content validity.
Criterion validity is considered to be the most objective validity test [31, 33]. To test concurrent criterion validity, as used by this study, a satisfactory criterion measure must be used. In this study the criterion measure to test criterion validity was an expert’s clinical diagnosis. Overall, the 'expert’ criterion measure used for this study were rated as having at least moderate categories of reliability for all but three items; any other (non-listed) co-morbidity, optimum footwear and UTWCS grade. The other (non-listed) co-morbidity and optimum footwear items rated in the weak categories in most other inter- or intra-rater reliability agreement tested and should be reviewed or removed in future versions of the QHRFF. The weak-moderate agreement for the UTWCS grade was unexpected as this tool has been validated extensively in the past [62], however, all other inter- or intra-rater agreements on the UTWCS were rated as substantial/strong categories of reliability. Thus, it is recommended that the UTWCS grade be retained in future with more emphasis applied on the UTWCS grading system in the training sessions provided. It is certainly possible that the criterion measure for each item may have been more robust if the individual gold standard test for each individual item were used; for example an angiogram to diagnose peripheral arterial disease [22] or nerve conduction studies to diagnose neuropathy [44]. However, such an approach would have been particularly resource and time intensive and was thus not practical within the resources available to this study.
The QHRFF items were tested for concurrent criterion validity by using two different blinded representative general clinicians’ ratings compared to the criterion measure (a reliable expert’s diagnosis) on the same patients. Positive predictive values from this testing indicated the majority of QHRFF items had at least moderate validity when used by clinicians with different levels of expertise. The high positive predictive values suggest that the proportion of people with a positive test result for individual QHRFF items actually had the relevant medical co-morbidity, high-risk factor or foot disease complication of interest. Although there were a few notable exceptions such as identifying different health professionals attending previously, recent blood glucose levels (BGLs) > 15 mmol/L, ESRF, optimal offloading and optimum footwear. However, overall the QHRFF tool appears to demonstrate acceptable validity in the majority of its items to be considered a valid tool to test the foot disease construct in multiple at risk populations.
Reliability (or consistency) is a major prerequisite of any useful items to be measured [31, 33]. In this study inter-rater and intra-rater measures of agreement were used for reliability. Inter-rater reliability was primarily assessed using three clinicians with different representative levels of expertise in managing foot disease. Overall, all items consistently displayed at least moderate categories of reliability, except for any other (non-listed) co-morbidity, optimum footwear and previous foot ulcer. Optimum footwear was also identified to have weak categories of intra-rater reliability; along with identifying foot deformity and optimum offloading. Thus, the QHRFF tool appears to demonstrate satisfactory reliability to collect the majority of items in the foot disease construct. However, the authors recognise that it would have been preferable to test reliability with more clinicians across a larger sample of patients, yet this was beyond the resourcing available for this study.
Notwithstanding, the aforementioned methodological limitations, the study does incorporate many best practice research design methods for testing validity and reliability. These methods included testing the reliability of the criterion measure to diagnose and measure each item, testing the subsequent concurrent criterion validity of representative clinicians, and testing the reliability of the tool when used by clinicians with different levels of expertise. Furthermore, the study employed patient samples that were highly representative of the demographics and co-morbidity prevalence rates found in multiple 'at risk’ populations for high risk factors and foot disease complications. These high prevalence rates not only suggest the tool was tested in the construct it was designed to measure, but literature suggests higher prevalence rates improve the statistical robustness of validity and reliability results [31]. Lastly, many existing validated single-item tools [23, 46, 60, 62] were incorporated within the QHRFF tool and this potentially adds weight to the validity reported in this study. Thus, overall the QHRFF tool appears to be a valid and reliable tool to collect the vast majority of items contained within the foot disease construct and can be used as a general tool to complement existing specific tools in the field of foot disease.

Limitations

Several limitations have already been outlined in this study including the use of a general criterion measure of an experts’ clinical diagnoses for each item, only using a limited number of representative clinicians and testing intra-rater reliability on one clinician in a small sample. Other limitations include: not performing a systematic literature review (and thus some evidence based items and tools may have been overlooked); using only podiatrists as the clinical raters; using historically defined strength categories for validity and reliability; not testing the tool for construct validity, predictive validity, or factor analysis; and the settings used were existing HRFS only. It is recommended that any future research into this tool should address these methodological limitations by investigating the factor analysis, construct and predictive criterion validity of the tool’s items in a much larger and broader sample of patients with more multi-disciplinary clinician raters. Furthermore, if resources permit, the use of individual gold standard criterion measures and/or existing concurrent validated single-item tools should be considered.
However, there are a number of recommendations from this study’s findings that should be considered for implementation in future versions of the QHRFF tool. Firstly, those items reporting weaker categories of validity or reliability and that potentially collect duplicate information to other more reliable items should be removed; including any other (non-listed) co-morbidity, combined surface area (change), recent BGLs greater than 15 mmol/L and new ulcer. Secondly, some items’ definitions should be reviewed in an attempt to improve the item’s future reliability and validity. For example, previous foot ulcer definition could be modified to identify a previously “healed” foot ulcer, 'health professionals attending’ previously requires an exact retrospective time period that patients were in attendance in alignment with similar literature (such as “in the previous two weeks”) [64, 65] and optimum offloading should have a more explicit definition referring directly to non-removable offloading definitions in the existing literature [14]. Thirdly, the validation of criteria for identifying a foot deformity and optimum footwear are urgently required for these 'at risk’ populations. To the best of the author’s knowledge such tools for these items have been developed [14, 50] yet are to be validated and this study was no exception. However, in the meantime the authors recommend adding the numerical foot deformity score to the QHRFF to improve accuracy and test for validity. Lastly, a systematic literature review for other reported independent associates for foot disease (such as cancer, arthritis, depression, trauma, vision impairment, mobility impairment and social determinant factors) and other outcome measures for foot disease (such as quality of life measures) should be performed to ensure all 'at risk’ populations and outcome measures for foot disease are identified, considered and potentially included in the next QHRFF version.

Conclusions

The QHRFF tool appears to be the first multi-item tool developed and tested to identify multiple high-risk factors and foot disease complications in multiple at risk populations. Acceptable validity and reliability were demonstrated for the majority of items; particularly in the domains of identifying relevant medical co-morbidity history and clinically diagnosing high-risk factors and foot disease complications. However, recommendations to remove, add or redefine items registering weak validity or reliability scores should be implemented to improve future versions of the QHRFF tool; particularly in the domains of health professionals attending and clinical management principles. Overall, the QHRFF tool appears to demonstrate the practicality, validity and reliability required to facilitate robust clinical assessment and data capture to measure the large burden of foot disease facing our multiple at risk populations.

Acknowledgements

The authors wish to acknowledge the significant contribution made to the project by Ms Helen Martin, Ms Kimberley Canning, Ms Helen Sheahan and the entire Metro North Hospital and Health Service Podiatry Department. Furthermore, the authors wish to acknowledge the significant support of the Queensland Health Statewide Diabetes Clinical Network, Diabetes Foot Working Group, Renal Clinical Network and Podiatry Network. The authors are also grateful to the grant funding for the project provided by the Metro North Hospital & Health Service Clinical Practice Incentive Payment Grants. None of these contributors have a relevant conflict of interest. Lastly, the authors wish to acknowledge Dr Sheree Nix for reviewing final drafts of the manuscript.
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Competing interests

The authors have no relevant conflict of interest to disclose.

Authors’ contributions

PAL conceived, designed, researched data, contributed to discussion, wrote and reviewed/edited the manuscript. VN provided research assistance, researched data, contributed to discussion and reviewed/edited the manuscript. EMK conceived, contributed to discussion and reviewed/edited the manuscript. MSK, SSK and LR designed, contributed to discussion, and reviewed/edited the manuscript. CH designed, researched data and reviewed/edited the manuscript. All authors read and approved the final manuscript.

Onze productaanbevelingen

BSL Podotherapeut Totaal

Binnen de bundel kunt u gebruik maken van boeken, tijdschriften, e-learnings, web-tv's en uitlegvideo's. BSL Podotherapeut Totaal is overal toegankelijk; via uw PC, tablet of smartphone.

Bijlagen

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.
Literatuur
1.
go back to reference Bakker K, Apelqvist J, Schaper NC: Practical guidelines on the management and prevention of the diabetic foot 2011. Diabetes/Metab Res Rev. 2012, 28 (Suppl 1): 225-231.CrossRef Bakker K, Apelqvist J, Schaper NC: Practical guidelines on the management and prevention of the diabetic foot 2011. Diabetes/Metab Res Rev. 2012, 28 (Suppl 1): 225-231.CrossRef
2.
go back to reference Australian Institute of Health & Welfare (AIHW): Diabetes: Australian facts 2008. 2008, Canberra: Australian Government, http://www.aihw.gov.au/publications/cvd/daf08/daf08.pdf Accessed 25th June 2013 Australian Institute of Health & Welfare (AIHW): Diabetes: Australian facts 2008. 2008, Canberra: Australian Government, http://​www.​aihw.​gov.​au/​publications/​cvd/​daf08/​daf08.​pdf Accessed 25th June 2013
3.
go back to reference Barrois B, Labalette C, Rousseau P, Corbin A, Colin D, Allaert F, Saumet JL: A national prevalence study of pressure ulcers in French hospital inpatients. J Wound Care. 2008, 17: 373-CrossRefPubMed Barrois B, Labalette C, Rousseau P, Corbin A, Colin D, Allaert F, Saumet JL: A national prevalence study of pressure ulcers in French hospital inpatients. J Wound Care. 2008, 17: 373-CrossRefPubMed
4.
go back to reference Lazzarini PA, Gurr JM, Rogers JR, Schox A, Bergin SM: Diabetes foot disease: the Cinderella of Australian diabetes management?. J Foot Ankle Res. 2012, 5: 24-10.1186/1757-1146-5-24.CrossRefPubMedPubMedCentral Lazzarini PA, Gurr JM, Rogers JR, Schox A, Bergin SM: Diabetes foot disease: the Cinderella of Australian diabetes management?. J Foot Ankle Res. 2012, 5: 24-10.1186/1757-1146-5-24.CrossRefPubMedPubMedCentral
5.
go back to reference Moxey PW, Gogalniceanu P, Hinchliffe RJ, Loftus IM, Jones KJ, Thompson MM, Holt PJ: Lower extremity amputations–a review of global variability in incidence. Diabetic Med: J Br Diabetic Assoc. 2011, 28: 1144-1153. 10.1111/j.1464-5491.2011.03279.x.CrossRef Moxey PW, Gogalniceanu P, Hinchliffe RJ, Loftus IM, Jones KJ, Thompson MM, Holt PJ: Lower extremity amputations–a review of global variability in incidence. Diabetic Med: J Br Diabetic Assoc. 2011, 28: 1144-1153. 10.1111/j.1464-5491.2011.03279.x.CrossRef
6.
go back to reference Ephraim PL, Dillingham TR, Sector M, Pezzin LE, MacKenzie EJ: Epidemiology of limb loss and congenital limb deficiency: a review of the literature. Arch Phys Med Rehabil. 2003, 84: 747-761.CrossRefPubMed Ephraim PL, Dillingham TR, Sector M, Pezzin LE, MacKenzie EJ: Epidemiology of limb loss and congenital limb deficiency: a review of the literature. Arch Phys Med Rehabil. 2003, 84: 747-761.CrossRefPubMed
7.
go back to reference Global Lower Extremity Amputation Study Group: Epidemiology of lower extremity amputation in centres in Europe, North America and East Asia. Br J Surg. 2000, 87: 328-337. 10.1046/j.1365-2168.2000.01344.x.CrossRef Global Lower Extremity Amputation Study Group: Epidemiology of lower extremity amputation in centres in Europe, North America and East Asia. Br J Surg. 2000, 87: 328-337. 10.1046/j.1365-2168.2000.01344.x.CrossRef
8.
go back to reference Lazzarini PA, O’Rourke SR, Russell AW, Clark D, Kuys SS: What are the key conditions associated with lower limb amputations in a major Australian teaching hospital?. J Foot Ankle Res. 2012, 5: 12-10.1186/1757-1146-5-12.CrossRefPubMedPubMedCentral Lazzarini PA, O’Rourke SR, Russell AW, Clark D, Kuys SS: What are the key conditions associated with lower limb amputations in a major Australian teaching hospital?. J Foot Ankle Res. 2012, 5: 12-10.1186/1757-1146-5-12.CrossRefPubMedPubMedCentral
9.
go back to reference Lavery LA, Houtum WHV, Armstrong DG: Institutionalization following diabetes-related lower extremity amputation. Am J Med. 1997, 103: 383-388. 10.1016/S0002-9343(97)00163-0.CrossRefPubMed Lavery LA, Houtum WHV, Armstrong DG: Institutionalization following diabetes-related lower extremity amputation. Am J Med. 1997, 103: 383-388. 10.1016/S0002-9343(97)00163-0.CrossRefPubMed
10.
go back to reference Dillingham TR, Yacub JN, Pezzin LE: Determinants of postacute care discharge destination after dysvascular lower limb amputation. PM&R. 2011, 3: 336-344.CrossRef Dillingham TR, Yacub JN, Pezzin LE: Determinants of postacute care discharge destination after dysvascular lower limb amputation. PM&R. 2011, 3: 336-344.CrossRef
11.
go back to reference Armstrong DG, Wrobel J, Robbins JM: Guest editorial: are diabetes-related wounds and amputations worse than cancer?. Int Wound J. 2007, 4: 286-287. 10.1111/j.1742-481X.2007.00392.x.CrossRefPubMed Armstrong DG, Wrobel J, Robbins JM: Guest editorial: are diabetes-related wounds and amputations worse than cancer?. Int Wound J. 2007, 4: 286-287. 10.1111/j.1742-481X.2007.00392.x.CrossRefPubMed
12.
go back to reference Canavan RJ, Unwin NC, Kelly WF, Connolly VM: Diabetes- and nondiabetes-related lower extremity amputation incidence before and after the introduction of better organized diabetes foot care: continuous longitudinal monitoring using a standard method. Diabetes Care. 2008, 31: 459-463.CrossRefPubMed Canavan RJ, Unwin NC, Kelly WF, Connolly VM: Diabetes- and nondiabetes-related lower extremity amputation incidence before and after the introduction of better organized diabetes foot care: continuous longitudinal monitoring using a standard method. Diabetes Care. 2008, 31: 459-463.CrossRefPubMed
13.
go back to reference Krishnan S, Nash F, Baker N, Fowler D, Rayman G: Reduction in diabetic amputations over 11 years in a defined U.K. population: benefits of multidisciplinary team work and continuous prospective audit. Diabetes Care. 2008, 31: 99-101.CrossRefPubMed Krishnan S, Nash F, Baker N, Fowler D, Rayman G: Reduction in diabetic amputations over 11 years in a defined U.K. population: benefits of multidisciplinary team work and continuous prospective audit. Diabetes Care. 2008, 31: 99-101.CrossRefPubMed
14.
go back to reference National Health & Medical Research Council (NHMRC) Guidelines: National evidence-based guideline on prevention, identification and management of foot complications in diabetes (Part of the guidelines on management of type 2 diabetes). 2011, Melbourne: Baker IDI Heart & Diabetes Institute National Health & Medical Research Council (NHMRC) Guidelines: National evidence-based guideline on prevention, identification and management of foot complications in diabetes (Part of the guidelines on management of type 2 diabetes). 2011, Melbourne: Baker IDI Heart & Diabetes Institute
15.
go back to reference Kaminski M, Frescos N, Tucker S: Prevalence of risk factors for foot ulceration in patients with end-stage renal disease on haemodialysis. Intern Med J. 2012, 42: e120-e128. 10.1111/j.1445-5994.2011.02605.x.CrossRefPubMed Kaminski M, Frescos N, Tucker S: Prevalence of risk factors for foot ulceration in patients with end-stage renal disease on haemodialysis. Intern Med J. 2012, 42: e120-e128. 10.1111/j.1445-5994.2011.02605.x.CrossRefPubMed
16.
go back to reference Freeman A, May K, Frescos N, Wraight PR: Frequency of risk factors for foot ulceration in individuals with chronic kidney disease. Intern Med J. 2008, 38: 314-320. 10.1111/j.1445-5994.2007.01528.x.CrossRefPubMed Freeman A, May K, Frescos N, Wraight PR: Frequency of risk factors for foot ulceration in individuals with chronic kidney disease. Intern Med J. 2008, 38: 314-320. 10.1111/j.1445-5994.2007.01528.x.CrossRefPubMed
17.
go back to reference Tapp RJ, Shaw JE, Courten MPD, Dunstan DW, Welborn TA, Zimmet PZ: Foot complications in type 2 diabetes: an Australian population‒based study. Diabet Med. 2003, 20: 105-113. 10.1046/j.1464-5491.2003.00881.x.CrossRefPubMed Tapp RJ, Shaw JE, Courten MPD, Dunstan DW, Welborn TA, Zimmet PZ: Foot complications in type 2 diabetes: an Australian population‒based study. Diabet Med. 2003, 20: 105-113. 10.1046/j.1464-5491.2003.00881.x.CrossRefPubMed
18.
go back to reference Reed T, Veith FJ, Gargiulo NJ, Timaran CH, Ohki T, Lipsitz EC, Malas MB, Wain RA, Suggs WD: System to decrease length of stay for vascular surgery. J Vasc Surg. 2004, 39: 395-399. 10.1016/j.jvs.2003.09.015.CrossRefPubMed Reed T, Veith FJ, Gargiulo NJ, Timaran CH, Ohki T, Lipsitz EC, Malas MB, Wain RA, Suggs WD: System to decrease length of stay for vascular surgery. J Vasc Surg. 2004, 39: 395-399. 10.1016/j.jvs.2003.09.015.CrossRefPubMed
19.
go back to reference Lynch EA, Hillier SL, Stiller K, Campanella RR, Fisher PH: Sensory retraining of the lower limb after acute stroke: a randomized controlled pilot trial. Arch Phys Med Rehabil. 2007, 88: 1101-1107. 10.1016/j.apmr.2007.06.010.CrossRefPubMed Lynch EA, Hillier SL, Stiller K, Campanella RR, Fisher PH: Sensory retraining of the lower limb after acute stroke: a randomized controlled pilot trial. Arch Phys Med Rehabil. 2007, 88: 1101-1107. 10.1016/j.apmr.2007.06.010.CrossRefPubMed
20.
go back to reference Jeffcoate WJ, van Houtum WH: Amputation as a marker of the quality of foot care in diabetes. Diabetologia. 2004, 47: 2051-2058. 10.1007/s00125-004-1584-3.CrossRefPubMed Jeffcoate WJ, van Houtum WH: Amputation as a marker of the quality of foot care in diabetes. Diabetologia. 2004, 47: 2051-2058. 10.1007/s00125-004-1584-3.CrossRefPubMed
21.
go back to reference American College of Cardiology Foundation/American Heart Association (ACCF/AHA): ACCF/AHA focused update of the guideline for the management of patients with peripheral artery disease (Updating the 2005 guideline): a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines. Circulation. 2011, 124: 2020-2045. 10.1161/CIR.0b013e31822e80c3.CrossRef American College of Cardiology Foundation/American Heart Association (ACCF/AHA): ACCF/AHA focused update of the guideline for the management of patients with peripheral artery disease (Updating the 2005 guideline): a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines. Circulation. 2011, 124: 2020-2045. 10.1161/CIR.0b013e31822e80c3.CrossRef
22.
go back to reference Caruana MF, Bradbury AW, Adam DJ: The validity, reliability, reproducibility and extended utility of ankle to brachial pressure index in current vascular surgical practice. Eur J Vasc Endovasc Surg: The Official J Eur Soc Vasc Surg. 2005, 29: 443-451. 10.1016/j.ejvs.2005.01.015.CrossRef Caruana MF, Bradbury AW, Adam DJ: The validity, reliability, reproducibility and extended utility of ankle to brachial pressure index in current vascular surgical practice. Eur J Vasc Endovasc Surg: The Official J Eur Soc Vasc Surg. 2005, 29: 443-451. 10.1016/j.ejvs.2005.01.015.CrossRef
23.
go back to reference Young D, Schuerman S, Flynn K, Hartig K, Moss D, Altenburger B: Reliability and responsiveness of an 18 site, 10-g monofilament examination for assessment of protective foot sensation. J Geriatr Phys Ther (2001). 2011, 34: 95-98. 10.1519/JPT.0b013e31820aabe5.CrossRef Young D, Schuerman S, Flynn K, Hartig K, Moss D, Altenburger B: Reliability and responsiveness of an 18 site, 10-g monofilament examination for assessment of protective foot sensation. J Geriatr Phys Ther (2001). 2011, 34: 95-98. 10.1519/JPT.0b013e31820aabe5.CrossRef
24.
go back to reference Stausberg J, Lehmann N, Kröger K, Maier I, Niebel W: Reliability and validity of pressure ulcer diagnosis and grading: an image-based survey. Int J Nurs Stud. 2007, 44: 1316-1323. 10.1016/j.ijnurstu.2006.06.006.CrossRefPubMed Stausberg J, Lehmann N, Kröger K, Maier I, Niebel W: Reliability and validity of pressure ulcer diagnosis and grading: an image-based survey. Int J Nurs Stud. 2007, 44: 1316-1323. 10.1016/j.ijnurstu.2006.06.006.CrossRefPubMed
25.
go back to reference Armstrong DG, Lavery LA, Harkless LB: Treatment-based classification system for assessment and care of diabetic feet. Ostomy Wound Manage. 1996, 42: 50- Armstrong DG, Lavery LA, Harkless LB: Treatment-based classification system for assessment and care of diabetic feet. Ostomy Wound Manage. 1996, 42: 50-
26.
go back to reference Ince P, Abbas ZG, Lutale JK, Basit A, Ali SM, Chohan F, Morbach S, Mollenberg J, Game FL, Jeffcoate WJ: Use of the SINBAD classification system and score in comparing outcome of foot ulcer management on three continents. Diabetes Care. 2008, 31: 964-967. 10.2337/dc07-2367.CrossRefPubMed Ince P, Abbas ZG, Lutale JK, Basit A, Ali SM, Chohan F, Morbach S, Mollenberg J, Game FL, Jeffcoate WJ: Use of the SINBAD classification system and score in comparing outcome of foot ulcer management on three continents. Diabetes Care. 2008, 31: 964-967. 10.2337/dc07-2367.CrossRefPubMed
27.
go back to reference Lipsky BA, Weigelt JA, Sun X, Johannes RS, Derby KG, Tabak YP: Developing and validating a risk score for lower-extremity amputation in patients hospitalized for a diabetic foot infection. Diabetes Care. 2011, 34: 1695-1700. 10.2337/dc11-0331.CrossRefPubMedPubMedCentral Lipsky BA, Weigelt JA, Sun X, Johannes RS, Derby KG, Tabak YP: Developing and validating a risk score for lower-extremity amputation in patients hospitalized for a diabetic foot infection. Diabetes Care. 2011, 34: 1695-1700. 10.2337/dc11-0331.CrossRefPubMedPubMedCentral
28.
go back to reference Bergin SM, Brand CA, Colman PG, Campbell DA: A questionnaire for determining prevalence of diabetes related foot disease (Q-DFD): construction and validation. J Foot Ankle Res. 2009, 2: 34-10.1186/1757-1146-2-34.CrossRefPubMedPubMedCentral Bergin SM, Brand CA, Colman PG, Campbell DA: A questionnaire for determining prevalence of diabetes related foot disease (Q-DFD): construction and validation. J Foot Ankle Res. 2009, 2: 34-10.1186/1757-1146-2-34.CrossRefPubMedPubMedCentral
29.
go back to reference Lazzarini PA, O¿Rourke SR, Russell AW, Derhy PH, Kamp MC: Standardising practices improves clinical diabetic foot management: the Queensland Diabetic Foot Innovation Project, 2006–09. Aust Health Rev. 2012, 36: 8-15. 10.1071/AH10978.CrossRefPubMed Lazzarini PA, O¿Rourke SR, Russell AW, Derhy PH, Kamp MC: Standardising practices improves clinical diabetic foot management: the Queensland Diabetic Foot Innovation Project, 2006–09. Aust Health Rev. 2012, 36: 8-15. 10.1071/AH10978.CrossRefPubMed
30.
go back to reference Lazzarini PA, O’Rourke SR, Russell AW, Derhy PH, Kamp MC: Standardising practices improves ambulatory diabetic foot management and reduces amputations: the Queensland Diabetic Foot Innovation Project, 2006–2009. J Foot Ankle Res. 2011, 4: O25-10.1186/1757-1146-4-S1-O25.CrossRefPubMedCentral Lazzarini PA, O’Rourke SR, Russell AW, Derhy PH, Kamp MC: Standardising practices improves ambulatory diabetic foot management and reduces amputations: the Queensland Diabetic Foot Innovation Project, 2006–2009. J Foot Ankle Res. 2011, 4: O25-10.1186/1757-1146-4-S1-O25.CrossRefPubMedCentral
31.
go back to reference Portney LG, Watkins MP: Foundations of clinical research applications to practice. 2009, New Jersey, USA: Pearson Prentice, 3 Portney LG, Watkins MP: Foundations of clinical research applications to practice. 2009, New Jersey, USA: Pearson Prentice, 3
32.
go back to reference Terwee CB, Mokkink LB, Knol DL, Ostelo RWJG, Bouter LM, de Vet HCW: Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res: An Int J Qual Life Aspects Of Treatment, Care And Rehabil. 2012, 21: 651-657. 10.1007/s11136-011-9960-1.CrossRef Terwee CB, Mokkink LB, Knol DL, Ostelo RWJG, Bouter LM, de Vet HCW: Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res: An Int J Qual Life Aspects Of Treatment, Care And Rehabil. 2012, 21: 651-657. 10.1007/s11136-011-9960-1.CrossRef
33.
go back to reference Trevethan R: Self-assessment of foot health: requirements, issues, practicalities, and challenges. J Am Podiatr Med Assoc. 2009, 99: 460-471.CrossRefPubMed Trevethan R: Self-assessment of foot health: requirements, issues, practicalities, and challenges. J Am Podiatr Med Assoc. 2009, 99: 460-471.CrossRefPubMed
34.
go back to reference Queensland Industrial Relations Commission: Health Practitioners (Queensland health) Certified Agreement (No. 2). 2011, Brisbane: Queensland Government, http://www.health.qld.gov.au/eb/hp.asp Accessed 25th June 2013 Queensland Industrial Relations Commission: Health Practitioners (Queensland health) Certified Agreement (No. 2). 2011, Brisbane: Queensland Government, http://​www.​health.​qld.​gov.​au/​eb/​hp.​asp Accessed 25th June 2013
35.
go back to reference Queensland Health Statewide Diabetes Clinical Network: High risk foot form (manual): Statewide Diabetic Foot Working Group. 2011, Brisbane: Queensland Health, http://www.health.qld.gov.au/caru/networks/diabetes.asp Accessed 25th June 2013 Queensland Health Statewide Diabetes Clinical Network: High risk foot form (manual): Statewide Diabetic Foot Working Group. 2011, Brisbane: Queensland Health, http://​www.​health.​qld.​gov.​au/​caru/​networks/​diabetes.​asp Accessed 25th June 2013
36.
go back to reference Bowling FL, King L, Paterson JA, Hu J, Lipsky BA, Matthews DR, Boulton AJM: Remote assessment of diabetic foot ulcers using a novel wound imaging system. Wound Repair And Regeneration: Official Publication Of The Wound Healing Society [And] The European Tissue Repair Society. 2011, 19: 25-30. 10.1111/j.1524-475X.2010.00645.x.CrossRef Bowling FL, King L, Paterson JA, Hu J, Lipsky BA, Matthews DR, Boulton AJM: Remote assessment of diabetic foot ulcers using a novel wound imaging system. Wound Repair And Regeneration: Official Publication Of The Wound Healing Society [And] The European Tissue Repair Society. 2011, 19: 25-30. 10.1111/j.1524-475X.2010.00645.x.CrossRef
37.
go back to reference Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33: 159-174. 10.2307/2529310.CrossRefPubMed Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33: 159-174. 10.2307/2529310.CrossRefPubMed
38.
go back to reference Landis JR, Koch GG: An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977, 33: 363-374. 10.2307/2529786.CrossRefPubMed Landis JR, Koch GG: An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977, 33: 363-374. 10.2307/2529786.CrossRefPubMed
39.
go back to reference Scottish Intercollegiate Guidelines Network (SIGN): Management of diabetes: a national clinical guideline. Edited by: Scotland NQI. 2010, Edinburgh: SIGN Scottish Intercollegiate Guidelines Network (SIGN): Management of diabetes: a national clinical guideline. Edited by: Scotland NQI. 2010, Edinburgh: SIGN
40.
go back to reference Scottish Intercollegiate Guidelines Network (SIGN): Management of diabetes: a national clinical guideline. 2010, Edinburgh: SIGN Scottish Intercollegiate Guidelines Network (SIGN): Management of diabetes: a national clinical guideline. 2010, Edinburgh: SIGN
41.
go back to reference National Vascular Disease Prevention Alliance (NVDPA): Guidelines for the management of absolute cardiovascular disease risk. 2012, : National Stroke Foundation National Vascular Disease Prevention Alliance (NVDPA): Guidelines for the management of absolute cardiovascular disease risk. 2012, : National Stroke Foundation
42.
go back to reference UK Prospective Diabetes Study (UKPDS) Group: Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet. 1998, 352: 837-853.CrossRef UK Prospective Diabetes Study (UKPDS) Group: Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet. 1998, 352: 837-853.CrossRef
43.
go back to reference National Health & Medical Research Council (NHMRC) Guideline: National evidence based guideline for diagnosis, prevention and management of chronic kidney disease in type 2 diabetes. 2009, Canberra: Diabetes Australia National Health & Medical Research Council (NHMRC) Guideline: National evidence based guideline for diagnosis, prevention and management of chronic kidney disease in type 2 diabetes. 2009, Canberra: Diabetes Australia
44.
go back to reference Perkins BA, Olaleye D, Zinman B, Bril V: Simple screening tests for peripheral neuropathy in the diabetes clinic. Diabetes Care. 2001, 24: 250-256. 10.2337/diacare.24.2.250.CrossRefPubMed Perkins BA, Olaleye D, Zinman B, Bril V: Simple screening tests for peripheral neuropathy in the diabetes clinic. Diabetes Care. 2001, 24: 250-256. 10.2337/diacare.24.2.250.CrossRefPubMed
45.
go back to reference Smieja M, Hunt DL, Edelman D, Etchells E, Cornuz J, Simel DL: Clinical examination for the detection of protective sensation in the feet of diabetic patients. International Cooperative Group for Clinical Examination Research. Journal Of General Internal Medicine. 1999, 14: 418-424.CrossRef Smieja M, Hunt DL, Edelman D, Etchells E, Cornuz J, Simel DL: Clinical examination for the detection of protective sensation in the feet of diabetic patients. International Cooperative Group for Clinical Examination Research. Journal Of General Internal Medicine. 1999, 14: 418-424.CrossRef
46.
go back to reference Romanos MT, Raspovic A, Perrin BM: The reliability of toe systolic pressure and the toe brachial index in patients with diabetes. J Foot Ankle Res. 2010, 3: 31-10.1186/1757-1146-3-31.CrossRefPubMedPubMedCentral Romanos MT, Raspovic A, Perrin BM: The reliability of toe systolic pressure and the toe brachial index in patients with diabetes. J Foot Ankle Res. 2010, 3: 31-10.1186/1757-1146-3-31.CrossRefPubMedPubMedCentral
47.
go back to reference Pérez-Martin A, Meyer G, Demattei C, Böge G, Laroche JP, Quéré I, Dauzat M: Validation of a fully automatic photoplethysmographic device for toe blood pressure measurement. Eur J Vasc Endovasc Surg: The Official J Eur Soc Vasc Surg. 2010, 40: 515-520. 10.1016/j.ejvs.2010.06.008.CrossRef Pérez-Martin A, Meyer G, Demattei C, Böge G, Laroche JP, Quéré I, Dauzat M: Validation of a fully automatic photoplethysmographic device for toe blood pressure measurement. Eur J Vasc Endovasc Surg: The Official J Eur Soc Vasc Surg. 2010, 40: 515-520. 10.1016/j.ejvs.2010.06.008.CrossRef
48.
go back to reference Scanlon C, Park K, Mapletoft D, Begg L, Burns J: Interrater and intrarater reliability of photoplethysmography for measuring toe blood pressure and toe-brachial index in people with diabetes mellitus. J Foot Ankle Res. 2012, 5: 13-10.1186/1757-1146-5-13.CrossRefPubMedPubMedCentral Scanlon C, Park K, Mapletoft D, Begg L, Burns J: Interrater and intrarater reliability of photoplethysmography for measuring toe blood pressure and toe-brachial index in people with diabetes mellitus. J Foot Ankle Res. 2012, 5: 13-10.1186/1757-1146-5-13.CrossRefPubMedPubMedCentral
49.
go back to reference Holland-Letz T, Endres HG, Biedermann S, Mahn M, Kunert J, Groh S, Pittrow D, von Bilderling P, Sternitzky R, Diehm C: Reproducibility and reliability of the ankle-brachial index as assessed by vascular experts, family physicians and nurses. Vasc Med (London, England). 2007, 12: 105-112. 10.1177/1358863X07077281.CrossRef Holland-Letz T, Endres HG, Biedermann S, Mahn M, Kunert J, Groh S, Pittrow D, von Bilderling P, Sternitzky R, Diehm C: Reproducibility and reliability of the ankle-brachial index as assessed by vascular experts, family physicians and nurses. Vasc Med (London, England). 2007, 12: 105-112. 10.1177/1358863X07077281.CrossRef
50.
go back to reference Abbott CA, Carrington AL, Ashe H, Bath S, Every LC, Griffiths J, Hann AW, Hussein A, Jackson N, Johnson KE, et al: The North-West Diabetes Foot Care Study: incidence of, and risk factors for, new diabetic foot ulceration in a community-based patient cohort. Diabetic Med: J Br Diabetic Assoc. 2002, 19: 377-384. 10.1046/j.1464-5491.2002.00698.x.CrossRef Abbott CA, Carrington AL, Ashe H, Bath S, Every LC, Griffiths J, Hann AW, Hussein A, Jackson N, Johnson KE, et al: The North-West Diabetes Foot Care Study: incidence of, and risk factors for, new diabetic foot ulceration in a community-based patient cohort. Diabetic Med: J Br Diabetic Assoc. 2002, 19: 377-384. 10.1046/j.1464-5491.2002.00698.x.CrossRef
51.
go back to reference Monteiro-Soares M, Boyko E, Ribeiro J, Ribeiro I, Dinis-Ribeiro M: Risk stratification systems for diabetic foot ulcers: a systematic review. Diabetologia. 2011, 54: 1190-1199. 10.1007/s00125-010-2030-3.CrossRefPubMed Monteiro-Soares M, Boyko E, Ribeiro J, Ribeiro I, Dinis-Ribeiro M: Risk stratification systems for diabetic foot ulcers: a systematic review. Diabetologia. 2011, 54: 1190-1199. 10.1007/s00125-010-2030-3.CrossRefPubMed
52.
go back to reference Peters EJG, Lavery LA: Effectiveness of the diabetic foot risk classification system of the International Working Group on the Diabetic Foot. Diabetes Care. 2001, 24: 1442-1447. 10.2337/diacare.24.8.1442.CrossRefPubMed Peters EJG, Lavery LA: Effectiveness of the diabetic foot risk classification system of the International Working Group on the Diabetic Foot. Diabetes Care. 2001, 24: 1442-1447. 10.2337/diacare.24.8.1442.CrossRefPubMed
53.
go back to reference Widatalla AH, Mahadi SEI, Shawer MA, Elsayem HA, Ahmed ME: Implementation of diabetic foot ulcer classification system for research purposes to predict lower extremity amputation. Int J Diabetes In Developing Countries. 2009, 29: 1-5. 10.4103/0973-3930.50707.CrossRef Widatalla AH, Mahadi SEI, Shawer MA, Elsayem HA, Ahmed ME: Implementation of diabetic foot ulcer classification system for research purposes to predict lower extremity amputation. Int J Diabetes In Developing Countries. 2009, 29: 1-5. 10.4103/0973-3930.50707.CrossRef
54.
go back to reference Schaper NC: Diabetic foot ulcer classification system for research purposes: a progress report on criteria for including patients in research studies. Diabetes/Metab Res Rev. 2004, 20 (Suppl 1): S90-S95.CrossRef Schaper NC: Diabetic foot ulcer classification system for research purposes: a progress report on criteria for including patients in research studies. Diabetes/Metab Res Rev. 2004, 20 (Suppl 1): S90-S95.CrossRef
55.
go back to reference Oyibo SO, Jude EB, Tarawneh I, Nguyen HC, Armstrong DG, Harkless LB, Boulton AJ: The effects of ulcer size and site, patient’s age, sex and type and duration of diabetes on the outcome of diabetic foot ulcers. Diabetic Med: J Br Diabetic Assoc. 2001, 18: 133-138. 10.1046/j.1464-5491.2001.00422.x.CrossRef Oyibo SO, Jude EB, Tarawneh I, Nguyen HC, Armstrong DG, Harkless LB, Boulton AJ: The effects of ulcer size and site, patient’s age, sex and type and duration of diabetes on the outcome of diabetic foot ulcers. Diabetic Med: J Br Diabetic Assoc. 2001, 18: 133-138. 10.1046/j.1464-5491.2001.00422.x.CrossRef
56.
go back to reference Sheehan P, Jones P, Caselli A, Giurini JM, Veves A: Percent change in wound area of diabetic foot ulcers over a 4-week period is a robust predictor of complete healing in a 12-week prospective trial. Diabetes Care. 2003, 26: 1879-1882. 10.2337/diacare.26.6.1879.CrossRefPubMed Sheehan P, Jones P, Caselli A, Giurini JM, Veves A: Percent change in wound area of diabetic foot ulcers over a 4-week period is a robust predictor of complete healing in a 12-week prospective trial. Diabetes Care. 2003, 26: 1879-1882. 10.2337/diacare.26.6.1879.CrossRefPubMed
57.
go back to reference Zimny S, Pfohl M: Healing times and prediction of wound healing in neuropathic diabetic foot ulcers: a prospective study. Experimental And Clinical Endocrinology & Diabetes: Official Journal, German Society Of Endocrinology [And] German Diabetes Association. 2005, 113: 90-93. 10.1055/s-2004-830537.CrossRef Zimny S, Pfohl M: Healing times and prediction of wound healing in neuropathic diabetic foot ulcers: a prospective study. Experimental And Clinical Endocrinology & Diabetes: Official Journal, German Society Of Endocrinology [And] German Diabetes Association. 2005, 113: 90-93. 10.1055/s-2004-830537.CrossRef
58.
go back to reference Cardinal M, Eisenbud DE, Phillips T, Harding K: Early healing rates and wound area measurements are reliable predictors of later complete wound closure. Wound Repair And Regeneration: Official Publication Of The Wound Healing Society [And] The European Tissue Repair Society. 2008, 16: 19-22. 10.1111/j.1524-475X.2007.00328.x.CrossRef Cardinal M, Eisenbud DE, Phillips T, Harding K: Early healing rates and wound area measurements are reliable predictors of later complete wound closure. Wound Repair And Regeneration: Official Publication Of The Wound Healing Society [And] The European Tissue Repair Society. 2008, 16: 19-22. 10.1111/j.1524-475X.2007.00328.x.CrossRef
59.
go back to reference Warriner RA, Snyder RJ, Cardinal MH: Differentiating diabetic foot ulcers that are unlikely to heal by 12 weeks following achieving 50% percent area reduction at 4 weeks. Int Wound J. 2011, 8: 632-637. 10.1111/j.1742-481X.2011.00860.x.CrossRefPubMed Warriner RA, Snyder RJ, Cardinal MH: Differentiating diabetic foot ulcers that are unlikely to heal by 12 weeks following achieving 50% percent area reduction at 4 weeks. Int Wound J. 2011, 8: 632-637. 10.1111/j.1742-481X.2011.00860.x.CrossRefPubMed
60.
go back to reference Lavery LA, Armstrong DG, Murdoch DP, Peters EJG, Lipsky BA: Validation of the Infectious Diseases Society of America’s diabetic foot infection classification system. Clinical Infectious Diseases: An Official Publication Of The Infectious Diseases Society Of America. 2007, 44: 562-565. 10.1086/511036.CrossRef Lavery LA, Armstrong DG, Murdoch DP, Peters EJG, Lipsky BA: Validation of the Infectious Diseases Society of America’s diabetic foot infection classification system. Clinical Infectious Diseases: An Official Publication Of The Infectious Diseases Society Of America. 2007, 44: 562-565. 10.1086/511036.CrossRef
61.
go back to reference Lipsky BA, Berendt AR, Cornia PB, Pile JC, Peters EJG, Armstrong DG, Deery HG, Embil JM, Joseph WS, Karchmer AW, et al: Executive summary: 2012 Infectious Diseases Society of America clinical practice guideline for the diagnosis and treatment of diabetic foot infections. Clinical Infectious Diseases: An Official Publication Of The Infectious Diseases Society Of America. 2012, 54: 1679-1684. 10.1093/cid/cis460.CrossRef Lipsky BA, Berendt AR, Cornia PB, Pile JC, Peters EJG, Armstrong DG, Deery HG, Embil JM, Joseph WS, Karchmer AW, et al: Executive summary: 2012 Infectious Diseases Society of America clinical practice guideline for the diagnosis and treatment of diabetic foot infections. Clinical Infectious Diseases: An Official Publication Of The Infectious Diseases Society Of America. 2012, 54: 1679-1684. 10.1093/cid/cis460.CrossRef
62.
go back to reference Armstrong DG, Lavery LA, Harkless LB: Validation of a diabetic wound classification system. The contribution of depth, infection, and ischemia to risk of amputation. Diabetes Care. 1998, 21: 855-859. 10.2337/diacare.21.5.855.CrossRefPubMed Armstrong DG, Lavery LA, Harkless LB: Validation of a diabetic wound classification system. The contribution of depth, infection, and ischemia to risk of amputation. Diabetes Care. 1998, 21: 855-859. 10.2337/diacare.21.5.855.CrossRefPubMed
63.
go back to reference Oyibo SO, Jude EB, Tarawneh I, Nguyen HC, Harkless LB, Boulton AJM: A comparison of two diabetic foot ulcer classification systems: the Wagner and the University of Texas wound classification systems. Diabetes Care. 2001, 24: 84-88. 10.2337/diacare.24.1.84.CrossRefPubMed Oyibo SO, Jude EB, Tarawneh I, Nguyen HC, Harkless LB, Boulton AJM: A comparison of two diabetic foot ulcer classification systems: the Wagner and the University of Texas wound classification systems. Diabetes Care. 2001, 24: 84-88. 10.2337/diacare.24.1.84.CrossRefPubMed
64.
go back to reference Menz HB, Gill TK, Taylor AW, Hill CL: Predictors of podiatry utilisation in Australia: the North West Adelaide Health Study. J Foot Ankle Res. 2008, 1: 8-10.1186/1757-1146-1-8.CrossRefPubMedPubMedCentral Menz HB, Gill TK, Taylor AW, Hill CL: Predictors of podiatry utilisation in Australia: the North West Adelaide Health Study. J Foot Ankle Res. 2008, 1: 8-10.1186/1757-1146-1-8.CrossRefPubMedPubMedCentral
65.
go back to reference Fletcher AK, Dolben J: A hospital survey of the care of elderly patients with diabetes mellitus. Age & Ageing. 1996, 25: 349-352. 10.1093/ageing/25.5.349.CrossRef Fletcher AK, Dolben J: A hospital survey of the care of elderly patients with diabetes mellitus. Age & Ageing. 1996, 25: 349-352. 10.1093/ageing/25.5.349.CrossRef
Metagegevens
Titel
The Queensland high risk foot form (QHRFF) – is it a reliable and valid clinical research tool for foot disease?
Auteurs
Peter A Lazzarini
Vanessa Ng
Ewan M Kinnear
Maarten C Kamp
Suzanne S Kuys
Cameron Hurst
Lloyd F Reed
Publicatiedatum
01-12-2014
Uitgeverij
BioMed Central
Gepubliceerd in
Journal of Foot and Ankle Research / Uitgave 1/2014
Elektronisch ISSN: 1757-1146
DOI
https://doi.org/10.1186/1757-1146-7-7

Andere artikelen Uitgave 1/2014

Journal of Foot and Ankle Research 1/2014 Naar de uitgave