Efficient and precise Ultra-QuickDASH scale measuring lymphedema impact developed using computerized adaptive testing

We analyzed patient-reported outcome (PRO) data collected from 285 English speaking American adults with a diagnosis of lymphedema affecting the upper extremity in the lymphedema clinic of the University of Texas MD Anderson Cancer Center between 2016 and 2020. Mean age was 57.52 years and 197 (69.12%) were adults (< 65). The mean score for International Society of Lymphology stage was 1.98. All patients in the study had lymphedema diagnosed by measurements, including bioimpedance spectroscopy using the LDex score or limb volume using a perometer, and/or imaging, including indocyanine green (ICG) fluorescent lymphography or radionucleotide lymphoscintigraphy. The mean LDex score and limb volume difference were 22.46 and 21.73%, respectively, with cutoff thresholds of 7 for the LDex score [15‐17] and 5% for limb volume measurement index used in a clinic setting for lymphedema diagnosis [18, 19].

The QuickDASH PRO measures patients’ symptoms and ability to perform activities using their upper limbs during the previous week. The QuickDASH has 11 items scored on a 5-point Likert scale from 1 to 5 with a strong test–retest reliability [2‐5, 20] and internal consistency reliability [4]. A higher item score indicates a higher level of disability or greater symptom severity [21].

We assessed a series of assumptions to evaluate the scale fit to the IRT model. These assumptions included unidimensionality of the scale, scalability of items, and local independence of items. We also assessed potential issues arising from disordered items or differential item function (DIF) within the dataset. These critical terms and the corresponding mechanisms and principles behind them were shown in more details in Online Appendix B [13].

We conducted confirmatory factor analysis (CFA) with maximum-likelihood estimator to confirm the factor structure of the QuickDASH scale, and then assessed the fit of this model based on five main indices from the goodness of fit and residual fit statistics. We interpreted the fit of models based on relevant indicators with corresponding recommended acceptable thresholds to assist evaluation, that is, Tucker-Lewis index (TLI ≥ 0.9), comparative fit index (CFI ≥ 0.9), root mean square error of approximation (RMSEA < 0.08), root mean square of the residual (RMSR < 0.08) and a non-significant chi-square test (p > 0.05), however, we were mindful of type I error caused by large sample sizes in chi-square analyses [14]. We conducted Mokken analysis to further investigate the dimensional structure of the model and establish the scalability of each item [22]. Items with low scalability (Loevinger’s H < 0.30) were eliminated from further analysis [13].

We then fitted the data to Samejima’s graded response model (GRM) [23]. The GRM is suitable for developing item banks for CAT [24]. Local dependency was assessed using Yen’s Q3 with a residual correlation cut-off of + 0.20 [25]. Disordered thresholds were collapsed and rescored into adjacent categories based on proximity and logical anchor semantics. The data were re-analyzed with the eligible items left after all assumptions of IRT had been met. The fit of polytomous GRM was assessed using M2 statistics [26].

After the remaining items were calibrated to establish a bank of items using the GRM, personalizing patient assessment became possible using CAT. CAT automatically administers an item that matches the patient’s level of symptoms or functional ability based on their prior responses. In contrast to the fixed-length QuickDASH version, the CAT QuickDASH version scale can be of varied length, meaning that the specific items administered will differ from patient to patient during this adaptive testing process [27]. Specifically, the first item with the greatest information function at the distribution mean was administered by the CAT algorithm to estimate the latent trait of the lymphedema patient. After scoring based on the patient’s prior answers, the CAT algorithm will determine which is the most appropriate test question that the patient should be administered next. This estimation process will repeat until a pre-set “stopping rule” is reached. The max posterior-weighted information (MPWI) was chosen as the item selection method and the Bayesian expected a posteriori (EAP) with a prior distribution of N (0, 1) was used as theta estimator. The normal IRT scaling constant was set at 1.7 and the theta scale ranged from − 4 to 4. The excellent performance of these widely used parameter settings for CAT simulation with polytomous items has already been demonstrated in previous studies [28, 29]. In this study, we conducted CAT simulations for 500 repetitions each time at the stopping rule of standard errors (SE) at 0.32, 0.45, and 0.55, respectively, to explore the most efficient or precise test. As the inversed relationship between marginal reliability and SE is illustrated as reliability = 1 − SE^²[13], we performed three CAT simulations with different reliability of 0.9, 0.8, and 0.7 at the population mean of 0 and population standard deviation (SD) of 1.

The CFA was conducted with the “lavaan” package; the DIF detection was performed with “lordif” package; an IRT analysis was carried out with the “mokken” and “mirt” packages. The FIRESTAR code generator was adopted to simulate CAT administration [30]. All analyses were performed in the R Statistical Computing Environment [31].

Table 1 of CFA presents the information on the item descriptive statistics and factor loading for the QuickDASH scale. Results show that all the factor loadings are above the cutoff point of 0.3 and loaded on the same factor, indicating adequate loadings and unidimensional structure of the QuickDASH PROM.

Due to the local independence issue identified in the later analysis, the finalized CFA (χ² = 132.67, df = 35, p < 0.00) with 10 items substantially improved the model fit according to the goodness of fit statistics (TLI = 0.93, CFI = 0.95, RMSEA = 0.1, RMSR = 0.04),compared with the initial CFA (χ² = 220.6, df = 44, p < 0.001) with 11 items (TLI = 0.89, CFI = 0.91, RMSEA = 0.12, RMSR = 0.05).

Results from the Mokken analysis validated the unidimensional structure identified by CFA. Loevinger’s H coefficient for each item was greater than the recommended threshold for the entire scale and its constituent items (Table 2).

As unidimensionality and scalability assumptions for the IRT model were met through CFA and Mokken analyses, and no DIF items were detected in the groups of adults and older adults (≥ 65), the GRM based on the IRT framework was conducted using all 11 items. The estimated parameters of discriminations (a) and difficulty (b) are presented in Table 3 and utilized to illustrate the relationship between the overall disability level and the corresponding item. To facilitate the interpretation, these parameters were transformed into Z-scores with a mean of 0 and an SD of 1. The assumption of Local independence of items was assessed within the GRM. The largest residual correlation among item 9 (severity of pain) and item 11 (sleeping difficulty due to pain) (Yen’s Q3 = 0.46) was above the acceptable threshold of + 0.2 [32]. Item 11 (sleeping difficulty due to pain) was therefore removed from further analysis completely.

Additionally, initial GRM analysis with 11 items detected the issue of disordered response categories for item 5 (cutting food) based on its item characteristic curve. And the new rescored item 5 (cutting food) with 4 response categories is displayed in Fig. 1.

The GRM was re-analyzed with the remaining 10 items. Results of the residual correlation indicated the assumption of local independence of items had been reasonably met. Estimated parameters for the 10 items also are shown in parentheses in Table 3. All the 10 items left had a strong level of discrimination (a), indicating they are better at distinguishing between patients at specific disability levels.

The test information curve of the GRM with the remaining 10 items (Fig. 2), calculated by accumulating each item information together, showed that the entire QuickDASH instrument provides much more information for respondents with a higher level of disability symptom due to the peak of the curve is located above the average theta (θ) 0. The latent trait of patients with disability symptoms above the average theta level (θ) 0 will be precisely estimated through this instrument.

The results of the average number of items used for each time, the correlation between thetas (θ), mean SE, item mean, item media, item range are summarized in Table 4. During the 500 iterations, there 78 participants with SEs were higher than the pre-specified SE of 0.32.

Among them, item 2 (doing heavy chores) and item 8 (limiting work or daily activities) were most exposed during the CAT simulation due to their more item information providing (Fig. 3). Table 5 shows that 72.78% of the information provided by Items 2 and 8 were centered on the theta range of (− 2, + 2). The estimates of the level of QuickDASH trait score provided by the simulated CAT algorithm and the original QuickDASH trait score derived from the fixed-length questionnaire correlated highly up to 0.98 with mean score of − 0.01 (SD = 0.97), 0.97, and 0.96, respectively.

Tables 6 and 7 present the comparison results of participant score among these three versions of established DASH. Full QuickDASH had the highest mean participant score of 0.001 (SD = 0.96); the root mean square deviation (RMSD = 0.19) and SD of difference (0.19) between CAT and full QuickDASH comparison were lower. Both full QuickDASH and Ultra-QuickDASH provided much more information for participants with disabilities above the average level (θ) in Fig. 4.

We showed that the QuickDASH could be made to fit the IRT model, with minor modification, for evaluation of limb function in patients with upper extremity lymphedema. Once fitted to the IRT model we demonstrated that the assessment length can be dramatically reduced without sacrificing assessment accuracy using either CAT or a 2-item short form. The demonstrated strengths of the CAT approach in improving efficiency and precision in this study are consistent with the findings in previous studies [13, 14].

Studying only an American sample substantially exempts the study from the issues of DIF resulting from cultural diversity and hence provides a suitable item bank to be used within the US society setting. Cronbach alpha value (α = 0.93) from CFA indicated the excellent internal consistency in the entire QuickDASH scale. However, after applying the GRM to fit the data, the Yen’s Q3 value showed that items 9 (severity of pain) and 11(sleeping difficulty due to pain) violated the assumption of local independence of items. As many other reasons can attribute difficulty in sleeping, this may explain the reduced item information in item 11 (sleeping difficulty due to pain) compared to item 9 (severity of pain).

The excellent performance of the CAT simulation with the remaining 10 items provides useful information for the QuickDASH instrument in clinical practice. It is foreseeable that the developed CAT QuickDASH, as a supporting instrument for healthcare providers’ decision making regarding individual patient care, not only reduces the burden of patient-reported assessment and facilitates quick data collection and storage but also further promotes the enforceability and actionability of feedback and ultimately benefits patients with all upper extremity disorders from improved health care and research.

Also, we note that items 2 (doing heavy chores) and 8 (limiting work or daily activities) were most frequently administered during the CAT simulation. As the ability to do heavy labor may invoke swelling for lymphedema patients and activities of daily living were most affected by lymphedema [33], this reasonably explains why these two highly discriminative items dominated the information of the QuickDASH scale. The two items that dominated the CAT administration suggest that a reasonable ultra-short and technology-free version of QuickDASH can be developed only including items 2 and 8. Furthermore, results indicate that the correlation among the factor score of level of disability for lymphedema patients calculated from the QuickDASH including 11 items, and the Ultra-QuickDASH version only containing these two best items, was exceptionally high (r = 0.90). In this way, it is more operable for these health institutions or clinics that are not familiar with the CAT algorithm as they can use a super shorter and more accurate Ultra-QuickDASH questionnaire.

This study comes with several limitations. First, negative results of residual correlation between items come out from the assumption of local independence of items test revealed the possibility of multidimensional structure of the QuickDASH data, which just provides one plausible explanation for slightly high RMSEA (0.09) for the QuickDASH scale with 10 items. Further research is warranted to investigate the reason behind this and refine the QuickDASH instrument. Second, to address the local dependency issue, we removed the item with lower item information from further analysis directly and have not compared with the other widely used method of collapsing the items into a testlet on the possible influence on the analysis results [14]. Third, the analysis results are based on the data collected from the MD Anderson Cancer Center institute only. Additional data from other clinical centers are needed to externally validate, even update these findings, and further promote the application of Ultra-QuickDASH into clinic practices widely. Fourth, a cross-cultural adaptive test of different language versions of the Ultra-QuickDASH scale on measuring disability and symptoms related to lymphedema needs to be conducted in future research although DIF is not applicable in this study. Fifth, the relatively small number of items used to calibrate the item bank will slightly affect the precision of underlying construct estimation during CAT simulation [14]. Sixth, the sample size is relatively small (n = 285), however, the distribution of DASH score (factor score of disability) was normal with acceptable skewness (0.28) and kurtosis (− 0.34) [34], which may suggest that item parameters will be stable in larger populations. Seventh, we wish to caution users that the reduced length version inevitably will exclude some relevant questions from participants. While we demonstrate that this has a limited impact on DASH scores at the population level, it is foreseeable that some individual scores may differ substantially between the full-length DASH and both the CAT and fixed-length Ultra-QuickDASH versions. Additionally, The Ultra-QuickDASH provides less information on assessment participants than the complete QuickDASH and is not recommended in a situation where assessment reliability should be prioritized over brevity.

By utilizing CAT simulation based on the IRT framework to shorten the QuickDASH substantially, we found that a more efficient and precise estimation of disability level and symptom severity for American lymphedema patients can be achieved. In the meanwhile, the Concerto, as an emerging open-source CAT delivery platform, makes this CAT QuickDASH application in a real clinic setting possible [35]. All the improvements achieved will facilitate the PROM development and ultimately improve the health care and research to benefit patients. Moreover, the developed Ultra-QuickDASH mainly consisting of two best performing items and maintaining efficient and accurate estimations could be used as a CAT technology-free version of QuickDASH. Its application and promotion can break the obstacles of complex technology on health care professionals and providers on the use of CAT QuickDASH, making this super shortened instrument more convenient to apply into routine clinic practices.