Item selection, scaling and construct validation of the Patient-Reported Inventory of Self-Management of Chronic Conditions (PRISM-CC) measurement tool in adults

Development of the PRISM-CC followed the Patient-Reported Outcomes Measurement Information System (PROMIS)^® Scientific Standard [19] and the COSMIN Study Design checklist for Patient-Reported outcome measures [22‐24]. The first step in development was to operationally reframe the seven TEDSS domain definitions to reflect perceived success or difficulty in self-managing each domain (Table 1). Item generation (Phase 1) and assessment of relevance and understanding to people living with chronic conditions (Phase 2) are previously reported [19]. Briefly, item development and selection were conducted by an international, multidisciplinary study team, which included clinicians, researchers and people living with chronic conditions. Potential items (N = 250) were generated for each domain using qualitative data employed to develop the TEDSS, and by examining items from other tools [9, 10]. After an initial conceptual review, 231 items were retained and administered to 40 persons living with chronic conditions to assess their relevance and understandability. Poorly performing items were discarded. Moderately performing items were further assessed during face-to-face cognitive interviews with ten of the 40 participants [25, 26]. We retained 100 items (11–17 items per domain—none from existing scales) for item selection, scaling, and validation.

Between February 2020 and April 2021 1,213 persons aged 18 years and over, who self-reported having at least one chronic condition, and who could read English, completed an anonymous, 20–30-min online survey. To be eligible, subjects had to select at least one of 13 conditions from a list (see footnote, Table 2) or specify “other” conditions in a free-text field. Recruitment sought a diverse sample with respect to sociodemographic characteristics and types of chronic conditions through posters displayed in health care settings; information distributed to patient groups, health charities and organizations; social media (Facebook and Twitter), and online advertising sites (e.g., Kijiji). Some (N = 159) were also recruited using Amazon Mechanical Turk (https://​www.​mturk.​com). We excluded 158 persons (13%) who had missing data on 50% or more of the items, for a final sample size of 1055. Those excluded were not statistically different than those included with respect to age, gender, household type, number of reported chronic conditions, self-reported health status, or self-reported mental health; however, they had lower educational attainment.

The study procedure adhered to the Canadian Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans, and ethics approval was obtained from the Nova Scotia Health Research Ethics Board (Romeo File #: 1025263). Informed consent was obtained from all survey respondents.

The survey included 100 potential PRISM-CC items in two versions with different item orders. The preferential response scale assessed difficulty: unable to do, with extreme difficulty, with much difficulty, with a little difficulty, easily, very easily (66 items). When this was not appropriate, two other response scales were used: (1) never, rarely, sometimes, often, almost always, always (24 items), and (2) strongly disagree, disagree, moderately disagree, moderately agree, agree, strongly agree (10 items). For each item, high scores indicated self-management success. Each item included a “not applicable” (NA) response option.

The survey also included sociodemographic questions (age, gender, country of residence, current living situation, and educational status), type(s) of chronic conditions, self-reported general health and mental health status, a question on impact of chronic condition(s) on daily life, and the 6-item Self-Efficacy in Managing Chronic Conditions (SEMCD) measure [27]. The SEMCD is widely used to evaluate patients’ confidence in managing chronic conditions and is strongly correlated with other measures of self-management [18, 28]. Self-reported general (and mental health) were measured with a single question “In general, would you say your (mental) health is”: excellent, very good, good, fair, or poor [29‐31]. Impact on daily life was measured with the question: “Overall, how much do you feel that your chronic condition(s) affect(s) your life?”: not at all, a little bit, moderately, quite a bit, or extremely.

As per our published protocol [19], analyses proceeded through item selection, model refinement and then assessment of construct validity. Confirmatory factor analysis (CFA) and item response theory (IRT) models provided complementary approaches to inform item selection, scaling, and assessment of construct validity of the PRISM-CC [32, 33]. Models appropriate for ordinal response scales were employed. CFA models were estimated using the R ‘Lavaan’ package based on pairwise polychoric correlations and diagonally weighted least squares [34, 35]. For IRT, multidimensional graded response models were estimated with full information maximum likelihood using the R ‘mirt’ package [36, 37].

Missing data resulted from both item non-response and “NA” responses. Items with high proportions of “NA” responses resulting from life contexts uncommon to many respondents (e.g., managing chronic conditions at work) were excluded; however, missing and “NA” responses are unlikely to be missing completely at random [38]. While our estimation methods handled item non-response, sensitivity analyses were conducted to assess the impact of non-random patterns of missingness on reported model parameters and fit statistics. Multivariate chained equations were used to impute 10 sets of missing and “NA” item responses using the “mice v2.9” package in R using all items in our final model as predictors [39]. Parameter estimates and fit statistics were then re-estimated and pooled across the 10 chained imputation data sets, and compared to our reported results.

The item selection process identified the best performing set of items to create a parsimonious measurement instrument for the seven PRISM-CC domains while differentiating between domains. Item selection proceeded through six iterations of modeling and item assessment. Only a few items were excluded at each iteration, and models were refitted before further decisions were made. At later iterations, items previously discarded were reconsidered by adding them back into more refined model versions to evaluate how they performed. Item selection decisions considered multiple statistical criteria combined with ongoing assessment of face and content validity by our interdisciplinary team. Cognitive interview data and qualitative analyses, which informed item development, were often used to inform decisions. Statistical information informing item selection decisions included: (1) low item variance or very low item-rest correlations; (2) high rates of item non-response or not applicable responses resulting from context-specific items; (3) weak standardized factor loadings (< 0.6) or weak discrimination parameters in the IRT models (< 1.35) [40, 41]; (4) examination of IRT item location parameters, item-category response function plots and item information plots to assess the contribution of each item to measuring the latent variable; (5) indications of problems with item performance based on the 1–2 largest modification indices at each model iteration, such as correlated errors (which violates the conditional independence assumption) or cross-loading of an item between domains (indicative of weak discriminant validity of an item); (6) evidence of differential item functioning by sociodemographic variables; and (7) areas of local strain based on residual correlations. A more detailed description of how each of these criteria were used is provided in Supplementary Appendix.

Once items for the PRISM-CC were selected, we assessed construct validity by assessing the extent to which data corresponded to our conceptual framework (structural validity), and hypothesis testing based on known relationships [42].

Structural validity was assessed by the fit of the data to correlated domain CFA and IRT measurement models reflecting our full conceptual framework, and individual measurement models for each of the seven domains. Internal consistency of each domain was assessed with Cronbach’s alpha and measures of marginal reliability. To assess global fit of models, we computed the Root Mean Square Error of Approximation (RMSEA), Squared Root Mean Residual (SRMR), Tucker Lewis Index (TLI) and Comparative Fit Index (CFI). Standard cut-offs widely used in the literature as indicative of good fit were used (SRMR < 0.05, RMSEA < 0.06, and TLI and CFI > 0.95) [43]. However, a RMSEA of 0.06 can indicate poor fit in a model with low standardized factor loadings (e.g., 0.40) while a RMSEA of 0.20 can indicate good fit in a model with high standardized factor loadings (e.g., > 0.90) [40]. So, we also assessed the fit of individual items using CFA standardized factor loadings, IRT item discrimination parameters, and the infit and outfit mean square fit statistics. Standardized factor loadings in the range 0.60–0.74 were interpreted as high, and values ≥ 0.75 were interpreted as very high [40]. IRT item discrimination parameters in the range 1.35–1.69 were considered high, and those ≥ 1.70 were considered very high [41]. Good infit and outfit statistics have values close to 1.0 and should be in the range of 0.5 to 1.5 [44]. Residuals between observed and model predicted inter-item correlations were examined to identify local areas of poor fit. To further assess structural validity, discrimination between latent variables was assessed by testing for collapsibility of domains that were highly correlated (≥ 0.80).

Non-reflective responses in online surveys can result in underestimation of model fit [45, 46]. Of the 1,055 respondents included in our analysis, 5% spent, on average, less than 3.6 s per question, giving plausibility to this concern. Accordingly, sensitivity analyses were conducted to assess the potential impact of non-reflective responses on the model fit of each PRISM-CC domain (see Supplementary Appendix).

Finally, as detailed in our protocol [19], we further assessed construct validity of the PRISM-CC using structural equation models to test hypothesized associations between each domain and variables previously shown to be associated with self-management. Briefly, we hypothesized that the SEMCD would be positively associated with all domains; higher education would be associated with higher Process, Resource, and Healthy Behaviors domain scores; a higher number of chronic conditions would be associated with lower Disease Controlling domain scores; higher self-perceived general health would be associated with higher Disease Controlling domain scores; and higher self-perceived mental health would be associated with higher Internal and Social Interaction domain scores.

Thirty-six items, with 4–8 items per domain, were selected for the PRISM-CC (Table 1). Early in the selection process, items were most frequently eliminated because of poor item-rest correlations and weak factor loadings, combined with evidence of poor face validity. Later in the item selection process, content validity (e.g., redundant items), poor discriminant validity, and evidence of differential item functioning where more common reasons for item elimination. We selected items for each domain that best captured the construct while resulting in very good to excellent model fit. See Supplemental Appendix for detailed description of the item selection.

Face and content validity of selected items was assessed by our team to be high for all domains except the Resource domain, which was assessed as moderate because some content of the domain definition was not captured. None of the selected domain items explicitly ask about locating or using community and/or social services, or informal supports, though the fourth item may do so implicitly. Two potential items about self-perceived difficulty in finding “community” and “social” services were eliminated due to ambiguity in meaning. Potential items to measure difficulty accessing informal supports (e.g., “When I need to, I ask family or friends to help me.”) were eliminated due to poor discriminant validity with the Social Interaction domain.

Evidence for the structural validity of the PRISM-CC is strong. The data provided excellent fit to our hypothesized correlated factors CFA model and corresponding multidimensional IRT graded response model. The seven factor CFA model had an RMSEA of 0.057 (90% CI 0.055–0.060), an SRMR of 0.039, a CFI of 0.951, and a TLI of 0.947. Table 3 shows that items for all domains had high measurement quality. All standardized loadings were high or very high (19 of 36 are very high), and nearly all IRT graded response item discrimination parameters were very high (34 of 36). Infit and outfit statistics were generally close to 1.0, and all met fit criteria. Measurement quality for items in the Healthy Behaviors domain was weaker, on average, than other domains.

Evidence of model fit and internal consistency was also assessed individually for each PRISM-CC domain (Table 4). Internal consistency, as measured by Cronbach’s alpha and marginal reliability, exceeded 0.80 for all domains. The CFI, TLI and SRMR fit statistics indicated excellent fit for all domains in both CFA and IRT models. However, the upper confidence intervals for RMSEAs for both the CFA and IRT models were higher than the commonly used cut-off of 0.06. However, examination of residual correlations did not indicate localized areas of poor fit in any domain, and modification indices did not identify substantial model misspecifications. Sensitivity analyses (see Supplemental Appendix) indicated that non-reflective responses resulted in underestimation of the fit of the PRISM-CC. Dropping potentially non-reflective respondents increased standardized factor loadings by 0.05 or more, and substantially improved indices of model fit, including the RMSEA. Sensitivity analyses re-estimating all CFA models using multiply imputated missing and not applicable responses resulted in small changes to standardized factor loadings and fit statistics but did not alter any study conclusions.

Estimated correlations between PRISM-CC domains were positive and exceeded 0.50, for both the CFA and IRT graded response models (Table 5). The Healthy Behaviors domain had particularly high estimated correlations with the Internal (~ 0.85) and the Activity (~ 0.87) domains. However, the conceptual distinction between these domains was clear, and statistical tests for each pair, comparing a two-factor model with a constrained one-factor model, indicated better fit for the two-factor models. Furthermore, modification indices for the CFA model did not identify unspecified cross loadings contributing to high correlations between these domains.

Testing of pre-specified hypotheses provided further evidence of the construct validity. All hypotheses were confirmed (Tables 5 and 6). There were strong and statistically significant associations between the SEMCD and each of the PRISM-CC domains (Table 5). As shown in Table 6, higher education was associated with higher Process, Resource, and Healthy Behavior domain scores; a larger number of chronic conditions was associated with lower Disease Controlling domain scores; self-perceived general health was associated with higher Disease Controlling domain scores; and higher self-perceived mental health was associated with higher Internal and Social Interaction domain scores.

The PRISM-CC includes 36 items with 4–8 items per domain. All domains, with the partial exception of the Resource domain, have good face and content validity when measured against our conceptual framework. The operational definition of the Resource domain was “self-perceived success in seeking, pursuing and/or managing needed formal or informal supports and resources”. Items retained in this domain measure success in managing formal health care but do not explicitly capture success managing informal supports (help from family and friends) or accessing community and social services. Items generated to measure management of informal supports had poor discriminant validity with the Social Interaction domain, so were not retained. This suggests that management of informal supports, while important to patients, is indistinguishable from the management of social contacts and networks, which is captured by the Social Interaction domain. As well, items expected to measure difficulty managing community and social service supports performed poorly and were not retained due to possible differences among respondents in defining social and community services across different contexts.

Evidence of the PRISM-CC’s structural validity, one aspect of construct validity, is strong. Our data had very good to excellent fit to the pre-specified, correlated factor CFA and IRT models. While estimated upper confidence intervals for the RMSEA in many of the CFA and IRT domain-specific models were above the widely used cut-off of 0.06, we believe this is not of much concern. An RMSEA greater than 0.06 may indicate good fit when standardized factor loadings are greater than 0.70, which is the case for most of our items [40]. Moreover, sensitivity analyses of the potential impact of non-reflective responses to survey questions suggest that our estimates of standardized factor loadings and model fit are likely conservative relative to what would be obtained from application of the PRISM-CC in clinical settings.

Sample characteristics may limit the generalizability of results. While we had a large and diverse sample, data came from people living in the community, not from clinical settings. The online nature of the survey contributed to the younger cohort whose chronic disease patterns likely differ from older and/or clinical populations. To address this, further assessment of measurement equivalence across patients with different socio-demographic and chronic disease attributes is needed.

Confirmation of a priori, hypothesized associations between PRISM-CC domains and variables, as detailed in the study protocol [19], provides additional evidence of construct validity. While this is encouraging, additional evidence of validity is needed. The critical next step is to assess the ability of the PRISM-CC to inform patient care, measure clinically meaningful change, and evaluate the critical ingredients of self-management interventions.

While strong evidence for internal consistency of each domain was confirmed, assessment of test–retest reliability, measurement error and responsiveness of the PRISM-CC remains unassessed.

Self-management is the work done by people themselves [3, 5‐7]. The TEDSS articulates that work in seven domains, identified and described by people living with chronic conditions and labelled using non-medical language [9, 10]. Built on this foundation, the PRISM-CC is expected to be understandable, intuitive, and relevant to patients’ experiences. It is not a measure of underlying ability or capacity, but rather patients’ personal assessment of their success/difficulty within each domain. For example, difficulty in the Disease Controlling domain represents perceived difficulty managing “medications and treatments, monitoring symptoms and limiting complications” regardless of whether it is the result of managing a complex medication and treatment regimen or having insufficient knowledge to manage a single medication. In either case, and regardless of the underlying reason, perceived difficulty is uncovered so that self-management support can be offered.

Measuring perceived difficulty, the PRISM-CC stands in contrast to tools that focus on single constructs, such as self-efficacy or patient activation, believed to bolster self-management [27, 47]. However, many other factors, such as health literacy, social support networks, access to transportation, use of technologies or tools, and the development of specific skills or strategies also enable self-management [48‐50]. The PRISM-CC facilitates a patient-centered approach in which patients identify areas of difficulty. If indicated, contributing factors can then be investigated. Given that the success of most self-management support interventions is dependent on the actions of patients, and often necessitates behaviour change, patients are most likely to engage in interventions that are of perceived difficulty [51].

With its multi-dimensionality and focus on patient-perceived difficulty, the PRISM-CC is positioned to facilitate tailored interventions, addressing a concern with “one size fits all” self-management interventions [14, 52, 53]. Its seven domains explicate, from the perspective of the patient, strengths or challenges in self-managing traditional areas of medical (Disease Controlling and Healthy Behaviours domains), roles (Activities and Social Interaction domains) and emotional management (Internal domain) while also assessing problem solving (Process domain) and aspects of health navigation (Resource domain), providing valuable insight into how to tailor support [6]. For example, a newly diagnosed patient who identifies difficulty finding information, problem solving and decision-making (Process domain) but no difficulty with routine exercise and healthy food choices (Healthy Behaviours domain) needs different support than a patient with multiple conditions and limited mobility who self-reports greatest difficulty in the Activities and Social Interaction domains. Using the PRISM-CC to pinpoint areas of patient concern coupled with skillful clinical assessment, tailored interventions can be offered, and appropriate referrals made. Success or difficulty in some domains will likely cluster together, as indicated by the high correlations between them.

PRISM-CC also has the potential to improve the functioning and patient-centeredness of multidisciplinary and interprofessional teams. Assigning disciplinary case managers based on patient needs and provider scope of practice are two examples. This work could be strengthened using the Team Assessment of Self-Management Strategies (TASMS), also based on the TEDSS framework, to better understand the types of self-management support teams provide [54].

Finally, the PRISM-CC is expected to be feasible for both research and clinical settings, and for individuals living with one or multiple chronic conditions. It is short to administer (36 items) and does not need specific training. Visualized reporting tools are under development, and licensing and apps for administration and linkage to electronic clinical records are being investigated.