Identifying the minimal important difference in patient-reported outcome measures in the field of people with severe mental illness: a pre–post-analysis of the Illness Management and Recovery Programme

We performed pre- and post-tests to measure the effect of attending the IMR programme on different PROMs. To examine which PROM captured the most relevant and meaningful change, we used the concept of the MID [17, 20‐22]. Estimating MIDs of the PROMs was based primarily on an anchor-based method but supported by distribution-based methods. We used an additional index to assess whether the effect of the IMR programme in a given PROM was large or small seen from a patient perspective, hence in terms of the MID. In this way, we arrived at the effect/MID as an index. An effect/MID of ≥ 1 indicates that the effect is at least the MID: the higher the index the more patients have a change score above the MID. Typically, within-patient changes are normally distributed around the mean change at group level. By comparing the index across different outcome measures, we could identify on which PROM the participants improved the most from the IMR programme. We considered that the PROM with the highest effect/MID index represents the most relevant and meaningful change as a result of the IMR programme. The study population consisted of participants from the e-IMR trial [23, 24]. Eligible participants met the following criteria: above 18 years of age, capable of giving informed consent and meeting the Dutch severe mental illness criteria according to Delespaul [25]: being diagnosed with a psychiatric disorder that causes, and is due to, serious impairment in social and/or occupational functioning that lasts longer than at least a couple of years and necessitates coordinated multidisciplinary care. The IMR programme was delivered groupwise in Dutch mental health institutions and lasted about 1 year for all the groups. Further information on the e-IMR trial is published elsewhere [23, 24].

In this study, we used the pre–post-data that were gathered in the e-IMR trial between January 2015 and October 2016 [23]. The first author and research assistants sampled all data in structured interviews. Face-to-face interviews were held, because we estimated that too many participants would not respond to telephone or online questionnaires. In the population of people with severe mental illness, low computer experience, skills [26] and availability [24] exist. Furthermore, this population is known for their cognitive impairments [27]. We decided to use the advantages of face-to-face interviewing: spoken language that can be better understood; the possibility of responding to misunderstanding and probing for complete answers and using cards with answering options to overcome memorizing difficulties [28]. The interviewers tried to create an easy-going atmosphere using small talk between the different questionnaires. The interviews took 30–60 min and data were sampled using the following six PROMs:

During the pre-test, participant characteristics were also sampled: age, gender, psychiatric diagnoses conforming to DSM-IV criteria, psychiatric and somatic comorbidities, treatment history, cultural background, housing, socioeconomic status and highest education (see Table 1). At the endpoint, non-completers were identified as those who attended fewer than 50% of the IMR programme sessions.

Analyses were conducted using SPSS software, version 23 [44]. To determine the pre–post-effects of the IMR programme on all PROMs, we performed mixed-model multilevel regression analyses, taking into account the clustering of participants. Because the IMR programme was delivered via group sessions, the participants were clustered in these groups. This method automatically uses the ‘missing at random’ assumption to handle missing data. Random effects on cluster and individual participants nested within the cluster and fixed main effects for time trend were included in the model. The analyses were executed according to the intention-to-treat principle. To take into account the influence of covariates, the pre–post-effects were controlled for the participant characteristics one by one, including for non-completers. The pre–post-effects in the PROMs were used to estimate the effect/MID index.

To estimate a MID, both anchor-based and statistical distribution-based methods are recommended [17, 20‐22]. The anchor-based method uses a criterion that measures a concept that health professionals are familiar with and is widely used in assessing patients’ health status [20], such as clinical endpoints, global transition questions or QoL measures [17]. As there is no such widely used criterion in mental health, we searched for a criterion in our own data that captures the richness and variation of the construct of a QoL measure [17]. We examined four QoL anchor candidates: the endpoint data on the Rand-HC global transition question and the change scores on MANSA-1, the total MANSA and Rand-GHP. Rand-HC and MANSA-1 were only used in the search for an anchor. The strength of the association between the anchor and the PROM needs to be determined, because low or no correlation can provide misleading information [17, 45]. A correlation of at least 0.30–0.35 is recommended [17], therefore, Spearman’s correlation coefficients between the anchor candidates and the PROMs were calculated. Outliers should not drive a correlation to a significant level. In SPSS, scores above 2.58 times the standard deviation (SD) are assigned as probable outliers [46]. Probable outliers were assessed on their appropriateness and impact on the correlation and a decision was made about removing or recoding to a reasonable level [47, 48]. The anchor candidate with the highest correlation with the change scores in most PROMs was considered to be the best anchor and, therefore, will be used in the MID-anchor calculations.

Estimation of the MID-anchor proceeds as follows: the scores on the anchor were used to categorize participants into five groups that reflected relevant and meaningful change (large negative, small negative, no, small positive, large positive); the mean of the four differences between change scores in the PROMs for two succeeding change groups is the PROM’s MID-anchor [20]. The MID-anchor was used to estimate the effect/MID-anchor index.

It is recommended that the MID be estimated primarily by anchor-based methods [17, 49] and to use distribution-based methods as supportive information [17]. Therefore, we examined two statistical distribution-based MID methods based on the effect size (ES) and standard error of measurement (SEM) of the PROM [17]. The ES estimates the effect of the intervention related to the SD, with ½ES as standard for estimating the MID [17, 20‐22, 50]. In MID studies, the SDs of the baseline scores and change scores are used to estimate the ES [21, 50]. A change score is the endpoint score minus the baseline score of a participant. The SD of the change scores (SD_c) relates to between-patient variation in change scores. Our index (effect/MID-SD_c) is 2 × effect/SD_c, which is also known as the standardised response mean [45, 51]. To calculate the effect/MID-SD_c index, we used the estimated effects from the mixed-model analyses.

The SEM is computed using the SD and the test–retest coefficient index: SD × √(1 − r_xx) [20, 28, 47, 52]. To estimate the SEM of the PROMs used in our study, we used the SD and r_xx reported in psychometric studies of the PROMs in populations comparable to ours as much as possible. A change smaller than the SEM is likely to be a result of the measure’s unreliability rather than a true observed change, therefore, the PROM’s MID based on the SEM (MID-SEM) is equal to the SEM of the PROM. The MID-SEM and the estimated effects from the mixed-model analyses were used to estimate the effect/MID-SEM index.

When covariates modified the effects, we re-estimated the effect/MID indexes and stratified the participants according to the modifying covariate.

Ten IMR programme groups entered the trial, totalling 91 potential participants, 60 of whom (66%) participated. Baseline characteristics of the participants are presented in Table 1: 36 participants (60%) were female and 15 (25%) were lost to follow-up. In total, 45 participants completed the post-test measurements, 25 (56%) of whom were female; 14 (23%) were non-completers, meaning that they attended less than 50% of the IMR programme sessions.

Table 2 shows the estimated effects, which were significant for all the PROMs: illness management (IMRS), recovery (MHRM), self-management (PAM), burden of symptoms (BSI), QoL (MANSA) (p < 0.01) and general health perception (Rand-GHP) (p < 0.05). Only the gender covariate modified effects significantly for Rand-GHP (p < 0.01). For males, the estimated effects were significant only for the MHRM (p < 0.05), whereas for females, they were significant for all the PROMs (p < 0.01; see Table 2).

The results of the MID analyses are shown in Table 2. The change scores on all the PROMs were normally distributed. The health change measure (Rand-HC) showed no significant correlations to the other PROMs and was therefore considered to be a non-feasible anchor. Compared to the QoL measures MANSA-1 and the MANSA, the general health measure Rand-GHP showed the most frequent, highest and statistically significant (p < 0.01) correlations with the following measures: illness management (IMRS), self-management (PAM), recovery (MHRM), burden of symptoms (BSI), and QoL (MANSA) (see Table 3). Two outliers drove the correlation between Rand-GHP and the BSI. Examination identified the outliers as true scores and to assess their impact on correlation they were recoded into twice the SD_c (= 1.02), after which the correlation remained significant (p < 0.01; see Table 3). Rand-GHP was selected as the anchor.

We categorized the participants in five change groups. The change scores in the Rand-GHP vary from − 35 to + 40 in steps of 5 (see Table 4). To form the five change groups, we estimated that each group consists of participants that have three consecutive scores in steps of 5, so the score difference between succeeding change groups was 15, which is comparable to the SD_c of 17.4. In the ‘no change’ group, we categorized participants with change scores around zero (− 5, 0, 5). We subsequently defined the other groups and the remaining score + 40 was assigned to the ‘large positive change’ group. This resulting five change groups are: large negative (n = 3), small negative (n = 7), no (n = 18), small positive (n = 10), and large positive change (n = 7) (see Table 4).

The effect/MID-anchor index was highest in the recovery measure (MHRM), with a value of 1.29.

Regarding the supportive MIDs, the MID-SD_c was calculated for the PROMs (see Table 2) and the effect/MID-SD_c index was highest in the MHRM with a value of 1.33. The MID-SEM values for the PROMs were equal to the SEM, which we calculated for the PROMs with data from referent studies (see Table 5). The effect/MID-SEM index was also highest in the MHRM with a value of 1.40.

Owing to the modifying effect of the gender covariate, we also stratified the MID calculations (see Table 2). For the stratified calculations of the MID-anchor, we reorganized the change groups into four groups based on the 15-point difference, because the male/female proportions in the different change groups were skewed (see Table 4). Males were overrepresented in the negative change groups and females were overrepresented in the positive change groups. Only one male was present in the large positive change group and females were absent in the large negative change group. For the groups with male participants we merged the two positive change groups, resulting in large negative (n = 3), small negative (n = 4), no (n = 10) and positive (n = 3) change groups; for the groups with female participants, we omitted the large negative change group, resulting in small negative (n = 3), no (n = 8), small positive (n = 8) and large positive (n = 7) change groups. The mean differences between the four change groups estimated the MID-anchors for males and females separately. The effect/MID-anchor index was highest for males (0,88) in the self-management measure (PAM) and for females (2,63) in the recovery measure (MHRM).

For the supportive MIDs, the MID-SD_c was stratified for males and females. The effect/MID-SD_c indexes for both males (0.83) and females (1.71) were highest in the MHRM. The MID-SEM for males and females separately was not calculated, because the referent studies did not provide data for males and females. The effect/MID-SEM indexes for both males (0.89) and females (1.76) were highest in the MHRM.

The strength of our study is that we contributed to the scientific literature on PROMs and explored the use of MIDs in the field of severe mental illness. We need to be cautious about drawing definite conclusions based on our findings because of the relatively low sample size and the significant gender confounder. The statistical power of the results is low and our sample might not be good representation of the population of persons with severe mental illness. More men living in a supported housing facility might coincidently determine the variance in our observed scores. In a confirmative trial or in other existing datasets with a bigger sample, this study might necessarily be repeated. Although our sample size was small, it was large enough (> 40) to detect correlation coefficients of 0.50 or higher with a power of 96% [47] and it, therefore, properly based the MID-anchor calculations. Another strength of our study is that we were able to include non-completer participants with a low attendance rate, which makes the findings more realistic.

On one hand, our interviewer-administered method of data collection can be considered as a limitation. The face-to-face interviews might have caused response bias, in terms of acquiescence bias [60], and also social desirability, which is stronger in women compared to men [61]. This might have influenced the gender effect difference in our study. Respondents may deliberately answer questions inaccurately, either by underreporting or overreporting of normative or stigmatized issues such as sexual behaviour or eating patterns [62, 63]. In our data, we did not find a gender difference in the response to the item of satisfaction with their sexual life in the QoL measure (MANSA), which is an issue that could cause shame and be influenced by social desirable bias. Therefore, we could not conclude that social-desirability bias did lead to the gender difference found in our study and nor could we definitely rule out the presence of this bias. This bias, just as with acquiescence bias, could have occurred in the baseline as well as in the endpoint interviews. Therefore, we expect that the change we saw can be considered a real change. The length of the questionnaires could have caused cognitive fatigue and biased the results, because we did not change the order of the different questionnaires. In a confirmative trial, randomizing the order might prevent this bias.

On the other hand, the face-to-face interviews might have prevented non-response bias, by preventing attrition. We estimated that too many of our participants would not respond to self-administered questionnaires. Only participants with a higher level of functioning might have completed the questionnaires, which could have caused bias. We decided we could better use the advantages of face-to-face interviews [28] as mentioned before in the ‘Materials and method’ section.

Taking into account the low sample size and the gender covariate, we conclude with certain prudence that the MHRM was capable of showing the most relevant and meaningful change in persons with severe mental illness as a result of the IMR programme.

Our research can be used as an example of how to estimate MIDs in the context of people with severe mental illness. More research with a larger sample needs to be done to gain a more solid grounding for the MIDs. This research needs to account for the gender covariate. In future research on the effectiveness of interventions for people with severe mental illness, a recovery measure such as the MHRM should be used.

In the search for scientific information that can convince clinicians to change their treatment practices and convince policy-makers to change their treatment guidelines, our findings can be used, with certain prudence, in shared decision-making processes. When an outcome on recovery is desired, a person with severe mental illness can be assigned to the IMR programme. A recovery measure such as the MHRM is able to measure the effect and should be used uniformly.