Introduction
Most clinicians recognize that patients adapt and show remarkable resilience to health-state changes [
1], but work documenting such response-shift effects has largely focused on observational studies rather than clinical trials [
2‐
4]. Researchers have long posited that response-shift effects would alter measured treatment differences in a clinical trial, due to differential effects of treatment versus placebo on quality-of-life (QOL) changes over time [
5,
6]. In our companion paper [
7], we investigated response-shift effects in a clinical trial comparing Eculizumab versus Placebo for people with neuromyelitis optica spectrum disorder (NMOSD). The pivotal trial documented remarkable effects of Eculizumab in preventing relapse [
8], but subsequent analyses showed no such benefit on the SF-36™ mental component score (MCS) despite benefit on the SF-36™ physical component score (PCS) [
9]. This lack of benefit on this evaluative outcome led us to hypothesize that response-shift effects were obfuscating treatment arm differences in mental health.
Consistent with theory, response shift was conceptualized as an epiphenomenon, and therefore it is inferred by the behavior of other measured variables [
6,
10]. In our companion paper [
7], we sought to adapt the Oort Structural Equation Modeling response-shift detection approach [
11,
12] to the context of a small sample. Accordingly, we used random intercept modeling (RIM) [
13] as we investigated and detected response-shift effects related to Treatment Arm and, more specifically, to the experience of relapse. The companion paper’s results suggested that the benefit of Eculizumab was underestimated in standard analyses [
7]. These RIMs used VAS rather than MCS or PCS as an outcome. In order to explicate how response-shift effects may have clouded differences in MCS or PCS over the course of the trial, we sought to derive a method for communicating the VAS-based response-shift results in terms of MCS and PCS. This translation would move us closer to an estimate of response-shift adjusted change. We and others have long noted that response shift constitutes information, not ‘noise’ that should be removed [
5,
6,
10,
14,
15]. In order, however, to clarify the response-shift effects in the trial data, one must contrast it with something [
2]. This is why we are ‘back-translating’ the VAS scores into MCS/PCS scores with and without response-shift effects.
Discussion
We present a novel method for deriving response-shift-adjusted scores from the RIM response-shift detection method described in our companion paper [
7]. By using equating to generate crosswalks between scores that include and exclude response-shift effects, we are able to clarify how such effects could have altered the apparent Treatment Group differences on MCS in the clinical trial analyses [
8]. As noted in our companion paper [
26], published trial results likely underestimated Eculizumab vs. Placebo differences in mental health due to recalibration and reconceptualization. Thus, the difference in mental health between the Eculizumab and Placebo patients was likely wider than it appeared.
Our analyses document that Eculizumab patients’ MCS and PCS scores that include response-shift effects have a more truncated range, which generally makes them look better off than scores that remove response-shift effects. In contrast, Placebo patients’ crosswalks for both MCS and PCS exhibit similar ranges and similar linked scores whether including or excluding response-shift effects. Further investigation revealed, however, that the Placebo patients had the larger MCS response-shift effects at end of study, at very low and very high ends of the raw score distribution, whereas the Eculizumab patients’ response-shift effects were larger at baseline than at end of study. This would suggest that the Placebo patients, who experienced the vast majority of the relapses, engaged in mentalhealth response shifts after the relapse (i.e., at end of study), thereby enabling them to maintain homeostasis in mental health. These discrepancies could thus work to make analyses of group MCS differences at end of study appear to yield null results. In contrast, the PCS scores generally do not exhibit large differences between scores including and excluding response-shift effects.
As noted in our companion paper, our findings likely reflect the ‘shadow’ of response shift, inferred by the behavior of examined interactions and unique variance explained rather than characterized more directly. Our analyses suggest that response-shift effects were most prominent for MCS for both groups. Based on Fig.
6b as well as Table
2, there was some indication that such effects occurred to a lesser degree for PCS, and mainly for the Placebo group. In other words, people on placebo, who in this study had a much higher rate of relapse, were thinking differently about health due to their experiences: they emphasized the physical more and the mental less than did the Eculizumab group. Based on both groups’ strikingly low shared variance between MCS scores including and excluding response-shift effects (
R2 = 0.14 for both groups), the construct of mental health reflected in the two sets of scores must be substantially different. In contrast, the construct of physical health tapped by the two sets of PCS scores is relatively similar for Eculizumab patients but different for Placebo patients (
R2 = 0.38 vs. 0.21, respectively). It should also be noted that the number of visits and follow-up time are related strongly with relapse status. Indeed, the relapse analyses presented in the companion paper [
26] suggest that the response-shift effects found in the treatment group comparisons are even stronger when patients are grouped by relapse status.
While the present work has the advantage of providing more insight into the impact of response shift on evaluative mental- and physical- health indicators, its limitations must be acknowledged. First and foremost, our crosswalk approach utilized VAS scores from the RIMs that captured recalibration and reprioritization but not reconceptualization response-shift effects. These latter were captured in a separate series of RIMs examining unique group variance explained by SF-36™ domain scores and VAS, and were not feasible to include in the crosswalk method. Response-shift effects may thus be underestimated by leaving out reconceptualization in the translated scores. Further, the available data were drawn from the formal clinical trial, not the extension- study data. Since in the formal trial patients only contributed one additional data point after relapse (i.e., at end of study), the response-shift effects may be attenuated as compared to longer-term follow-up after relapse. Another limitation involves adding the residuals from the full model into model 2 to yield the estimated scores after removing response-shift effects. Residuals are random errors produced by a model. Adding the residuals as is, we treated them as fixed quantities, which is incongruent with the notion of random errors. This was a crude but pragmatic way to account for idiosyncratic variabilities after the response-shift effects were accounted for. Future research may devise a statistically more principled method to estimate these individual variabilities. For example, a term representing random error could be estimated independently using Monte Carlo simulation, bootstrapped, and added into model 3 to determine the range of results (i.e., confidence interval for effects as reported in Figs.
5 and
6). More straightforwardly, our analyses rest on certain assumptions. It assumes that specific values of residuals and person-specific random intercepts remain invariant when the residuals from one model are added into another. However, this may be viewed as a crude but pragmatic way to parse out response-shift effects from the measurements. There is also the possibility of misspecification of our RIM, which could affect our findings. Finally, a contrarian may raise the point that the evidence of response shift in MCS for those in the extreme groups (and not for the middle three categories) could represent a regression toward the mean. In fact, the pattern was not symmetrical, which would lend more support to a response-shift rather than statistical-artifact interpretation.
In summary, the method introduced herein provides a way to glean further information about response-shift effects in clinical trial data. Our results reveal Treatment Group differences in MCS response shifts, which have important implications for null results detected in previous work [
8]. It is our hope that the new applications of methods presented in both this paper and its companion will open new pathways for clinical research on new drug treatments and patient resilience.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.