Plain English summary
AA testing and cognitive debriefing
(n = 9)
(n = 17)
Age: mean (SD), range
39.5 (9.8), 30–55 (n = 1 not answered)
45.6 (14.1), 21–63
Gender: n (%)
School education degree: n (%)
General education (9 years)
Middle school (10 years)
Higher education (12 or 13 years)
Other (not specified)
Job situation: n (%)
Student / trainee
Type of multiple sclerosis: n (%)
Years since first diagnosis of MS: mean (SD), range
10.1 (6.8), 2–22
14.6 (8.9), 2–30
EQ VAS on subjective health statusa: mean (SD), range
74.9 (16.8), 55–97
73.0 (18.8), 30–98
EQ-5D-5L index scoreb: mean (SD), range
0.79 (0.19), 0.38–0.94
0.76 (0.20), 0.35–1.00
PDQ global score on cognitive impairmentc: mean (SD), range
18.1 (11.3), 0–34
25.1 (13.6), 0–46 (n = 1 not answered)
AA testing and cognitive debriefing
Rationale and/or reason for changing the preceding version (if applicable)
EQ-5D-AA version in which this specification was introduced
Sample quotation from study participants
Specification in preceding version (if applicable)
Sample quotation for the preceding version (if applicable)
Assessment is retrospective to the previous assessment of the item (e.g. the morning alert refers to the time period since the last evening).
This way, the complete day and night will be covered. As some items are only pertinent at specific times of the day (e.g. self-care), momentary assessment is not reasonable.
FG 2: “I would spontaneously prefer to have a longer period of time. Maybe two, three times a day or so, because then, you would avoid that like short-term incidents, well, which can dominate the moment that would maybe lead to a different result from answering.”
The exact assessment times are predefined individually for each participant at study onset.
Participants will not have to start the app actively, as this may be stressful and will probably be forgotten in many cases (based also on experience from a previous AA study). Individually defined times will take different bedtimes into account.
FG 2: “I guess it’s not my first thought directly after getting up to click into app on and say ‘I am awake’.” [Interviewer:] “Yes, actually that was our hope.” (Both laughing) [Participant:] “Yes, but I don’t think … “ [Interviewer: “It’s not realistic.”] [Other participant:] “Yes, I think so, too.”
The morning assessment is defined as the earliest time the participant is usually awake.
The app will not wake the participant (if the phone is not on silent) and the alert will not be missed due to being asleep (if the phone is on silent).
FG 2: “But if having a specific time of the day is not relevant for you, but each person individually, then you can actually enter sleep and activity times in the app. Because that would of course guarantee that in the morning at that time, that I actually complete the questionnaire and will not forget it.”
The evening assessment is defined as the latest time the participant is usually still awake. Participants will have the opportunity to prepone completion (i.e. initiate an earlier completion of the evening assessment).
This way, the evening alert will capture as much time as possible of afternoon and evening. However, when going to bed early, participants will not be woken up by the app or miss the evening alert.
FG 2: “I don’t know, maybe one could define that beforehand: What does midday mean for me? What does evening mean for me?” [Other participant:] “Yes, yes.”
The midday assessment is defined as the exact middle between the individual morning and evening alert times.
Two wave 1 participants suggested to have the midday alert closer to the exact time point between morning and evening alert. This way, morning-midday interval and midday-evening interval will be more similar and the alert will be closer to midday for many participants.
Wave 1: “You might do this exactly in the middle of the time between morning and evening. Because now, I had that at 5:30 pm, which I found a strange time (laughs) (…) for such a query.”
The midday alert was sent eight hours after the individual morning alert.
FG 2: “Then I would say, after a certain amount of hours. Then you could actually adjust your daily routines.”
The morning assessment time can be defined differently for weekdays vs. the weekend.
Many people sleep longer at the weekend than on weekdays. This specification was not based on FG results but on the study group’s considerations.
The mobility item is assessed three times a day.
It was discussed whether mobility assessment makes sense in the morning where it refers to the night. As people may have to go to the restroom during the night, the consensus was that the morning assessment should include mobility. Upon inquiry, participants of waves 1–4 did not raise objections against this.
Wave 3: “If you had asked me in the past, I would have said 'Oh my God, I really have big problems in the night'. (…) Because I used a rollator at that time. (…) So, I got out of bed, used the rollator to get to the toilet and sat down. And err, this was a dangerous time. All very error-prone.”
Within the morning alert, a page saying “Good morning!” is displayed before the items are asked.
This was suggested by a FG participant and was mentioned as being important by a wave 4 participant.
FG 2: “Couldn't you program it so that it says a friendly good morning?"
The pain/discomfort item is assessed three times a day.
In the FGs, all participants reported pain to fluctuate, either from day to day or within a day and sometimes quite quickly. Some participants reported pain to also occur in the night.
FG 1: “Well, before going to sleep, you usually focus on it (…) because suddenly you are alone with yourself. (…) And then you notice many things you did not notice during the day. And there are nights in which it massively prevents you from sleeping, from sleeping peacefully, from sleeping soundly. Sometimes, it takes it out of you so that you feel absolutely whacked the next morning.”
The anxiety/depression item is assessed three times a day.
Although some FG participants reported considerable fluctuation in depression/anxiety also within the day, some of them expressed concern that an assessment more than once daily would be psychologically too burdensome. However, after changing the assessment from one to three times a day after wave 1, none of the participants of waves 2–4 found this too burdensome.
Wave 2: "Especially since emotions can fluctuate extremely fast, I think that three times isn't too much. (…) What strains me in the morning can be completely unimportant to me in the evening. (…).” [Interviewer:] “And it wasn't too burdensome to you, that you would say 'I don't want to think about it three times a day'? As that was some people's concern.” [Participant:] “I try to approach my life as reflected as possible. That means (…) actually I thought it was quite pleasant, so to speak, 'Have I been annoyed by anything?' (…) 'Can I do something about it?' So, err, becoming aware of it.”
Depression was assessed only in the evenings, retrospective for the time since the evening assessment the day before
FG 1: “When you suddenly get a sensation of fear and, err or a feeling of being depressed, err, but in this moment, (…) I do not want to pose this question that often during the day because then you focus on it.” (…) [Second participant:] “Yes!” [Third participant:] “Hmh. (approving)”. [First participant:] “And I think it would bring me to the point at which I do not want to do this anymore. (…) Because I am permanently reminded of it.” [Second participant:] “Yes. (…) I would say it is also something that might theoretically to some extent affect the daily activities. And if you ask this three times a day.”
[First participant:] “Yes.”
The usual activities item is assessed two times a day (midday, evening).
Assessing midday and evening will cover the whole day. Activities will not be assessed in the morning, because 'usual activities' mostly does not apply to the night.
FG 2: “They list a lot of daily activities here, for example. That means that you would somehow start in the morning. Then you go to work, for example, or studying and then you come home, do the chores and then you may have a leisure activity before or afterwards, so, the app could ask at many different times and ask if you have problems with it.”
The self-care item is assessed once daily in the evening, retrospective for the time since the morning.
When self-care was assessed twice a day in the first AA version, participants handled the item differently: If they did not wash or dress in the afternoon, some would respond "no problems" in the evening alert while others reported the problems they would, hypothetically, have had if they had washed or dressed themselves. Asking only once a day, referring to the whole day, shall ensure that for each participant their personal washing and dressing time will be covered so they need not respond hypothetically anymore.
Wave 2: “and then there was this ‘dressing and washing’ that showed twice somehow (…) that somehow didn't make sense to me. (…) If you have regular working hours, maybe it would fit better, I don't know.” [Interviewer:] “And do you remember which answer you chose in the evenings? (…)” [Participant:] “Mhm … more about my general condition with … moderate. (…) or I thought ‘How would I have felt had I done that?'”
Self-care was assessed at midday (retrospective for the time since morning) and in the evening (retrospective for the time since midday).
FG 1: “I would consider (…) hygiene rituals you have in the morning and in the evening.”
The EQ VAS is assessed once daily in the evening.
Some FG participants preferred a once daily retrospective assessment of the EQ VAS (as this was sufficient and not as annoying as multiple assessments), while others could also imagine multiple assessments. We then decided on a once daily assessment. No participant of wave 1–4 suggested a more frequent assessment.
FG 2: “In this case I would wish, of course, that we query it once a day. (…) Not three times.” [Interviewer:] “That's clear to you, ok.” [Participant:] “I would say that. Like reviewing the day, that you can manage that as, like, a resume."
Participants can prepone the evening alert if they are going to bed early.
This option was introduced not based on FG results but based on experience with previous AA studies. The feature was used by many participants of waves 1–4.
After using the option to prepone the evening alert or after responding to the regular evening alert, the option will not be displayed anymore until the next evening.
It caused confusion that the button for preponing the evening alert still showed after the participant had already answered the evening questions.
Wave 1: “The only thing that confused me a bit was (…) I had answered the evening questions and then there was the option again, ‘if you wanna go to bed now’ and that of course I found a bit strange because I had already answered this question. That you maybe set it like that this question won't come anymore then.”
The button for preponing the evening alert still showed after the participant had already answered the evening questions.
Participants can prepone the midday alert if they are going to take a nap.
Participants will not be woken up from midday sleep if they do not think of putting the phone on silent (or are too conscientious to omit an alert and therefore did not want to silence the phone).
Wave 3: “Well, when I am working I leave the house at 7 and I come back at half past 2, 2 and I directly lie down and then I thought, 'Oh, now I have to wait till three, so I need not take a nap now.' I found that stupid somehow. So, I took the phone to bed with me and then it woke me up.”
No option to prepone the midday alert was available.
The AA is collected over nine full days, of which the first two days serve as acclimation period to become familiar with the items. Only data from day 3–9 are used for analysis.
Data from the first two days may not be fully comparable with those from the following days because respondents adjust their responses after becoming familiar with the items, as some participants reported. In the AA literature, it is reported that variability typically declines with AA duration, which indicates familiarisation .
Wave 1: “I cannot really classify myself and then the first two times it is 60 and then at some point I thought, that's actually a bit too much (…) That's rather a problem FOR ME to see, how do I really feel and maybe the 60 in the beginning is, err … some kind of fallacy.”
The AA was collected over seven full days with all data being used for analysis.
Feasibility of the EQ-5D-AA
Female, 57 years: “For me, that was okay. I did not feel bothered in any way. (…) It could easily be integrated into the changing everyday life that I have. (…) It does not take long, (…) one minute maximum.”
Male, 51 years: “I was doing fine with it. The questions are clearly worded so that you know what is asked for.”
Female, 62 years: “I got along well. (…) I only feared it could wake up my neighbour. (…) There also have been no difficulties with the mobile (which I had feared in the beginning), because the questions were always the same.”
Female, 28 years: “It was actually quite pleasant. Though sometimes I was interrupted in my daily habits, when suddenly the mobile rings and you’re like: No! Silence, silence, silence!”
Female, 57 years: “Those two times or so that I forgot … not forgot, but too late … I think that wasn’t so dramatic.”
Variation in EQ-5D-AA items over time
Agreement between standard EQ-5D-5L and EQ-5D-AA
Standard paper EQ-5D-5L: meana
Individual difference between EQ-5D-5L and EQ-5D-AAb
EQ-5D-5L index score