Search strategy and selection criteria.

We registered the systematic review protocol as PROSPERO no. CRD42015027346, and the project with the Core Outcome Measures in Effectiveness Trials initiative (COMET; http://www.comet-initiative.org/studies/details/819?result=true). After registration we changed the protocol and additionally searched for ongoing as well as published trials on clinical trials registries. We sought to identify published and ongoing disease modification trials, which we defined as interventions aiming to change the underlying pathology of the disease, as opposed to solely aiming to treat symptoms of dementia.

We conducted database searches using the key words for MEDLINE (via OvidSP; Appendix A in S1 File) and searched ALOIS, CENTRAL, CINAHL, EMBASE, LILACS, and PsycINFO for published trials from database inception to 11^th December 2015. We used the same strategy across databases, and adapted it for ALOIS. Additionally, we searched for ongoing trials on clinical trials registries (ISRCTN and clinicaltrials.gov) on 22^nd January 2016 and 29^th January 2016 respectively. We limited database searches to English language, where possible, as we were searching for instruments available in English. We also hand-searched the reference lists of relevant systematic reviews, and of a systematic review of non-pharmacological interventions for dementia led by members of the group (LL).

We included trials that met all of the inclusion criteria:

1. Full text is written in English
2. Published in a peer reviewed journal article or a registered ongoing trial
3. At least some of the participants have clinically diagnosed mild or moderate dementia
4. Any length of intervention that aimed to modify the dementia disease
5. It is a randomised controlled trial (RCT) or clinical controlled trial (CCT) with:
   1. the intervention directed at the person with dementia
   2. the control or comparator arm comprising treatment as usual, no intervention, sham therapy, other therapy or placebo
6. At least one quantitative outcome measure related to disease modification in mild or moderate dementia

We excluded studies where all participants had severe dementia or mild cognitive impairment (MCI), and studies set in care homes, as few people with mild to moderate dementia are resident in care homes [14]. We also excluded trials if the outcomes were only qualitative, economic or related only to carers or drug levels.

Two reviewers (DG and LW) piloted the screening of titles and abstracts by independently screening the first 20 records and comparing decisions, and had no disagreements. Three raters (AGS, DG and LW) also piloted the screening of full texts for inclusion by independently screening the first 20 papers. They agreed on 80% of decisions, with no decision made by any rater to exclude a paper that was eventually included. There was disagreement about four papers concerning whether the intervention was aiming to modify the underlying pathology of the disease of dementia. We agreed to solve this by examining in detail how the intervention was described in the background section, as well as the aim of the intervention within the trial, and to discuss any trials where there was uncertainty between the raters and principal investigator (PI), GL.

Data extraction.

We extracted characteristics from each trial, including location, dementia type and severity, how dementia diagnosis was made, participants’ sex and age, description of the intervention, number of participants (n), description of the comparator group, outcomes related to disease modification (primary or secondary if reported), and when outcomes were measured. We contacted authors for extra information where necessary. As our aim was to synthesise the trial outcome measures used, we did not assess the risk of bias of individual studies.

We divided outcomes into the domain measured (cognition, activities of daily living, biological markers (further divided into imaging, and fluid markers), neuropsychiatric symptoms, quality of life, and global). Initially we had not planned to consider global outcomes, but we added it as a separate outcome category as we found relevant measures. To assess accuracy of extracted data, two reviewers (DG and LW) independently extracted data from the same subsample of five trials; there were no differences between the raters’ extracted data. We used this exercise, and five additional papers, to pilot the data extraction tool and ensure all relevant data from trials were captured. As the raters were using the same criteria, all other trial data was then extracted by a single researcher, either DG or LW. We calculated how many trials used each outcome measure and the total number of participants across all trials. We searched Google Scholar to find a copy of each outcome measure in English, either the manual or a key paper relating to its development.

Outcome validation.

We conducted separate iterative searches on Google Scholar using the name of the measure and psychometric terms to find information on the measures’ psychometric properties relevant to people living with dementia. We also consulted within our expert group. We sought information about:

• if the measure is validated in people with mild to moderate dementia for the outcome in which it is used as a measure
• if there are any relevant populations in which the measure is validated e.g. mild to moderate dementia, ethnic groups, languages
• unit of measurement
• sensitivity to change
• minimal clinically important difference
• reliability (inter-rater and test-retest)
• acceptability
• ceiling and floor effects
• average time taken to complete
• who fills in the questionnaire i.e. researcher through patient, family carer, paid carer or observation or self-complete
• any risks identified of use of the measure

Outcomes.

In total, 81 outcome measures were used across trials; 72 questionnaire/interview based measures and nine biological techniques to measure biomarkers. Two authors provided extra information regarding outcomes in response to our enquiries [15, 16]. Table 1 lists the outcome measures and numbers of trials in which they were used. They included:

• Cognition (measured across 117 trials by 31 outcome measures)
• Quality of Life (QoL; measured across 16 trials by 3 outcome measures)
• Activities of daily living (ADL; measured across 68 trials by 12 outcome measures)
• Neuropsychiatric symptoms (measured across 58 trials by 16 outcome measures)
• Global assessment (measured across 80 trials by 10 outcome measures)
• Biological markers (measured across 71 trials by 9 biological techniques)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Overview of findings of the systematic review.

https://doi.org/10.1371/journal.pone.0179521.t001

Outcome validation.

The validation data for the outcomes in each domain is available in Tables D-I in S1 File. Due to the large number of cognitive outcomes we shortlisted five cognitive outcomes for which we would search for validation data (ADAS-Cog, Mini Mental State Examination [MMSE], CERAD Neuropsychological battery, the Neuropsychological battery and the CogState Alzheimer’s Battery), and excluded the other cognitive outcomes for a number of reasons. These were that they had only been used in one trial, they measure cognition in severe dementia or, they are not available in English, and they only refer to one domain. We also did not search for validation of cognitive tests measuring only one specific domain of cognition. As dementia is an impairment of more than one cognitive domain a global scale is more appropriate. For the neuropsychiatric outcomes we decided we would not recommend measures of only individual neuropsychiatric symptoms, such as agitation or depression, as they would add to the burden of a core outcome set therefore we did not search for validation data for these measures.

Focus groups and first email consultation.

Twelve people participated in face-to-face focus groups in London, Sheffield and Cambridge; including three people living with dementia, two current family carers, six former family carers, and one PPI group member. Some had participated in research before as a participant themselves, or supported a family member through participation; some had no research experience. Five of them (4 carers and 1 person with dementia) replied to the first email consultation after the groups about an overview of the findings from across all three focus groups.

The participants made general recommendations about what would help them to participate in research and complete outcomes, as well as recommendation specific to the outcome domains. At the start of each focus group session people were asked ‘what should be measured?’ Responses were varied but gave broad support to each of the six domains covered.

The recommendations around completing outcomes included the following.

Questionnaires content and delivery.

Participants thought questions should be really clear as the wording of the questions affects people’s different answers. In addition, they thought that too many questions or fast delivery can make people with dementia anxious, and can seem like someone is “getting at you” (carer)

There were differing reactions to standardised questionnaires which we showed to the groups including people living with dementia. Some thought questions should emphasise the positives, that is what people can do rather than the skills people had lost but others disagreed The difference in opinions were due to some thinking that focusing on what someone cannot do was a stigmatising way of considering people with dementia, but others thought that it was probably relevant information.

“It’s more relevant what they cannot do than they can do, because there’s an awful lot more we can’t do than we can do” (Person living with dementia)

Time and travel.

Participants at all three focus groups thought that both the time taken and travel required to complete outcome measures could be barriers to participation, and these aspect of the study should be well-thought-out, in the same way the scientific protocol would be. They stressed the importance of clearly setting out the time required, including waiting times and breaks, as carers felt that waiting time can be particularly difficult to explain to the person they support, and that breaks are even more necessary for people with dementia than for the general public.

“Where it’s got to the stage that you’re not aware that you’re not giving the right answer, or what you would have given at the beginning” (Person living with dementia)

Measures were deemed less acceptable if they required participants to travel to a specific centre each time, for example, biological techniques. Carers noted that being able to participate in research more locally would encourage and help with participation, regardless of whether measures included invasive biological techniques, due to the difficulties that travelling can bring to people who are physically unwell, as well as living with dementia.

“Going to have comorbidities… They’re going to have all the other things going on… so it makes it much more difficult” (carer).

Carers’ participation.

Participants agreed on the “importance of carer in the decision making” for participating in research, but also discussed the likely disparity in answers according to who answers the questionnaires, whether it was people living with dementia or their carers, though they thought carers answers can provide additional data. They also highlighted that not all people with dementia will have a defined carer, or someone who is available to act as an informant.

“You can’t expect them to take days of work to come with you to an assessment… it’s difficult because then you’re relying on what I’m saying” (Person living with dementia)

Engagement.

Many participants thought that clear communication during the study, including reminding them of what had already been discussed about the reasons for completing particular measures, would aid continued engagement in the trial. One participant also suggested that engagement may be enhanced if the particular measures used reflect the subtype of dementia, as they had experience of questions not reflecting the subtype of dementia and that can make the research seem “a waste of time”.

The recommendations specific to the outcome domains are the following.

Activities of daily living.

Participants across the groups differed on opinions about the use of ADL measures as a core domain, but generally judged that instrumental ADLs were more relevant to mild to moderate dementia than basic ADLs.

“A lot of what’s down there now doesn’t apply to me” (Person living with dementia on items of an ADL scale that refer to basic ADLs)

Participants suggested that questions should ask why or how someone was impaired in these activities, as this was not included in ADL measures. For example, participants with dementia pointed out that they may still be able to do activities but in a different way to before they had dementia, such as by wearing clothes that are easier to put on or cooking simpler meals.

“Some task you can still do, just differently, just adapted” (Person living with dementia)

Additionally, people living with dementia suggested that they may avoid ADLs, such as using the telephone, due to lack of confidence rather than an inability to complete them, and that measures do not seem to take this into account.

“I can use the telephone, but I don’t have the confidence, I don’t like using the telephone” (Person living with dementia)

Carers also raised the point that some ADL questionnaires they have had to complete were very similar to disability benefits assessments, and that this may affect the answers that participants give.

“The link across into disability benefits… it’s suspicious” (carer)

Biological markers.

Participants generally thought that biological markers should be a core domain, as they believed them to be the most reliable, objective measures of the disease process.

“Biological measures you’ve got something that everyone can agree on… you can compare like with like” (carer)

Carers had a number of general questions about the type and quality of data biological measures could provide including if one technique was more informative than another, what biological changes these measures detect, and how many people need to take part for any significant results.

“Do you get more information from a lumbar puncture than a scan?” (carer)

“Would you be looking for a reduction in amyloid or tau, or would you be looking for an arrest in the size of brain shrinkage?” (carer)

“How many people would you need for it to be significant?” (carer)

Overall, participants agreed that blood tests were acceptable and that it would not be difficult to commit to frequent tests, with the main barrier being travel.

“Weekly blood tests would be very happily tolerated” (carer).

Some volunteers particularly liked cerebral spinal fluid (CSF) measures even though they were aware of possible side-effects, but they thought misconceptions about what the procedure involves might discourage participation. Those who had experienced CSF tests, either directly or indirectly, did not mind them but found the need to have it done in a specific location, for example, a clinic, presented difficulties.

“A couple of hours and it’s done… it doesn’t put him off going the following year… he knows exactly what he’s let himself in for and he does it”

“I think the practicalities of it would be the bit that concerns me… practicalities should be made easier for the patient” (carer)

Most thought that imaging could be core measure as it could provide objective data, and that many people would consent to having scans, as giving biological data can make a person with dementia feel that they are contributing useful information.

“In going through that he is doing absolutely everything he can to further the research” (carer)

“It’s a positive action” (carer)

Practical issues around travel were raised, and volunteers agreed that scans may be difficult for some people with dementia, and thought lying still for 45–60 minutes would be impossible for some.

“Vascular and Alzheimer’s when it gets to the middle… you can’t keep people still… they don’t understand what’s happening” (carer)

“It bothers me about the scan… incredible noise being lashed down. I knew what was going on… but for people with dementia it could ruin them” (carer)

“different people react in different ways” (carer).

Having music or a screen to watch would improve the experience, giving the individual something to focus on; and that could be done for most participants “If there is some flexibility” within the MRI environment. Carers agreed that PET scanners were less restrictive than MRI, though the issue of having to keep still remained.

After the conference, participants were consulted about and agreed with the suggestion made there regarding the use of serial structural MRI in a select number of participants who had consented to the procedure, rather than including MRI as an essential component for every individual in the study design. They agreed that this could improve recruitment and retention, as the prospect of MRI scans can discourage participation.

Cognition.

Overall, participants agreed that cognition should be a core measurement. However, volunteers felt that it should not be used in isolation as the only measure and suggested that cognitive scores needed contextual, qualitative information. Carers in particular suggested that more weight should be placed on cognition in the context of previous ability rather than cognition alone and were unaware that this was standard clinical practice.

“It is more important to understand how a person’s cognitive impairments affect their activities of daily living and quality of life…than it is to rate their underlying cognitive skills” (carer)

Participants, including both people living with dementia and carers, felt that memory tests can be demoralising. Carers particularly felt that it is distressing “watching someone fail a test”. People with dementia described the distress of seeing their score and performance worsen, and a tendency to try to prepare for tests to prevent this from happening.

Although they acknowledged its use as a standard cognitive measure, performed in a range of research and clinical contexts, several people did not like the MMSE. Some people with dementia thought it seemed irrelevant, as it gives a restricted account of dementia and carers described how they thought it would be difficult to have a cognitive measure that seemed relevant to all people with dementia. Some people preferred the ADAS-Cog to the MMSE as it is more detailed.

“Cognitive testing, because we are so different and the way it progresses in the different forms of dementia makes it so complicated to try and produce some kind of standardised measure” (carer)

Neuropsychiatric symptoms.

Some participants said that behaviour is a core domain as changes in behaviour are often present in dementia, and may be more relevant than ADL, whilst others thought it was less applicable in the milder stages of dementia. In addition, they commented that questionnaires do not measure any reasons behind changes in behaviours or consider an individual’s personality. In particular, some felt that behaviour should be measured alongside cognition and understanding, which could affect whether someone becomes agitated or aggressive. Volunteers suggested that if behaviour were measured, it should, as it usually does, include sleep, agitation, wandering, repetitive behaviours, and changes in appetite.

“It’s teasing out the brain damage from actual personality traits” (carer)

“It’s about behavioural change over time” (carer)

Quality of life.

Volunteers had different opinions over the inclusion of QoL measures as a core domain. One volunteer thought it was core, as it could give a summary of an individual’s experience of dementia. Others were unsure about the sensitivity of QoL measures. In particular, some felt that QoL measures lack detail regarding the individual’s personality. For example one carer mentioned that someone may have always have disliked social events, and that a questionnaire may not pick this up. It was suggested that comparing carer and patient responses would give the most accurate account of QoL, as carers and patients may have different responses, and they thought a carer’s interpretation alone was not necessarily accurate.

“I don’t know how much you’d get out of it”

Global.

Volunteers had differing opinions about global rating scales, with some liking the breadth of the measures, while others suggested that global measures were superficial and depend too much on the day in which it is measured, and therefore were not meaningful. Those who did not like global scores suggested that a larger package of specific measures would give a holistic view of an individual and include more detail.

“I like the global one… it’s all encompassing” (carer)

“You might be feeling particularly bad that day” (carer)

“Is that really valuable… Am I giving my time for something that’s meaningful? I’m not really convinced” (carer)

Cognition.

It was agreed that cognition is a core domain for disease modification trials, and that a measure should be reliable and sensitive in showing decline, which both the ADAS-Cog and MMSE have been shown to do in the earlier cholinesterase inhibitor trials [17], as well as being the most commonly used and validated outcomes. Due to this, both of the measures were decided to be the only real contenders from the cognitive measures used across the studies found in the review. The PPI consultation emphasised that patients dislike failing in tests of cognition and at the conference it was discussed that to improve the experience of cognitive tests in the future timed measures which estimate cognitive processing speed should be considered, as they do not mean the patient feels they have passed or failed individual items, though these would need validating.

There was debate around the ADAS-Cog or MMSE, as the ADAS-Cog is longer and therefore more frequently has missing data, but may be more sensitive to change so it might need fewer participants than a trial powered on the MMSE. Due to the ADAS-Cog having a larger number of items, this may give the impression of potential greater sensitivity to change than the MMSE, but superiority was not seen in the trials of symptomatic treatments [17]. The MMSE is affected by education, and also lacks sensitivity in differentiating between very early AD or MCI and normal ageing [18, 19]. However, in mild to moderate AD it has reasonable psychometric properties and was sensitive to change in earlier cholinesterase and memantine trials [20]. Further, because the ADAS-Cog is not used in clinical practice, clinicians do not have a “feel” for what difference in score would constitute a meaningful change. The ADAS-Cog has also been criticised for not sufficiently assessing attention, planning, working memory and executive functioning, all of which are impaired at the earliest stages of AD, which led to the addition of extra tests to the original version [21]. These shortcomings also apply to the MMSE [22]. Overall for cognition we recommend the MMSE or ADAS-Cog, as they are the best available measures of those found in our review and considered by our consensus panel. Both of the cognitive outcomes chosen have minimum clinical important difference (MCID) available. The MCID in the MMSE is 1.4 points [23]. MCID, for the ADAS-Cog has been calculated as 3 or 4 for early Alzheimer’s disease [24, 25]

Fluid makers.

The use of CSF biomarkers was discussed. Our systematic review confirms that the three established markers Aβ42, tTau and pTau181 are also the most widely used markers in AD dementia clinical trials. Even though AD fluid markers are useful in order to enrich studies with true AD cases, and to measure target engagement, their usefulness as outcome measures is limited based on the available evidence. A meta-analysis showed changes in putative biomarkers of Aβ42 are not related to change in MMSE so they do not appear to be helpful biomarkers for disease modification [26]. In addition, this technique can be expensive due to potential medical care needed after [27]. It was therefore agreed not to recommend a fluid biomarker and that currently the measurement of CSF biomarkers is not a core outcome in disease modification trials.

Imaging.

Imaging is important for disease modification, though changes in imaging and cognition are not always highly correlated; between group differences may show mechanism of change as well as disease modification [28, 29]. There were only three potential outcome measures for imaging with sufficient evidence of sensitivity to consider using them as outcomes. These were serial structural MRI, serial amyloid Positron Emission Tomography (PET) and serial Fludeoxyglucose PET. Volumetric MRI does not show expected changes in terms of atrophy immediately if amyloid is removed, as the volume may go down in treated patients. It however shows brain changes over time. Thus overall there is usually a decrease of 2% over a year in AD and 0.5% in DLB [30]. Structural MRI was the best imaging technique to find sensitivity to change in untreated patients, is validated against underlying pathology, and is available in UK sites for use as measure [29, 31]. In addition, it is cheaper with the NHS tariff being £110 compared to an approximate cost of FDG PET is around £1000 and amyloid PET being £1600.

Earlier trials have found that they may need fewer participants than for cognitive changes to be fully powered, [32–34] meaning MRIs can therefore be conducted in a subgroup of participants in disease modification trials. This means that participants who were unable, or did not wish, to have an MRI would still be able to take part in trials.

Of the biological techniques found in our review, imaging was judged to be a core domain by the consensus panel. The evidence was that serial structural MRI was the best technique currently and we are recommending using an optional serial structural MRI in a voluntary subgroup of participants. As there is rapid innovation of biomarkers, this recommendation may change in the future.

Neuropsychiatric symptoms.

There was debate around whether neuropsychiatric symptoms should be a core domain, as not everyone with dementia, especially in the milder stages will have these symptoms, and if they do they may not be clinically significant, meaning that even if a treatment is disease modifying there may be no difference in these symptoms [35, 36]. Some members of the group thought neuropsychiatric symptoms were core, as a disease modifying treatment should improve behaviour, or at least not make it worse but overall the conference judged this to be important but not a core outcome for every future study of disease modification as many people with mild dementia do not have any neuropsychiatric symptoms. If neuropsychiatric symptoms are measured we recommend that studies use the Neuropsychiatric Inventory (NPI), as it is the best of the available neuropsychiatric measures that our panel considered as the only measure which includes both frequency and severity of symptoms and has appropriate psychometric properties (Tables D-I in S1 File).

Activities of daily living.

It was discussed that activities of daily living should be measured ideally through a proxy as impairment may be underestimated by people living with dementia, though it is not always [37]. Some scales try to differentiate between the performance and initiation of ADLs, and thus ask whether people with dementia need prompting to complete ADLs. This domain is being used less in new disease modification trials as it is affected by physical health and thus any changes may be unrelated to dementia. In addition, the measures may not have sensitivity to change in mild dementia as there may be very little impairment. Finally, the measures do not take into account that people come from a range of backgrounds and some may never have engaged in tasks such as cooking [38]. Overall it was therefore decided that ADL it is not a core measure of activity limitation. If one is used in disease modification trials, we recommend the DAD as it is the best available dementia specific measure of ADL from the 12 measures the consensus panel had to consider (Tables D-I in S1 File).

Quality of life.

Quality of life was discussed and it was considered to be different to other domains as people with severe dementia can have good life quality [39–41], and therefore it is not a sensitive outcome to measure disease modification. However, it is useful to understand negative or positive impacts of pharmacological interventions. As the self-reporting of QoL becomes more difficult over time, it is ideal to get a proxy report too so that if the participant deteriorates and is less able to give information reports can be compared from earlier in the trial. It was also discussed that quality of life is important in phase three trials where economics need to be measured for cost by a dementia QALY (Quality-adjusted life years). The conference did not include economists so did not have the required expertise to discuss or make recommendations about QALYs. If quality of life is measured, the conference recommended the DEMQOL, which is able to measure quality of life from both the person with dementia and proxy perspectives, and therefore considered the best measure of the three found in our review. We also note that QALY values can be calculated from it.

Global functioning.

The group considered that global functioning was an important, but not core, measure as it encompasses several domains. If it was used they recommended either a staging or impression of change outcome measure, rather than a multi domain scale. A staging instrument was generally thought to be best by the consensus panel as it is less subjective than an impression of change. It was decided not to recommend a global instrument as core, but if it was used that the Clinical Dementia Rating global score CDR would be the instrument of choice from the ten measures considered by the panel. It has good psychometric properties and that the global score may be more appropriate than the sum of boxes so that particular domains were not prioritised or contributed disproportionately to the overall score.

Main findings.

The attendees decided that both cognition (as the primary symptom in dementia) and biological markers (measuring disease modification) were the core outcomes that should be measured in all disease modification trials in mild-to-moderate dementia. They recommended that cognition should be measured using the Mini Mental State Examination (MMSE) [42] or Alzheimer’s Disease Assessment Scale–Cognitive Subscale (ADAS-Cog) [43], and as a biological marker serial structural MRI should be used as a measure for the subset of participants who consent to have it.

The other domains covered potentially important additional effects in disease modification treatments but were not assessed as core, in that dementia could be modified without necessarily improving them. These were: ADL, neuropsychiatric symptoms, QoL, and global ratings. All were deemed important although not essential and it was concluded that they should be used dependent on study goals and to consider effects of interventions other than disease modification. The conference recommended the Disability Assessment for Dementia (DAD) [44] for ADL, the Neuropsychiatric Inventory (NPI) [45] for neuropsychiatric symptoms, DEMQOL [46] for quality of life, and the CDR [47] for global ratings.

We also discussed the issue of informant rated scales, and how it was necessary that the informant had seen the person frequently in the weeks prior to completing the scale, so the availability of an informant or supporter or each participant is part of the study design.

Second email consultation.

After the core recommendations from the conference, we conducted a second electronic consultation with the AS research network. We received 6 responses from people with experience of dementia; one person with dementia, and five carers (three current carers and two former carers). Those who answered this additional consultation agreed with the main recommendations from the conference.