Limitations of previous methods for COS development

Less than a fifth of previous COS studies have involved public representatives, and the majority of those included only a handful of patients [11]. Despite evidence that patients may hold different views from health professionals, and a recommendation to include patients in the process [9], it is unclear whether different stakeholder group views were transparent to those involved [11]. This current study implemented a novel design for COS development, including a new method proposed by children and young people to elicit opinion from children. We investigated the influence of the method of stakeholder feedback on subsequent opinion. We report the results from this work here.

Systematic Review and generation of the list of outcomes

A list of outcomes previously reported in studies of the treatment of OME was generated by updating a 2009 systematic review [9] using the same search strategy. The review of papers was completed by two authors independently (NLH and IAB) and a list of all outcomes measured in identified papers generated (detailed information is given in the protocol) [13]. This list of outcomes was further refined to standardize the name given to each of the outcomes. All outcomes and domains were discussed with members of the Study Advisory Group (SAG) prior to being finalised. Due to the varied health professional groups likely to be completing the Delphi an ‘outcome tip’, further describing the outcome, was also written for each outcome and reviewed by the SAG. The SAG comprised of Speech and Language Therapists (n = 2), Cleft Surgeons (n = 1), ENT Surgeons (n = 2), Audiologists (n = 2) and Clinical Psychologists (n = 1). A patient representative was approached during the study and accepted membership of the SAG but then, due to unforeseen personal circumstances, needed to withdraw from membership. prior to attending an SAG meeting. The SAG and also the Study Steering Committee (SSC, comprising a trial methodologist, patient representative, health economist and a paediatric otorhinolaryngologist) were also given the opportunity to add outcomes to the list that they considered important.

Health Professional Delphi Survey

The opinion of health professionals was sought through a three round Delphi survey delivered using a bespoke online system[13]. Health professionals were eligible to participate in the Delphi survey if they were affiliated to a UK Cleft Centre and were a cleft surgeon, ENT surgeon, audiologist, cleft nurse specialist, clinical psychologist or speech and language therapist. Potentially eligible health professionals were identified through contact with clinical leads at each of the 15 UK centres who provided a list of current members of their cleft team and their clinical role (S3 Table).

Prior to completion of round 1 it was agreed that for the feedback of results in round 2 to be presented by stakeholder groups, approximately 10 participants per stakeholder group would be required for the presentation of results to be meaningful. In the second round results were presented by health professional stakeholder group, with an individual seeing only the aggregated results from their particular group together with a reminder of their own round 1 score. In the third and final round the results of all stakeholder groups, including parents and children, presented separately, were shown to each participant together with a reminder of their round 2 score.

In each round of the Delphi survey, health professionals were asked to score a list of outcomes using the Grading of Recommendations, Assessment, Development and Evaluations scale of 1 to 9, with 1 to 3 labelled ‘not important’, 4 to 6 labelled ‘important but not critical’ and 7 to 9 labelled ‘critical’ [14] (S1 Fig). Consensus on outcomes for inclusion in the COS was determined using a pre-defined definition of consensus (S1 Table).

Opinions of patients and parents

The views of parents of children with a CP aged 0–11 years, and children with a CP aged 6–11 years, were explored in semi-structured interviews conducted by one of the authors (ST), who does not have a clinical background and was not known to parents or children prior to the interview. A purposive sample was recruited to provide maximum variation in terms of a child’s age and gender and type of treatment experienced for OME. Participants were recruited from two cleft centres in the UK with contrasting approaches to audiology care, one a centralized service, the other distributed across a hub and spoke” model. Interviews were audio-recorded, transcribed verbatim, and then Framework analysis [14] was used to manage and interpret data. Discussion around outcomes took place throughout interviews. However, there was a specific section of the topic guide used within interviews that focused on capturing data on this issue. Results of interviews relating to experiences of OME are reported elsewhere [15,16].

Semi-structured interviews with parents and children gave in-depth information on outcomes of importance for these groups. Interviews did not include a discussion of the outcomes list generated through systematically reviewing the literature because we wanted participants to express their opinions, in their own words, of what they felt were important results or indicators of successful management of OME. In order to give parents and children the same opportunity to score outcomes, an online survey, similar to that completed by health professionals, was developed. This involved review and re-wording, using a plain language description, of each outcome scored by health care professionals. Each re-worded outcome was tested for readability using the NIACE SMOG calculator [17]. Understanding was explored with the Cleft Lip and Palate Association (CLAPA) children and young person’s council (CYPC) and a local CLAPA ‘Happy Faces’ group. The same outcome wording was used for all participants with the exception of minor changes such as “your/your child’s” to ensure appropriate context.

Parents and children aged 7–16 years were asked to consider the appropriate question, described in Fig 2, and to score each of the outcomes. The labels of the 1–9 scale were modified for parents whilst children, under the recommendation of the CYPC, scored each outcome using a traffic light system where the scores 1–3 were represented by a red box labelled as “not that important”, scores 4–6 as an amber box labelled as “important” and scores 7–9 as a green box labelled “really important” (S1 Fig). Both parents and children were provided with a free text box to add anything else that they considered relevant. The definition of consensus (S1 Table) was applied to the results from both parents and children.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Initial question asked prior to scoring outcomes for parents, adults and children with cleft palate.

The question asked of health professionals is included for comparison.

https://doi.org/10.1371/journal.pone.0129514.g002

Participants of the survey for parents and children were independent of those who completed a semi-structured interview and were identified using the CLAPA mailing list and social media pages with a potential reach of 4,710 and 9,564 respectively. An individual email was sent to all those on the CLAPA mailing list together with a reminder in their e-newsletter. A link was posted on the Facebook page which included the researcher’s name and photograph (NLH) and a link to the online survey. There is likely to be substantial overlap with membership of multiple Facebook pages and groups however, it was not possible to assess this.

Final consensus meeting

The consensus meeting brought together all sources of information. The results from the Delphi survey of health professionals and the survey of patients and parents were presented along with the opinions from 43 parents and 37 children who took part in a semi structured interviews. An invitation to attend was sent to: health professionals who had completed all rounds of the online Delphi survey and expressed an interest in attending future meetings; all parents who had completed an online survey and expressed an interest and provided contact details to be informed about future meetings; parents who had taken part in a semi-structured interview whose contact details were still valid; CLAPA members in the North West who subscribed to the CLAPA mailing list. The format of the consensus meeting comprised a short study overview, a summary of results from the semi-structured interviews and a review of each outcome on the scored list in turn, including presentation of how each stakeholder group had scored the outcome, and the number of stakeholder groups who achieved consensus. Discussion of each outcome was followed by anonymous electronic scoring by those at the consensus meeting. A written report of the final meeting was circulated to participants for comment.

Ethics statement

Ethical approval was received from the National Research Ethics Service North West – Greater Manchester East Research Ethics Committee (Reference 11/NW/0586) for the completion of semi-structured interviews and invitation of interviewees to the final consensus meeting. Written consent was sought for participation in semi-structured interviews, with written proxy consent sought from parents/guardians for their child’s participation. Written assent was also sought from children aged 6 years and older. Attendance at the consensus meeting was considered to be implied consent for participation with no written consent provided, this process was approved by the Research Ethics Committee. Advice was sought from the National Research Ethics Service who did not consider that ethical approval was required for an online survey of parental and child opinion. However, full information about initial the study was given in the initial pages of the online survey and survey completion considered to imply consent.

Systematic review

The search retrieved 85 potentially eligible studies with an additional 42 identified from other sources, after screening titles and abstracts, all but nine studies were deemed to be irrelevant (S2 Fig). After further analysis of the full texts one further study was excluded as it was undertaken to determine the frequency that children with CP pass their new born hearing test [18]. Two non-English papers were identified [19,20], for these the abstract and the review by Ponduri et al were used to assess eligibility and extract outcomes. A total of 49 studies were included: eight studies of children with CP [3,21–27]; seventeen studies [19,20,28–42] identified in the previous review [9] and 24 studies [43–66] identified from six Cochrane systematic reviews relating to OME [67–72].

Generation of an outcome list

Each outcome measured was listed by study. Only individual outcomes were included, for example, where an outcome was measured using different methods this was counted as one outcome but the methods of measurement noted (S2 Table).

The number of outcomes measured in an individual study varied with a median of 6 outcomes (range 1–14 outcomes) per paper. Outcomes related to resource use were considered to be outside the scope of the COS. Consequently, the outcomes, “necessity to visit doctor” and “level of speech therapy support required” were not considered in the list of outcomes. The final list of outcomes used in round 1 of the Health Professionals Delphi comprised 45 individual outcomes (43 identified from the systematic review and two added by the Study Steering Committee) grouped under 14 domains (Table 1). The list of outcomes scored by parents and children included the combination of some outcomes, for example those that related to specific clinical observations, so that a total of 36 outcomes were scored (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Outcomes scored by health professionals, parents and children.

https://doi.org/10.1371/journal.pone.0129514.t001

Identification of outcomes of importance to parents and children with CP

Semi-structured interviews were completed with 43 parents of 37 children, and 22 children, according to the sampling matrix described in the trial protocol [13]. Interviews with parents lasted, on average, 40 minutes, whilst those with children took, on average, 20 minutes. Parent and child responses to specific questions about outcomes were cross checked against the outcomes list generated from the systematic review of the literature and, as the semi-structured interviews were in parallel to the first round of the health professionals Delphi, were also checked against any free text responses provided by health professionals in round 1. Two investigators (IAB and NLH) mapped each outcome from the interviews against the list of outcomes after round 1. No new outcomes were identified.

Online survey of parents and children

Two hundred and ninety three people accessed the online survey and 253 answered the initial question regarding eligibility. Of the 235 eligible only 51 (22%) completed the survey. Responses were received from 35 parents, eight adults and eight children. Of the eight children, four were in the 7–10 years age group and four aged 11–16 years.

The results were reviewed against the definition of consensus agreed prior to the start of the study [13]. Using this definition, parents and children had reached “consensus in” for 35 and 11 outcomes respectively (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of all groups reaching consensus for individual outcomes scored in the health professional Delphi survey and online survey for parents and children.

https://doi.org/10.1371/journal.pone.0129514.t002

Identification of Outcomes of Importance to Health Professionals

Round 1 –Health Professionals.

The overall response rate per round is given in S4 Table. The number of outcomes reaching consensus within each stakeholder group in each round is shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Number of outcomes achieving consensus.

https://doi.org/10.1371/journal.pone.0129514.t003

The 18 free text responses provided by health professionals in round 1 were reviewed by the SMG and SAG. Eight responses represented a comment and two relating to the use of hearing aids were considered outside the scope of the study. The remaining eight described potential outcomes of which two (listening skills and psychosocial wellbeing) were not already represented and therefore were taken forward to round 2. Thus in round two health professionals scored 47 outcomes.

Consensus Meeting

Twenty five participants attended the consensus meeting of whom 14 were eligible to vote. All stakeholder groups with the exception of clinical psychologists were represented (S6 Table).

Outcomes were discussed in the order of the number of stakeholder groups achieving consensus. Each outcome was categorized based on the following: 1 Discussed and voted (19 outcomes); 2 Discussed and agreed to combine with another outcome and to be considered as part of the “how” an outcome is measured (14 outcomes); 3 Discussed and agreed that further discussion with parents was needed (seven outcomes); 4 Agreed not to discuss further or vote—not in the COS (nine outcomes). A full breakdown of outcomes discussed at the consensus meeting is provided in S5 Table.

The evidence from the health professional Delphi, the parent and child survey and discussion at the consensus meeting followed by voting, were integrated and a COS proposed. This was then further discussed and approved at a follow up meeting of the SAG.

The consensus meeting followed by discussion with the SAG identified 11 outcomes that required further discussion with parents. For example, at the consensus meeting the outcome “listening skills” reached consensus for inclusion in the COS. This outcome was added by health care professionals as part of their Delphi and was not scored by parents or children. Consequently “listening skills” was considered to require further discussion with parents. Outcomes identified at the consensus meeting as requiring additional input, were discussed with parents at a follow up meeting held as a parallel workshop at the CLAPA annual conference, October 2014. Nine parents and one cleft surgeon took part in the workshop. The session included discussion and, if needed, further explanation of each outcome. Each outcome was scored anonymously using an electronic scoring system for immediate feedback. The scores from the workshop were combined with the scores from the consensus meeting and the definition of consensus applied. Following voting, “listening skills” was confirmed for inclusion in the COS. Two additional outcomes, “cholesteatoma” and “otalgia”, also reached consensus in. All outcomes meeting the definition of consensus and included in the recommended COS are shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Recommended core outcome set.

https://doi.org/10.1371/journal.pone.0129514.t004

Strengths and limitations

We have used an efficient online system to deliver a multiple round Delphi to health care professionals that allowed automated collating of scores and feedback in each round together with automated email alerts to promote completion. We have shown that individuals do reflect on, and are influenced by, other groups' opinions about the importance of outcomes. This online system was also successfully modified for use by parents and children in a one off survey.

Both clinical and patient stakeholders were engaged, with the response rate of health professionals similar to that reported in other Delphi surveys [73–75]. Notably the attrition rate was low with those taking part in round 1 likely to complete all rounds; as a result, no attrition bias was introduced. Whilst clinical stakeholder representation was good the number of parents and children completing the online survey and attending the face to face consensus meeting was lower than expected. In the present study it was not possible to ascertain whether the length of the survey for parents and children, the method of delivery or indeed the importance of the research question, due to perceived impact of OME, contributed to the low response rate. It is possible that those individuals completing the online survey are not representative of the wider group in terms of their views about important outcomes. Thirty seven parents were interviewed and these considered fewer outcomes to be critically important compared to parents who completed the online survey. However, with the exception of “externalising behaviour”, mentioned by one parent but not as the most important outcome, the opinions of those parents who were interviewed were comparable with the opinions expressed by parents completing the online survey. Research is needed on how best to engage with patients and/or their parents to facilitate patient involvement with the different stages of the COS development.

Future Work

The consensus meeting and follow up meeting with parents has resulted in a COS which recommends what to measure. However, if future research measures these outcomes in different ways it will still be difficult to compare studies. The next steps will involve consideration of how each of these outcomes should be defined and measured. For each outcome, definitions and measurement instruments will need to be reviewed, whether a validated tool already exists and what methods have been used to measure this outcome in previous studies, as described in the systematic review (S2 Table).

For each of the outcomes included in the recommended COS this will include: consideration of methods of assessing hearing that might be influenced by the intervention, for example, differing methods depending on ventilation tube or hearing aid use; agreeing a definition of COM and methods of measurement; determining which aspects of speech development should be measured and identifying whether methods of measurement are already available; reviewing methods for assessment of receptive language, psychosocial development, AOM, listening skills, cholesteatoma and otalgia; establishing the most appropriate way to measure side effects of treatment; consideration of the impact of patient age group on the chosen method of assessment. With the exception of “listening skills” all outcomes have been measured in one or more of the studies identified in the systematic review.

Guidelines for the selection of outcome measurement instruments to be included in a COS are being developed by the Core Outcome Measurement Instrument Selection (COMIS) project [76] and will be consulted when available. Furthermore, the UK Cleft Audit means that for some outcomes there are potentially methods of measurement that have already been agreed by health professionals providing cleft care in the UK [77,78].

Our study recommends a core outcome set but also acknowledges that further work is needed to identify agreed methods of measurement for each of the outcomes as this is beyond the scope of the current study. As part of the methods development process consideration will be given to the age at which outcome assessments are appropriate and this will inform the length of follow up needed in a given trial.

The length of follow up will not be mandated and a similar approach will be adopted as by the OMERACT group where, if appropriate, outcomes will only be considered core should they be appropriate to the age group of participants and duration of follow up. For example, the OMERACT core outcome set includes one outcome which is only relevant should the duration of follow up be greater than 52 weeks[79]. The MOMENT study has involved multiple key stakeholder groups from the UK to ensure that a COS is suitable and well accepted in future research. However, to promote good uptake of the COS into future studies international consensus is needed. Cleft organisations exist in both Europe (The European Cleft Organisation) and the United States (American Cleft Palate-Craniofacial Association) and we plan to work with COMET [80]to pursue engagement of international health professionals through their membership.

It should also be noted that a COS is not static and should be revised or updated as new information becomes available. Should future trials using the COS recommended in this paper identify difficulties in its application then review of the included outcomes would be warranted.

Whilst OME affects around 75% of children with CP, it is also a common condition for children without cleft, with almost a fifth of children aged 1–5 years affected [1]. The COS described in the current study includes outcomes that have been identified from previous studies in both cleft and non-cleft populations suggesting that they may also be of relevance to studies of OME in children without CP. Further work in this area and engagement with stakeholders is warranted.