Enhancing Social-Emotional Outcomes in Early Years (E-SEE): Randomized Pilot Study of Incredible Years Infant and Toddler Programs

The study did not achieve the target sample size at baseline. There were initial barriers around participant identification and recruitment, for example, participant identification began the week that commissioning for health visiting (HV) in England moved from the NHS to Local Authorities. Although senior site management were fully committed to the trial, health visitors faced a lot of uncertainty and had limited capacity which may have led to reduced identification and recruitment. Documentation with regard to working partnerships, expectations and site requirements should be clear and detailed in a definitive trial. Awareness raising and training resources for health visitors and other staff involved in identifying potential participants, in an electronic format that can be easily shared and accessed by professionals working in a range of contexts, might also improve recruitment rates. One site had a late start, due to delays in securing finance to cover health visitor time to train in and deliver the IY programs. This impacted on time available for participant identification and recruitment, however a simplified process for seeking intervention delivery costs and approvals from the UK’s Health Research Authority (HRA) for trials conducted in health and social care settings should alleviate this issue for a future definitive UK trial of E-SEE STEPS.

The challenges of recruiting to RCTs are well documented, with one recent review suggesting that 56% of publicly funded RCTs in the UK fail to achieve their target sample size (Walters et al., 2017). The Qualitative Research Integrated within Trials (QuinteT) Recruitment Intervention (Donovan et al., 2016) provides a methodology for understanding the process of recruitment to RCTs, identifying difficulties and producing a plan to address those difficulties, all in real-time. Our pilot study suggests that this approach should be considered in the design of a definitive trial.

Despite achieving only 205 (71%) of the target sample of 288, the high retention rate of 181 (88%) participants at final follow-up approximated the expected number at trial end (n = 192, 94%), as 32% attrition was estimated, but only 12% occurred. This level of retention is encouraging, particularly when considering that families were recruited during a major life event (soon after birth of a child), the inclusion of four timepoints and final follow-up was approximately 18 months after baseline. The range of strategies we employed to support retention should be replicated in a definitive trial. The PAC reviewed the measures and materials and participated in training our data collectors in data collection procedures (using role-play scenarios) during study set-up. This ensured that measures were ‘parent friendly’ and acceptable to parents of young children, and that data collectors were confident in administering them. Data collectors left ‘change of address’ forms and freepost envelopes along with E-SEE branded trolley tokens and pens (that had our contact telephone number on) with participants at each data collection visit. Parents who notified us of a change of address received a £10 shopping voucher as a token of appreciation. When booking data collection visits we contacted parents by phone to discuss and arrange appointments and they received a letter in the mail confirming the dates and times of the appointment and the name of the visiting data collector. We had teams of data collectors local to each site, and wherever possible we ensured the researcher who visited the family was consistent across each timepoint. When we could not reach a parent by phone, we sent an ‘appointment by letter’ notifying the parent that a named data collector would visit them at a specific date and time or they could arrange a more convenient time (if the letter was unsuccessful twice we marked the participant ‘lost to follow-up’). Those who we were unable to contact and marked lost to follow-up were contacted again at the next follow-up timepoint. We are unable to pinpoint specifically which, if any, of these strategies resulted in the successful retention of parents, and therefore recommend future studies explore the differential effectiveness of such strategies (for example using ‘Studies within a trial’ (SWAT) approaches, Clarke et al., 2015).

We initially envisaged Health Visitors co-leading all intervention groups with LA staff, however with the move of health visiting commissioning arrangements to LAs, changes in service structures and budgets, and the fact that services are delivered differently across sites we propose that delivery agents should be specific to local context needs, that is, both HVs and other LA staff could be IY group leaders, or just LA staff such as children center staff, or just health visiting staff. This flexibility reflects variation in real-world service set-up.

The planned number of IY-T groups was delivered in the two sites, however we delivered fewer IY-I groups than planned, due to a lower than expected percentage of eligible parents at FU1. Implementation partners in both sites also struggled to sufficiently engage and retain eligible parents to meet the target of at least five parents attending at least 50% of sessions in each group, although eight or more parents were invited per group. Reasons why uptake of the invitation and initial attendance may have been low are outlined below.

The number of parents meeting the designated threshold on the depression screener (PHQ-9) was much lower than expected. This could have been because not all families were informed of the trial, or that families with threshold scores chose not to hear more about the trial from the researchers, or that the population depression data we used to estimate prevalence of maternal depression was inaccurate. The low number of parents with depressive symptoms meant fewer eligible parents for the IY-I and IY-T programs at FU1 and FU2. We therefore reduced the threshold temporarily from a PHQ-9 score of 5 to 4 for the pilot, and added the ASQ:SE-2 as a screener as it was the primary outcome plus we noticed that some parents who scored their child in the ASQ:SE-2 monitoring zone did not score themselves as depressed on the PHQ-9 screener. Since the IY programs were chosen to enhance child social emotional well-being, and the demonstrated relationship between children’s early difficulties and parental mental health (Quist et al., 2019), using two screeners is acceptable. Both screeners should be used in a definitive trial.

Timing of groups and availability of wrap-around support may have been a factor for low uptake. One site had staff capacity issues that meant that the IY-T program could only be run on limited weekdays, with no evening sessions and no crèche facility (a crèche is supervised childcare offered to parents while parents undertake some activity, usually on the same premises). Inflexible delivery times made it difficult for working parents, or those with childcare challenges to attend a group. Flexible group delivery times, to suit parents, should be included in site level agreements in a definitive trial. In addition, some groups were delivered during school holidays to meet study timeframes, which reduced parents’ ability to attend (e.g., those with older children). The timings for IY-I and IY-T delivery in a definitive trial should be mapped to ensure they can be delivered in term time to retain inclusivity.

To ensure future viable groups (minimum of 5 parents) we recommend inviting up to 12 parents per group, working with sites to implement effective engagement and retention strategies, and allowing ‘non-research’ participants to be invited to low-uptake groups to promote collaborative learning. Non-research participants would need to have similar levels of need to eligible research parents and have children a similar age, while no research data would be collected from them, their presence enables group dynamics to operate (a key element of the IY-I and IY-T theory of change). Additional support within the model to boost engagement of parents in the intervention at the outset should be considered. For example, Fosco et al. (2014) included motivational interviewing techniques in the Family Check-Up model and found a significant relationship with intervention response.

Group retention levels for participants who attended at least one of the weekly sessions were high; 73% for IY-I and 87% for IY-T. This suggests that participants found the intervention acceptable and is consistent with reported high levels of satisfaction in weekly and end of program questionnaires. These findings are comparable to other IY studies (e.g., Morpeth et al., 2017; Jones et al., 2016). However, participants did highlight potential areas of improvement for the programs that could be considered in a future trial (see Table 1).

We encountered unanticipated challenges in relation to practitioner perspectives on the acceptability of IY. The IY programs were developed in the US and, during the pilot, sites raised concerns about risk to their UNICEF Baby Friendly Initiative (BFI) accreditation as the IY-B book and IY-I program contained some information around feeding and sleeping that did not conform to BFI recommendations (although it conformed to US and World Health Organization guidance). This was initially managed by inserting a one-page addendum to the book informing parents that the book was from the US and signposting them to UK guidance. Local BFI Infant Feeding Coordinators, HVs, IY trainers and members of our PAC inputted to the addendum. The research team were transparent about the BFI concerns and worked closely with the IY developer, IY trainers, local service staff and local BFI staff while the developer revised the book and published an updated version in 2017 which is now used worldwide. These challenges highlight the limitations of printed books, as they can quickly become outdated as guidance inevitably changes. E-books or online materials can be more easily updated (although there are associated risks in relation to digital exclusion).

Notional thresholds of 80% fidelity were largely achieved in relation to different aspects of intervention fidelity (particularly in relation to quality of delivery and participant responsiveness). There were some discrepancies between self-reported adherence and objective coding of videotaped sessions by researchers. Subjective, desirability biases, can influence the outcomes of self-report measures of fidelity (Bywater et al., 2019) highlighting the importance of incorporating independent measures of fidelity in trials. Overall IY-I and IY-T were being delivered as intended and the training and supervision model for E-SEE STEPS is sufficient. However, “dry run” delivery practice for newly trained group leaders is recommended to promote intervention fidelity within a definitive trial.

The E-SEE trial was originally designed to deliver four primary analyses examining the evidence for the overall effect of the intervention at FU3, and the effect of the individual dose levels of the intervention (i.e., the IY book, IY-I and IY-T). However, the evaluation of the component level effects was always going to be compromised due to the lack of randomization at each subsequent proportionate stage. While a series of two further randomized stages would have enabled us to evaluate the individual components, it would have made the trial prohibitively large and hence power to detect an overall effect of the intervention was prioritized. We therefore, propose an alternative design consisting of a single research question “Do the scores of children in the IY arm, on average, stay below those scores for children in SAU over the three follow-up measures?” The benefit of asking a single research question is that power is increased and there is no need for Bonferroni adjustment. The disadvantage is that we are unable to examine the potential effectiveness of each of the three levels, and each of the four doses of intervention. Arguably this approach is more appropriate, given that the intention is to implement and scale E-SEE STEPS as a whole proportionate universal intervention, rather than individual components or programs.

We will use all the follow-up measurements in a single mixed effects model repeated measures analysis informed by the results generated by the secondary endpoints from the pilot. Rather than simply expecting scores to be lower at FU3 adjusted for baseline, we now expect the average scores to be lower at each follow-up in IY compared to SAU. We assume that correlation with baseline and FU3 will be the same as observed in the pilot for baseline and FU2 (0.26). As was justified in the original design we set the clinically important difference at FU3 to be 5 units of the ASQ:SE-2. We expect this effect to be consistently seen over the three follow-up points (original analyses assumed 5.5 points difference at FU2, so we are allowing for a slightly lower average effect overall). The sample size for the full definitive trial is calculated bearing in mind we need to ensure that there are sufficient numbers of eligible and willing to attend parents for the IY-I and IY-T groups to be viable. Groups need to have at least 5 parents attending. So our calculations are driven to deliver an expected total of 48 parents who will attended the IY-T sessions. We further assume that the IY-I and IY-T groups may lead to treatment induced clustering of outcomes and hence apply a design effect inflation factor to account for this clustering in one arm. These two requirements lead to an unbalanced allocation ratio. Assuming an average effect of 5 units below SAU, the SD at FU3 is 18 and a design effect of 1.25 for the IY arm with a two-sided 5% significance level and 90% power we would require retention at FU3 of 441 in SAU and 92 in IY. Allowing for the overall attrition of 12% we would require 606 to be randomized with an allocation ratio of 4.8:1 for the definitive trial.

The study demonstrates that LA and HV services can work together to deliver IY-I and IY-T in a proportionate universal model. When delivering parent programs services should consider using both a parent and child screener to ensure families get the support they need, and enhanced engagement activities may be required in order to improve the uptake of offer to IY. The findings imply that any trial of an intervention to promote social and emotional development of infants needs to pay careful attention to whether intervention materials comply with BFI guidance as BFI accredited services may be wary about delivering interventions which may threaten their accreditation status.

The sample were not as depressed as population data suggested, which led to lower eligibility numbers for the IY groups. The E-SEE STEPS model is a proportionate universal approach and a representative sample is key to test the model. We are exploring the issues of representative samples within trials and generalizability (Gridley et al., submitted) using this pilot data and data from a similar study (Leckey et al., 2019). Low take up of IY-I and IY-T is a limitation that needs to be address in a future definitive trial.