Measures of Facilitator Competent Adherence Used in Parenting Programs and Their Psychometric Properties: A Systematic Review

Numerous parenting programs use observational or non-observational measurement tools to assess facilitator competent adherence. Observational methods require assessors to watch live or video-recorded sessions whereas non-observational methods require assessors to listen to audio recordings of sessions or use facilitator self-report (Girard & Cohn, 2016). Although time and resource intensive (Horvath et al., 2011), observational methods may be more accurate due to their objectivity and ability to capture facilitator and participant body language (Gardner, 2000). Non-observational methods are less reliable due to factors such as social desirability (e.g., facilitator self-reports) (Stone et al., 2000) and because they miss capturing important elements of delivery, including participation engagement (e.g., audio assessments). However, with observational methods, the presence of an observer or video-recorder may alter the facilitator’s natural delivery and, thus, introduce reactivity bias (Gardner, 2000).

Measures of facilitator competent adherence frequently use response types with dichotomous (e.g., ‘yes’ and ‘no’), frequency, or Likert-scale options. Dichotomous responses are clear cut, making it easier to establish inter-rater reliability between assessments, yet may miss nuances in program delivery, particularly intricacies associated with measuring skills. Frequency items ask assessors to capture the number of times a facilitator implements a particular activity or skill. With Likert items, assessors are given options that capture gradations in delivery that make it challenging to establish reliability.

Measures can also be designed to capture varying combinations of competence and adherence; measures may assess competence-only, adherence-only, competent adherence, or competence and adherence. Competence-only measures seek to assess the quality with which facilitators deliver a program. Competence is a subjective concept making the development of a tool to measure it complex; it is difficult to pinpoint, and therefore assess, the precise impactful outward manifestations of high-quality delivery (Mowbray et al., 2003). Using such a tool is challenging as interpretations of what is being observed may differ by assessor, thus varying their ratings on measure items. Adherence-only measures assess the extent to which facilitators implement a program as designed. Previous literature indicate that adherence-only measures are most common, potentially because they are quick and simple to design and use (Goense et al., 2015) but may miss capturing the nuance of program delivery. Measures of competent adherence detail whether and how well a facilitator delivers a program as intended. Measures that assess both domains are particularly insightful as they attempt to capture a complete picture of facilitator delivery.

It is important to study facilitator delivery as stronger facilitator delivery may be associated with better program outcomes. While some research in the parenting intervention literature has found that higher facilitator competent adherence is associated with better intervention outcomes (e.g., Eames et al., 2009, 2010; Forgatch et al., 2011; Hogue et al., 2008; Huey et al., 2000), other research has found the relationship unclear (e.g., Breitenstein et al., 2010; Cantu et al., 2010). The literature on the association between fidelity and outcomes may be mixed for a variety of reasons, including the difficulty of its study, the potential influence of confounding variables, and inaccurate measurement (such as by using measures that are not reliable and/or valid). Although numerous parenting programs have assessed facilitator competent adherence, no studies have systematically reviewed the measures used in these studies or documented their psychometric properties. Given measures of competent adherence capture a variety of dimensions using a range of approaches, this review provides a summary of their characteristics. The resulting synthesis is intended to provide practitioners and researchers with fundamental information and insights to assist them in selecting, adapting, developing, and using measures for their programs and studies. Even though many measures exist, they may not be reliable and/or valid. It is fundamental that fidelity measures are reliable and valid as these properties speak to whether they can be administered consistently and capture what they intend to measure (Mowbray et al., 2003). Information on measure reliability and validity is beneficial as it provides practitioners and researchers with a picture of the quality of implementation measures currently available; supports researchers in determining which measures need further testing and analysis; and it allows practitioners and researchers to quickly assess the desirability of using measures in future program implementation and evaluation. Thus, in compiling and analyzing the characteristics of measures of facilitator competent adherence used in parenting programs and by evaluating the psychometric properties and practicality of the observational measures found, this review takes the first step towards a comprehensive understanding of the role facilitator delivery plays in program effectiveness.

We developed a search string informed by related studies and systematic reviews (Barlow et al., 2017; Gardner et al., forthcoming):

(parent* OR caregiver* OR guardian* OR carer*.ab) AND (training OR program* OR intervention* OR treatment OR trial* or prevention.ab) AND (competen* OR quality OR adheren* OR fidelity* OR integrity OR compliance.ab) AND (child* OR kid* OR adolesc* OR teen* OR youth* OR baby OR babies OR toddler* OR neonate* OR infant* OR newborn OR juvenile* OR minor* OR early child* OR ECD.ab) AND (facilitator* OR practitioner* OR therapist* OR clinician* OR teacher* OR worker* OR provider* OR leader* OR specialist* OR professional* OR coordinator* OR administrator* OR counsellor* OR counselor* OR implementer* OR coach* OR instructor* OR trainer* OR mentor* OR educator*.ab) AND (scale* OR subscale* OR tool* OR measure* OR instrument* OR report* OR index* OR checklist* OR test*.ab).

We conducted our search in the following databases: Applied Social Sciences Index and Abstracts, Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials (CENTRAL), EconLIT, PsycINFO, EBSCO combined search (CINAHL, ERIC, MEDLINE), Global Health, The International Bibliography of the Social Sciences (IBSS), Social Science Premium Collection, and ProQuest Dissertations and Theses. Articles published from inception until July 2020 were eligible for inclusion. The initial search was conducted in January of 2020 and updated in July 2020.

We hand searched articles included in Gardner et al.’s (forthcoming) review to ensure representation of parenting programs from low- and middle-income countries (LMICs), where there are typically fewer studies. Backward reference searching was conducted using the reference lists of all included articles, and forward reference searching was conducted using Google Scholar. We also asked experts focused on parenting program fidelity, including Bearrs, Eames, Forgatch, Hogue, and Smith, to share relevant published or ongoing studies. Finally, once a list of measures used in each included study was generated, we searched the names of these measures in the databases listed above to capture any additional articles.

We evaluated the quality and risk of bias of both the included studies and measures identified using three risk of bias and quality checklists: one to evaluate measure properties, one to evaluate studies, and one to evaluate measure practicality. Each checklist is composed of questions outlining high- and low-quality criterion on a number of dimensions wherein ‘+’ represents high-quality, ‘−’ represents low-quality, and ‘?’ indicates insufficient information.

A Measure Risk of Bias and Quality Checklist was developed based on the COSMIN guidelines including adaptations made by other authors (Gridley et al., 2019; Mokkink et al., 2010a, 2010b; Terwee et al., 2007). A three-item Study Risk of Bias and Quality Checklist was developed based on relevant literature. Item one addresses the session sampling method with high-quality studies either observing or rating all program sessions or randomly selecting sessions for observation to reduce selection bias (Ellenberg, 1994; Walton et al., 2017). Item two addresses assessor bias with high-quality studies using two or more independent assessors to conduct observations so that assessments are not biased and so that inter-rater reliability can be established (Hallgren, 2012; Walton et al., 2017). Item three considers the role of facilitator reactivity with high-quality studies providing information on steps taken to reduce reactivity (Gardner, 2000; Kazdin, 1982).

We also developed a three-item checklist to evaluate measure practicality based on the feasibility and sustainability of the training provided to assessors, the measure’s utility in practice, and the availability of the measure in-text or online (Barkham et al., 1998; Milne et al., 2011).

To ensure replicability, inter-coder reliability was established at the title/abstract screening, full-text screening, and data extraction stages (Belur et al., 2018). Following training, the second author independently coded a random selection of 10% of the articles at each stage of study inclusion (Lombard et al., 2017). Discrepancies were resolved through discussion.

Electronic bibliographic searches surfaced 20,327 articles. Further, Gardner et al. shared 151 articles included in their review. No new measures surfaced from the 20 experts contacted. After duplicate removal, 9153 articles remained. Article screening resulted in 156 studies included in Part One and 41 in Part Two. All searches were conducted on the same day in January 2020 and updated in July 2020 (Fig. 1).

Reliability between coders was 94.4% at Part One title/abstract screening, 100% at Part One full-text screening, 92.8% at Part One data extraction 1, 93.8% at Part Two full-text screening, and 100.0% at Part Two data extraction.

Table 1 describes the characteristics of the measures found in the included studies. The 156 Part One studies assessed 65 measures of which 46 were named and 19 were unnamed. The most commonly reported were Therapy Adherence Measure (n = 16); Fidelity of Implementation Rating Scale (n = 8); Therapy Adherence Measure-Revised (n = 7); COACH Rating System (n = 5); and the Leadership Observation Tool (n = 3).

Of the 151 studies reporting on domains measured, 73 articles (48.3%) captured adherence-only, 52 (34.4%) captured competent adherence, 21 (13.9%) captured competence and adherence, four (2.6%) captured competence-only, and one (0.7%) captured adherence and competent adherence.

One hundred and thirty-three (85.3%) studies reported on the mode of data collection used to make assessments with 53 (40.0%) using observational methods, 55 (41.4%) using non-observational methods, and 25 (18.8%) using both. These approaches included video (31.6%), memory (29.3%), combination (21.8%), audio (6.8%), live (6.8%), and other (3.0%) (live or video; case notes).

One hundred and thirty-five (86.5%) studies reported on the type of individual who conducted assessments. Assessors were facilitators (15.6%), researchers (14.1%), third parties (12.6%), parents (8.9%), supervisors (7.4%), combinations of assessors (21.5%), ‘other’ (including program trainers, adolescents, community workers, independent facilitators, experts, and psychologists) (8.9%), or combinations and other (11.3%).

One hundred and seven (68.6%) studies reported on response options, which were dichotomous, Likert-scale, frequency, ‘other,’ or combinations of formats. Nineteen studies reported dichotomous formats either alone or in combination. Of these, 11 measured adherence-only, four measured competent adherence, four measured competence and adherence, and none measured competence-only. Thus, all studies using dichotomous measures reported on adherence either alone or in combination with competence. Ninety-five reported Likert response options either alone or in combination with other item formats, with most providing a definition (e.g., 5 = very much) for each Likert point. Response options ranged from 0 to 11. Of these 95 studies, 43 measured competent adherence, 20 measured competence and adherence, 30 measured adherence-only, and two measured competence-only. Of fifteen studies reporting frequency and ‘other’ formats (some in combination), five recorded minutes, eight coded the frequency of activities or skills, and two used a coding system with the notations ‘+’, ‘−’, and ‘not applicable’ to report on the assessment of facilitator delivery.

Fifty-two (33.3%) studies reported on whether assessors were provided with training on how to conduct assessments. Sixteen (30.8%) reported on the amount of training received, which averaged 20 h. Twenty-eight (53.8%) reported providing training for assessors measuring competent adherence, nine (17.3%) for competence and adherence, 15 (28.8%) for adherence-only, and none for competence-only. Thirty-two (61.5%) reported training for assessors of observational measures, 12 (23.1%) for non-observational measures, seven (13.5%) for both methods, and one (1.9%) did not report on the mode(s) of data collection. All training was provided to third-party assessors (e.g., researchers, program staff) and none was provided to facilitator or parent assessors. One study reported how long it took to complete an assessment. No studies reported on the cost of training assessors and conducting assessments.

The results regarding facilitator characteristics are described in Table 2. Thirty-eight (24.4%) studies reported on facilitator age which ranged from 23 to 68 years, with most studies (89.5%) having some or all facilitators between 31 and 40 years. Ninety-seven (62.2%) studies reported on the sample size of facilitators studied. The average sample was 65.8 with a median of 20.0 facilitators.

Fifty-seven (36.5%) studies reported on facilitator gender. The average percentage of female and male facilitators was 77.5% and 22.5%, respectively. Seven (12.3%) studies reported all-female teams. Twenty-eight (17.9%) studies reported on the amount of experience (years or number of times) facilitators had delivering parenting programs.

Ninety (57.7%) studies reported on the educational background of facilitators. Of these, 47 reported facilitators with a master’s degree (52.2%), 34 with a doctorate (37.8%), and 29 with an undergraduate degree (32.2%). Backgrounds in psychology (27 studies) and social work (28 studies) were most common. Mean years of career experience were 7.5 years in the 23 (67.6%) studies providing this information. Forty-one studies (26.3%) documented facilitator ethnicity.

The 156 studies reported on 63 different parenting programs, with some reported in more than one study. The programs reported most frequently were MultiSystemic Therapy (n = 22), Incredible Years (n = 16), Parent–Child Interaction Therapy (n = 8), Parent Management Training—Oregon Model (n = 8), and Triple P (n = 6). The studies sought to address one or more of the parent and child outcomes of interest, including child behavior (n = 119), positive parent–child relationships and interactions (n = 29), child maltreatment (n = 21), behavior management (n = 17), early childhood development (n = 13), parent–child attachment and bonding (n = 10), and harsh or dysfunctional parenting (n = 6). The studies were conducted in 35 countries. Ninety-eight (62.3%) studies were conducted in the USA, 23 (14.7%) in England, Wales, or Ireland, and seven (4.5%) in the Netherlands. Only eight (5.1%) were conducted in LMICs. Thirty-eight (24.4%) studies reported on parent/caregiver ages, which ranged from 20.0 to 75.5 years with a mean of 39.9 years. One hundred and eleven studies (71.2%) reported on child age with 12 reporting on infants (0–1 year), 24 on toddlers (1–3), 47 on preschoolers (3–5), 57 on middle childhood (6–11), 61 on young adolescents (12–14), and 44 on older adolescents (15–17).

Each study was evaluated using the Study Risk of Bias and Quality Checklist (Table 5).

Twenty (48.8%) studies reported on internal consistency (Table 6). Thirteen studies reported Cronbach Alphas (0.64 to 0.98), eight reported Spearman correlations (− 0.61 to 0.90), and one reported ICCs (0.50 to 0.90). Two (4.9%) studies reported on intra-rater reliability by providing percentage agreements (60.0 to 87.0%) and ICCs (− 0.143 to 0.935). Both studies used one assessor and did not report the interval between assessments. All studies reported on inter-rater reliability. Twenty-one reported ICCs (− 0.03 to 0.96), 17 reported percentage agreements (50.0% to 100.0%), 11 reported Cohen’s Kappas (− 0.01 to 0.97), three reported Pearson’s correlations (− 0.04 to 0.88), one reported Gwet’s ACs (0.85 to 0.91), and one reported a Cronbach Alpha (0.87). Thirteen reported on inter-rater reliability in multiple ways (e.g., percentage agreements and Kappas). No studies reported on test–retest reliability.

Based on the Measure Risk of Bias and Quality Checklist results (Table 5), 21 (51.2%) of the 41 studies reporting inter-rater reliability were high quality (ICC > 0.70; Kappa > 0.70; r > 0.80). Of the two studies reporting intra-rater reliability, one was high quality (ICC > 0.70). Of the 20 (48.8%) studies reporting internal consistency, 11 (63.0%) were high quality (Cronbach Alpha > 0.70).

Only three (7.3%) studies reported on content validity (Table 6). Of these, two indicated using the knowledge and advice of experts to develop the measure (Feely et al., 2018; Hogue et al., 1998), two reported asking experts to assess a draft measure (Feely et al., 2018; Hogue et al., 1998), one described seeking feedback from program trainers about the developed measure (Bywater et al., 2019), and one asked a program developer to review their draft measure (Hogue et al., 1998). They all provided very little detail about the process, feedback, or use of feedback.

Nine (22.0%) studies reported on construct validity (Tables 6 and 7). Four performed confirmatory factor analyses, three performed principal components factor analyses, one performed principal axis factor analysis, and one performed exploratory factor analysis. One (2.4%) study assessed convergent validity, reporting interscale correlations among the measure of interest and similar measures with Pearson’s correlations that were statistically significant at the p < 0.01 level ranging from 0.35 to 0.55. No studies reported on divergent validity or criterion validity.

Each study was evaluated using the Measure Risk of Bias Checklist (Table 5). The one study that reported on convergent validity did not meet the quality criteria as correlations between the scales were not all above 0.50. Of the three reporting on content validity, one was high quality and two did not provide sufficient information. The studies reporting on content validity provided insufficient details about the process, feedback, and use of the feedback to make an evaluation of the study’s content validity. Of the nine studies reporting on construct validity, four were high-quality and five were low-quality.

Part Two of the review found limited psychometric evidence on measures of facilitator competent adherence. Of the 41 studies analyzed, the only psychometric property reported consistently was inter-rater reliability. Internal consistency was the next most frequently (48.8%) reported property, with only 22.2% reporting on construct validity, 7.3% on content validity, 4.9% on intra-rater reliability, 2.4% on convergent validity, and none on test–retest reliability or criterion validity. Despite the limited evidence, most measures had moderate reliability and validity, with the measures reported in more than one study being the most robust.

The risk of bias and quality assessment results suggest that studies are of moderate quality. Further, an evaluation of measure practicality suggests that, among the studies providing sufficient information, the measures are moderately practical to use. It is fundamental that interventions and their components are practical, particularly at scale (Fixsen et al., 2019).

This systematic review has a number of strengths. It is the first to synthesize and critically appraise both measures of competent adherence and the psychometric properties of observational measures in the parenting program literature. The high inter-rater reliability achieved between coders suggests minimal human error in screening and data extraction. Moreover, a checklist was created wherein a number of considerations were added to the COSMIN guidelines to take the nature and challenges of observational measurement into account. Further, questions were created to assess the usability of measures in practice. On the other hand, the review was limited by the paucity of evidence on the psychometric properties of measures and this lack of data hinders strongly supported findings from being made. Additionally, measures of facilitator competent adherence are often program specific. As a result, the review’s ability to recommend the selection of certain measures for broader use is limited. However, the review provides critical information to help guide the adaptation and development of future measures and to conclude that assessments of facilitator competent adherence are being made based on measures with limited psychometric evidence.

Findings from this review suggest a number of recommendations for future research and practice. Reporting guidelines for studies reporting on fidelity measures should be created so that measure characteristics (e.g., method of completion, assessors) and psychometric properties, as well as essential facilitator and intervention characteristics, are consistently documented. Reporting guidelines would advance the knowledge generated by reducing reporting bias and providing a framework from which to evaluate measure validity and reliability. The findings also suggest that there are a variety of facilitator characteristics that need exploring in future studies to determine if outcomes are associated with facilitator characteristics and levels of competent adherence. Reporting guidelines would allow researchers to learn from each other in order to create, modify, and use high-quality measures of competent adherence in research and practice.

A subsequent systematic review exploring the predictive validity of facilitator competent adherence measures is recommended. Such a review would reveal whether increased facilitator competent adherence is associated with better parent and child program outcomes. The results of this further meta-analysis may contribute to an eventual clarification of the mixed evidence regarding the association between facilitator delivery and participant outcomes.

More rigorous research on measures of facilitator competent adherence should be conducted, particularly as it relates to reliability and validity. Findings as to whether measures of facilitator competent adherence are reliable and valid will help establish if assessment results can be relied on, enhance facilitator delivery, and determine whether competent adherence is correlated with outcomes for program beneficiaries (Fixsen et al., 2005). In addition, more studies of competent adherence are warranted in LMICs. The need for low-cost parenting programs is greatest in LMICs, thus, heightening the necessity for solid evidence on improving program delivery and outcomes (Mercy et al., 2008).