Comparing, Contrasting, and Integrating Dissemination and Implementation Outcomes Included in the RE-AIM and Implementation Outcomes Frameworks

As the field of dissemination and implementation science matures, there are a myriad of outcomes, identified in numerous frameworks, that can be considered across individual, organizational, and population levels. This can lead to difficulty in summarizing literature, comparing across studies, and advancing translational science. This manuscript sought to (1) compare, contrast, and integrate the outcomes included in the RE-AIM and Implementation Outcomes Frameworks (IOF) and (2) expand RE-AIM indicators to include relevant IOF dissemination and implementation outcomes. Cross tabular comparisons were made between the constitutive definitions of each construct, across frameworks, to reconcile apparent discrepancies between approaches and to distinguish between implementation outcomes and implementation antecedents. A great deal of consistency was identified across approaches, including adoption (the intention, initial decision, or action to employ an evidence-based intervention), fidelity/implementation (the degree to which an intervention was delivered as intended), organizational maintenance/sustainability (extent to which a newly implemented treatment is maintained or institutionalized), and cost. The IOF construct of penetration was defined as a higher-order construct that may encompass the reach, adoption, and organizational maintenance outcomes within RE-AIM. Within the IOF approach acceptability, appropriateness, and feasibility did not match constitutive definitions of dissemination or implementation but rather reflected theoretical antecedents of implementation outcomes. Integration of the IOF approach across RE-AIM indicators was successfully achieved by expanding the operational definitions of RE-AIM to include antecedents to reach, adoption, implementation, and organizational maintenance. Additional combined metrics were also introduced including penetration, individual level utility, service provider utility, organizational utility, and systemic utility. The expanded RE-AIM indicators move beyond the current approaches described within both the RE-AIM framework and IOF and provides additional planning and evaluation targets that can contribute to the scientific field and increase the translation of evidence into practice.

Introduction

As the field of dissemination and implementation science matures, there are a myriad of outcomes that can be considered across individual, organizational, and population levels (1–4). The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) Framework was one of the first outcomes-focused approaches to address individual and organizational factors that would, if assessed and optimized, improve the generalizability of efficacy trials, and speed the translation of evidence-based interventions into sustained practice (5, 6). More recently, Proctoret al. introduced the Implementation Outcomes Framework (IOF)—specific to dissemination and implementation trials (4). Across these two approaches, 12 dissemination and implementation outcomes are proposed—some are distinct, some overlap, and some are duplicated—which can lead to difficulty in summarizing literature, comparing across studies, and advancing translational science.

The RE-AIM Framework includes 6 dimensions that focus planning and evaluation on balancing internal and external validity to develop intervention approaches that can achieve a public health impact (see Table 1). The framework includes dissemination outcomes at the individual (i.e., patient; reach) and organizational (i.e., adoption) levels. It also includes implementation outcomes that are operationalized at the organizational level (i.e., implementation and organizational maintenance). Finally, clinical outcomes are operationalized at the patient/participant level (i.e., effectiveness, maintenance). The overarching planning and evaluation goals of RE-AIM could be described as developing and testing interventions that (1) have the potential to reach a large and representative proportion of the intended audience, (2) effectively improve and sustain positive health outcomes, (3) have high adoptability across a large and representative proportion of the population of staff and settings intended to enact the intervention, (4) can be consistently implemented with a high degree of fidelity to underlying evidence-based principles at a reasonable cost, and (5) can be sustained in typical clinical or community settings (8).

TABLE 1

Table 1. Definitions of IOF and RE-AIM outcomes.

Conceptualization of the RE-AIM framework has evolved over the past 20 years (6) to include a focus on qualitative research (1), cost across dimensions (9), and use in hybrid effectiveness-implementation research (10, 11). RE-AIM also provides composite metrics that address different aspects of intervention impact (Table 1) (12). Specifically, individual level impact can be determined using a composite measure of reach and effectiveness/maintenance using participation rate weighted by representativeness and standardized effect size weighted by differential effects across population subgroups. Attributable individual level impact can be determined by including population prevalence in the equation. Efficiency of individual level impacts was also proposed using the cost per unit of reach by effectiveness. Setting level impact measures can be calculated combining adoption rates and implementation fidelity—again with an option to include cost differentials. Finally, an overall summary index can be calculated by including composite equations for reach, effectiveness (or individual level maintenance), adoption, and implementation (12).

The IOF presents eight implementation outcomes, including: acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration, and sustainability (Table 1) (4). The IOF outcomes were conceptualized to improve the quality of dissemination and implementation trials, through the inclusion of measurable outcomes to enhance understanding of implementation success and processes. The IOF was also developed to distinguish between implementation, service, and client outcomes, develop a taxonomy of implementation outcomes, and highlight relationships across implementation outcomes at various stages in implementation research. For example, when considering the phases proposed in the Exploration, Preparation, Implementation, and Sustainment Model (EPIS) (13) differential levels of salience are proposed for each outcome. Acceptability, appropriateness, feasibility, and cost were considered most salient during the exploration phase, though each were also considered to have a lower degree of salience during preparation (i.e., appropriateness, feasibility), implementation (i.e., acceptability, cost), and sustainment (i.e., acceptability, cost) (7). Adoption was the only outcome considered to be most salient during preparation and was not considered salient at any other phase of implementation research. The outcomes that were considered to have primary salience for the implementation phase included fidelity and penetration while fidelity and sustainability were considered the most salient factors for the sustainment phase (7).

Similar to the RE-AIM framework, the conceptualization of IOF outcomes has also evolved over the 9 years since its first publication (4, 7). Of note, the concept of feasibility at the organizational level was extended to include feasibility at the service recipient level (7). Similarly, penetration was described as conceptually similar to reach and some researchers have extended the definition to include service recipients in addition to the service setting and its subsystems (7). Finally, similar to the RE-AIM framework the definition of cost has been refined to include cost of implementation (4), incremental costs (7), and overall financial impact of implementation efforts (14).

Both the RE-AIM framework and the IOF have had a significant impact on the field of implementation and dissemination science. RE-AIM provides a systematic planning and evaluation model that is based on individual and organizational outcomes, while the IOF provides conceptual clarity to distinguish between implementation, service, and client outcomes. Yet, there is considerable overlap between the frameworks and, based on the initial goals of the frameworks, key distinctions. This manuscript sought to compare, contrast and integrate dissemination, and implementation science outcomes included in these frameworks and provide working definitions that could extend the current RE-AIM indicators and outcome measurement approach.

Methods

Operationalization of RE-AIM and IOF Outcomes

Cross tabular comparisons were made between the constitutive definitions of the IOF and RE-AIM framework variables (see Table 1). Framework definitions were sourced from both the source papers as well as updated conceptualizations and models to increase the likelihood that definitions included advancements since initial publication (4, 6, 15). Using a content validity approach, all co-authors independently coded IOF constructs across the RE-AIM dimensions as either; (i) consistent between frameworks, (C); (ii) potential combined metrics (CM) (i.e., IOF construct aggregated across RE-AIM dimensions), or (iii) predictors (P) (or antecedents) of dissemination or implementation (Table 2). The study team met monthly between July and December 2019 to discuss and agree on coding for content analyses, to compare individual coding and resolve any discrepancies between team members through consensus. Coding of the framework definitions was completed by a senior scientist, post-doctoral fellow, and two doctoral candidates all specializing in dissemination and implementation science.

TABLE 2

Table 2. Cross-tabular comparison of RE-AIM and IOF outcomes.

For the purpose of this comparison, we defined “predictors” as constructs that act as precursors to implementation and dissemination of evidence-based interventions. For example, an intervention would need to be perceived as acceptable in order for it to be adopted. Characterization as a predictor was based on the degree to which the construct definition aligned with constitutive definitions of dissemination (i.e., an active approach to spreading evidence-based interventions to a target audience) and implementation (i.e., the process of delivering or enacting evidence-based interventions according to protocol or principles) (15). The level of analysis operationalized as individual (reflecting service recipients, patients, participants) or organizational (reflecting staff, settings, and organizations) was also identified. Gaps identified across the RE-AIM framework and the IOF were also considered and addressed through a proposed expansion of the operational definitions of RE-AIM indicators. Specifically, while cost and adaptation have both been discussed and examined in the context of both frameworks—methods to operationalize both have been limited (4, 7, 16–19). Framework operational definitions based on the cross tabular comparisons are shown in Table 2.

Results

Operationalisation of RE-AIM and Implementation Outcomes

A great deal of consistency was identified across approaches, including adoption (i.e., the intention, initial decision, or action to try or employ an evidence-based intervention), fidelity/implementation (i.e., the degree to which an intervention was delivered as intended), organizational maintenance/sustainability (i.e., extent to which a newly implemented treatment is maintained or institutionalized), and cost. However, cost was more explicitly defined in the IOF as cost of an implementation effort and of any strategies that targeted improvements in implementation whereas the RE-AIM conceptualization of cost focused on implementation and cost-effectiveness. The IOF construct of penetration was defined as a higher-order construct that may encompass the reach, adoption, and sustainability outcomes within RE-AIM. Within the IOF approach there were also a number of constructs that reflect theoretical antecedents of implementation outcomes including acceptability, appropriateness, and feasibility—rather than reflecting the constitutive definitions of dissemination or implementation. Table 2 outlines the cross tabulation of the constructs of the IOF and RE-AIM frameworks.

Expanded Operationalization of RE-AIM Dimensions

Integration of the IOF constructs with the RE-AIM framework was successfully achieved by extending the operational definitions of each RE-AIM dimension using IOF outcomes and antecedents. In addition, adaptation and cost considerations by RE-AIM dimension—both highlighted, but not explicitly included across dissemination and implementation outcomes were included in the expanded indicators; see Table 3. The dimensions of effectiveness and individual-level maintenance were the only RE-AIM components that were not expanded through this process. In addition, the IOF concepts of adoption and sustainability were identified as duplicates with the RE-AIM domains of adoption (staff/service provider- and organizational-level) and organizational-level maintenance, respectively, and were operationalized as such.

TABLE 3

Table 3. Expanded operationalization of RE-AIM.

The IOF constructs of acceptability, appropriateness, and feasibility were included as multi-leveled variables across reach, adoption, implementation, and organizational maintenance. However, the level of application is hypothesized to differ by dimension and temporality of assessment of the construct relative to the initial implementation (e.g., before, during, and after). Specifically, initial perceptions of intervention acceptability, appropriateness, and feasibility were operationalized as unique antecedents of reach (i.e., individual-level; participants/patients) and adoption (i.e., staff, setting, organization-level; service providers/organizational decision makers). In each of these cases the temporal assessment of these constructs and the potential for predictive validity is hypothesized to be dependent on the initial perceptions of the intervention prior to individual (reach) or organizational (adoption) decisions on engagement or uptake. In contrast, organizational experience—indicating a later temporal assessment following the initial actions of implementation—of acceptability, appropriateness, and feasibility were hypothesized to be antecedents of implementation fidelity (i.e., staff, setting, organizational-level) and organizational-level maintenance.

Cost specification was also expanded across reach, adoption, implementation, and organizational maintenance outcomes. An overarching consideration included that for most outcomes at least two categories of costs could be assessed—the cost of a dissemination or implementation strategy used to enhance a specific RE-AIM dimension and the cost of completing the activities associated with each dimension. For example, an implementation strategy could include the cost of training staff on the intervention delivery and the cost of implementing the intervention itself. The training costs are distinct from the ongoing operational costs for intervention implementation. In addition to these two costs, specific budget impact assessments (16) are included to provide practical information for implementation sites.

Of note, adaptation was not included in the original operational definitions provided by the IOF and RE-AIM framework. Recently, however, there have been suggestions to advance the consideration of adaptation within the context of implementation (17). To address this need we expanded the implementation dimension to include assessing indicators of adaptation prior to, during, and following implementation of the intervention. Initial indicators were based on suggestions from Stirman-Wiltsey et al. to document who, what, when, where, and why adaptations were made (18, 20). In addition, we included documentation on how the adaptation was consistent with the underlying evidence-based principles of the intervention as previously tested (20).

The final area of expansion of RE-AIM indicators was in the realm of combined metrics. Penetration was operationalized to include the product of reach, adoption, and organizational maintenance to provide an overarching system-based outcome. Other expanded combined metrics focused on determining the utility of an intervention at the participant, service provider, and organizational decision-maker level. In each case, utility was defined as the product of ratings of acceptability, appropriateness, and feasibility. Each of these metrics were further combined as an aggregate rating to produce a measure of systemic utility.

Discussion

This manuscript described the process used to compare, contrast and integrate dissemination and implementation science outcomes included in the RE-AIM framework and the IOF. We used a cross-tabular content analysis to compare between the frameworks which highlighted similarities and key differences. In addition, we integrated IOF within the context of the RE-AIM dimensions which generated an increased depth for a number of constructs and provided additional guidance on the possibility to examine combined metrics-particularly during later stages of scale-up activities. Based on this work we hypothesize that assessment of the expanded RE-AIM outcomes will improve the ability of dissemination and implementation scientists to document key outcomes that reflect the achievement of translating evidence-based interventions that promote public health.

The primary distinction between the two frameworks was an inclusion of individual level factors (RE-AIM) and predictors or antecedents of dissemination and implementation outcomes (IOF). The distinctions between these two models is not surprising when considering the rationale for the development of each (4, 5). The IOF was developed to better clarify dissemination and implementation outcomes for the specific field of dissemination and implementation research (4). In contrast, the RE-AIM framework was developed to be used across the translational spectrum of research and encourage some assessment of external validity in efficacy trials while also encouraging some assessment of internal validity in dissemination and implementation trials (21). The comparison between the frameworks allowed the consideration of variables that can be assessed at the individual, service-recipient level and those that can be assessed at an organizational and service provider level.

As Table 2 demonstrated, the primary overlap between the frameworks was within the organizational components of the RE-AIM framework. This highlighted a limitation of the IOF in the area of understanding a key dissemination outcome—reach. Reach, which can be considered an operationalization of consumer-demand for an evidence-based intervention, has been proposed as a key factor in organizational uptake and sustainability (22). While the explicit focus on reach may have been a limitation of the IOF, the focus on acceptability, appropriateness, and feasibility—albeit at the level of the service provider, organization, and organizational sub-systems—was a strength. We proposed that acceptability, appropriateness, and feasibility could be considered at multiple levels and at different temporal points across the translation research spectrum. First, these constructs would enhance the understanding of acceptability, appropriateness, and feasibility of service recipients. When applied to service recipients, the population that would have health benefits from the evidence-based intervention, understanding these variables can provide valuable information relative to the potential for an evidence-based approach to achieve high reach (23). By integrating these ideas within an expanded operationalization of RE-AIM indicators, it also provides additional planning and evaluation metrics that can heighten the likelihood of achieving broach reach when an intervention is taken to scale. Second, operationalizing these constructs temporally would entail the use of future and present tense language that could easily be applied to existing validated tools. For example, Weiner et al. (24) measures of intervention appropriateness and feasibility include temporal language appropriate for reach and adoption (e.g., this intervention seems doable) that could be adapted for prediction of ongoing implementation and organizational maintenance (e.g., this intervention is/was doable) reflecting experience in participation and delivery.

The newly expanded RE-AIM indicators has the potential to perform well due to its expanded definitions, in regards to assessment within staged research models such as the Pathways to Scale-Up Model (Pathways) (25), used primarily in Australia, to determine intervention readiness for broad application. “Pathways” describes four stages of scaling up evidence-based interventions: development, efficacy, effectiveness, and dissemination (25). As the RE-AIM framework was developed to be applicable across the translational research spectrum (21, 26), it has greater utility than the IOF for investigators using models such as “Pathways” —which can be applied to both evidence-based interventions as well as novel intervention approaches based on sound theory—and requires the assessment of service recipient outcomes (25). Similarly, the expanded RE-AIM metrics also may be ideal for hybrid effectiveness-implementation trials (27) that necessitate assessing effectiveness at the service recipient level and implementation at the service provider or organizational level. Contextual assessment is also a key component for hybrid type 1 trials that have a primary outcome of effectiveness. The assessment of context can include examining barriers and facilitators to future implementation efforts, potential for adoption and sustainability, and likelihood of high reach. The expanded RE-AIM metrics provide further contextual information related to acceptability, appropriateness, and feasibility that could advance understanding of how best to design implementation fit for the intended audience or service provider.

The assessment of cost was increased to move beyond cost of implementation and implementation strategies, cost effectiveness, and budget impact analysis to a more comprehensive assessment across RE-AIM dimensions. This aligns with the importance of a wide range of cost considerations used by policy makers and organizational leaders (16). The area of cost assessment and analysis in dissemination and implementation science is emerging (16, 19) and the expanded cost metrics provide a methodology for assessing costs related to reach, adoption, implementation, and organizational maintenance—with a focus on both the strategies used to enhance each outcome and the operational costs associated with each dimension. This will allow for the development of cost simulation models (28) that could vary dissemination and implementation strategy use and provide variable budget impact scenarios for systems considering the uptake of a new evidence-based intervention. For example, a new evidence-based diabetes prevention intervention, being introduced for community YMCAs could use dissemination and implementation strategies that include marketing strategies to increase adoption, participant incentives to increase reach, auditing and feedback processes to improve implementation, and a budget matrixing activity to improve likelihood of organizational maintenance. With the appropriate data on responsiveness of each RE-AIM outcome to the respective dissemination and implementation strategy, would allow a determination of the cost and impact with and without each strategy.

Adaptation was not explicitly defined in either the RE-AIM Framework or IOF, but is necessary to consider during the implementation of an intervention (6). Adaptations (i.e., changes to the intervention components, and delivery method) may elicit changes in the effectiveness of interventions (both positive and negative), and as such it is vital that these are noted and assessed when possible, serving as a useful insight into intervention components across the stages. Despite the potential for adaptations to alter effectiveness of interventions, there are several benefits that arise—such as addressing barriers to program adoption, implementation, and sustainment at the individual, service and organizational level (29). As noted by several authors, the key to determining the impact of adaptations is careful tracking and reporting of how, why, and by whom the adaptations were made and the resulting changes in individual and organizational outcomes (20, 30). It is of note that recent conceptual descriptions of adaptations related to the RE-AIM framework (31) highlighted the likelihood that adaptations are iterative and may be addressed across adoption, implementation, and sustainability—additional research in this area will help to determine at which points meaningful adaptation occurs.

The combined metrics proposed for the RE-AIM framework have not been broadly used across the extant public health or dissemination and implementation science literature. It is unclear whether this uptake is based on the lack of applicability of these metrics and/or the difficulty in gathering all the necessary data. Still, the original combined metrics provided an opportunity to consider a single number to assess individual and organizational impact (12). We proposed these metrics to allow for the scientific comparison of differential impact of various dissemination and implementation science strategies. Broader evaluation efforts that include attributable individual-level impact, penetration, and individual-level utility may help researchers and public health professionals better understand intervention reach and, if needed, adapt recruitment and retention efforts to improve individual-level engagement and sustained participation. This is also applicable to adoption at the service provider and system level. These combined metrics may provide additional, and potentially more practical, ways to assess utility at multiple levels and across time with relatively simple measures that can be proactively collected (24). Further, using the expanded RE-AIM outcomes may not only speed up the translation of evidence into practice, in an attempt to alleviate the stark difference that exists between research and policy timelines (32), but may also help researchers and policy makers to determine cost-impacts of interventions. For an intervention to be novel to policy makers, it needs to provide favorable outcomes at the individual and organizational level, aligned with their specific policy goals, as well as having cost benefit (33). The expanded RE-AIM indicators presented here moves beyond current approaches and provided additional planning and evaluations targets that can contribute to dissemination and implementation science and increase the translation of evidence into practice.

A potential limitation of our expanded RE-AIM approach is that, by including antecedents to dissemination and implementation outcomes we are initiating a shift from an outcome framework to a blended outcome framework and explanatory model (34). As such, the expanded outcomes we propose limit other factors that could provide explanation for specific reach, adoption, implementation, and organizational maintenance outcomes. For example, the Practical, Robust Implementation, and Sustainability Model (PRISM) evaluates the impact of a public health intervention on various domains of RE-AIM as they translate to real-world practice (35). The model considers organizational and patient perspectives of the intervention characteristics, drawing similarities to intervention beneficiary and organizational evaluations of acceptability, appropriateness, and feasibility—though they do not explicitly list these as potential constructs (35, 36). The expanded RE-AIM indicators presented here may simply set the stage to consider theoretically-compelling constructs that could be dissemination and implementation strategy targets to improve RE-AIM outcomes through theoretically derived mediators (37). Additionally, conducting concurrent validity testing on data collected on the indicators that are consistent between the RE-AIM and IOF frameworks would be valuable in a future study.

The expansion to RE-AIM indictors is intended to improve the planning and outcomes related to health-enhancing interventions. However, a potential unintended consequence of this paper is that it is counter to the intuitive nature of the RE-AIM framework (38). That is, by adding complexity to the breadth of RE-AIM indicators it could be a barrier to applying the framework. This paper highlights the similarities between RE-AIM and the IOF, pushes the boundaries of how best to consider dissemination and implementation outcomes, and provides opportunities for confirmation or rejection of the expanded RE-AIM indicators. It is hypothesized that the use of the expanded RE-AIM indicators across the dissemination and implementation research continuum may assist in speeding up the translation of evidence into practice—and advance the science surrounding that translation. Each of the proposed expansions should be examined, from a scientific and pragmatic perspective, to determine the salience of the indicators and metrics across research and practice stakeholder groups. Understanding the practicality, reliability, and validity of our approach will help to advance the planning and evaluation of future translational research studies focused on developing and testing evidence-based interventions.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

KR, SK, GP, and PE equally contributed to the conceptualization of this report, analyzing definitions, compiling cross-tabular comparisons, and manuscript writing. All authors contributed to the article and approved the submitted version.

Funding

PE time was supported in part by the National Institutes of Health (Great Plains IDeA CTR U54GM115458). GP time was supported in part by the American Heart Association (18PRE34060136).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Health Prof. (2013) 36:44–66. doi: 10.1177/0163278712446066

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. (2013) 45:237–43. doi: 10.1016/j.amepre.2013.03.010

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. (2019) 14:1. doi: 10.1186/s13012-018-0842-6

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. (2011) 38:65–76. doi: 10.1007/s10488-010-0319-7

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/AJPH.89.9.1322

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Glasgow RE, Harden SM, Gaglio B, Rabin B, Lee Smith M, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Lewis CC, Proctor EK, Brownson RC. Measurement issues in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. 2nd ed. New York, NY: Oxford University Press (2018). p. 15.

Google Scholar

8. Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. (2007) 28:413–33. doi: 10.1146/annurev.publhealth.28.021406.144145

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Estabrooks PA, Allen KC. Updating, employing, and adapting: a commentary on what does it mean to “employ” the RE-AIM model. Eval Health Prof. (2013) 36:67–72. doi: 10.1177/0163278712460546

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Almeida FA, Michaud TL, Wilson KE, Schwab RJ, Goessl C, Porter GC, et al. Preventing diabetes with digital health and coaching for translation and scalability (PREDICTS): a type 1 hybrid effectiveness-implementation trial protocol. Contemp Clin Trials. (2020) 88:105877. doi: 10.1016/j.cct.2019.105877

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Zoellner JM, Hedrick VE, You W, Chen Y, Davy BM, Porter KJ, et al. Effects of a behavioral and health literacy intervention to reduce sugar-sweetened beverages: a randomized-controlled trial. Int J Behav Nutr Phys Act. (2016) 13:38. doi: 10.1186/s12966-016-0362-1

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. (2006) 21:688–94. doi: 10.1093/her/cyl081

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. (2011) 38:4–23. doi: 10.1007/s10488-010-0327-7

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. (2015) 10:155. doi: 10.1186/s13012-015-0342-x

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Rabin BA, Brownson RC. Terminology in dissemination and implementation science. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. 2nd ed. New York, NY: Oxford University Press (2018). p. 19–45.

16. Eisman AB, Kilbourne AM, Dopp AR, Saldana L, Eisenberg D. Economic evaluation in implementation science: making the business case for implementation strategies. Psychiatry Res. (2020) 283:112433. doi: 10.1016/j.psychres.2019.06.008

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Rabin BA, McCreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front Public Health. (2018) 6:102. doi: 10.3389/fpubh.2018.00102

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. (2018) 15:E02. doi: 10.5888/pcd15.170271

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ward CE, Hall SV, Barnett PG, Jordan N, Duffy SA. Cost-effectiveness of a nurse-delivered, inpatient smoking cessation intervention. Transl Behav Med. (2019). doi: 10.1093/tbm/ibz101. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. (2019) 14:58. doi: 10.1186/s13012-019-0898-y

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Estabrooks PA, Gyurcsik NC. Evaluating the impact of behavioral interventions that target physical activity: issues of generalizability and public health. Psychol Sport Exerc. (2003) 4:41–55. doi: 10.1016/S1469-0292(02)00016-X

CrossRef Full Text | Google Scholar

22. Leeman J, Baquero B, Bender M, Choy-Brown M, Ko LK, Nilsen P, et al. Advancing the use of organization theory in implementation science. Prev Med. (2019) 129S:105832. doi: 10.1016/j.ypmed.2019.105832

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Lyon AR, Munson SA, Renn BN, Atkins DC, Pullmann MD, Friedman E, et al. Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: protocol for studies applying a framework to assess usability. JMIR Res Protoc. (2019) 8:e14990. doi: 10.2196/14990

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. (2017) 12:1–12. doi: 10.1186/s13012-017-0635-3

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Indig D, Lee K, Grunseit A, Milat A, Bauman A. Pathways for scaling up public health interventions. BMC Public Health. (2017) 18:68. doi: 10.1186/s12889-017-4572-5

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Dzewaltowski DA, Estabrooks PA, Glasgow RE. The future of physical activity behavior change research: what is needed to improve translation of research into health promotion practice? Exerc Sport Sci Rev. (2004) 32:57–63. doi: 10.1097/00003677-200404000-00004

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. (2012) 50:217–26. doi: 10.1097/MLR.0b013e3182408812

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Dopp AR, Mundey P, Beasley LO, Silovsky JF, Eisenberg D. Mixed-method approaches to strengthen economic evaluations in implementation research. Implement Sci. (2019) 14:2. doi: 10.1186/s13012-018-0850-6

PubMed Abstract | CrossRef Full Text | Google Scholar

29. McHugh RK, Murray HW, Barlow DH. Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: the promise of transdiagnostic interventions. Behav Res Ther. (2009) 47:946–53. doi: 10.1016/j.brat.2009.07.005

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Chambers DA, Norton WE. The adaptome: advancing the science of intervention adaptation. Am J Prev Med. (2016) 51(4 Suppl. 2):S124–31. doi: 10.1016/j.amepre.2016.05.011

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health. (2020) 8:134. doi: 10.3389/fpubh.2020.00134

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Glasgow RE, Brownson RC, Kessler RS. Thinking about health-related outcomes: what do we need evidence about? Clin Transl Sci. (2013) 6:286–91. doi: 10.1111/cts.12080

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Jones Rhodes WC, Ritzwoller DP, Glasgow RE. Stakeholder perspectives on costs and resource expenditures: tools for addressing economic issues most relevant to patients, providers, and clinics. Transl Behav Med. (2018) 8:675–82. doi: 10.1093/tbm/ibx003

PubMed Abstract | CrossRef Full Text

34. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. (2015) 10:53. doi: 10.1186/s13012-015-0242-0

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. (2008) 34:228–43. doi: 10.1016/S1553-7250(08)34030-6

PubMed Abstract | CrossRef Full Text | Google Scholar

36. McCreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the practical, robust implementation and sustainability model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. (2019) 9:1002–11. doi: 10.1093/tbm/ibz085

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the impact of implementation strategies in healthcare: a research agenda. Front Public Health. (2019) 7:3. doi: 10.3389/fpubh.2019.00003

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Kwan BM, McGinnes HL, Ory MG, Estabrooks PA, Waxmonsky JA, Glasgow RE. RE-AIM in the real world: use of the RE-AIM framework for program planning and evaluation in clinical and community settings. Front Public Health. (2019) 7:345. doi: 10.3389/fpubh.2019.00345

PubMed Abstract | CrossRef Full Text | Google Scholar