A retrospective assessment of the KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR)

Patient-Reported Outcome Measures (PROMs) are standardized, validated questionnaires that are completed by patients, such as a person’s perspective on their health, well-being, or symptoms [1, 2]. PROMs can be used for several purposes: at group level to study differences between disease populations, to describe the effects of treatment in clinical trials, and to assess quality of care or on an individual level to promote patient-centered care, guide clinical decision-making, and to facilitate communication [3]. There is widespread evidence for the effects of PROM applications on an individual level regarding an increase in Health-related Quality of Life (HRQOL) scores, satisfaction with care and communication about PROs in research settings, both in adult [4‐6] and pediatric [7‐12] samples. Yet the implementation of these evidence-based (EB) PROMs interventions is challenging.

The KLIK PROM portal (www.​hetklikt.​nu and www.​klik-uk.​org) is an example of an EB PROM intervention for patients (children or adults) who regularly visit the outpatient clinic [13]. Patients complete PROMs online, prior to their visit. Answers are transformed into an electronic PROfile (ePROfile; Fig. 1). Clinicians discuss this ePROfile with patients, to monitor well-being over time, identify problems, and provide tailored advice and interventions. The effects of using the KLIK PROM portal have been demonstrated in pediatric oncology [7] and in pediatric rheumatology [12], by showing an increased and more detailed discussion of HRQOL and psychosocial functioning during the consultation, less undetected problems, and a higher clinician-reported satisfaction with provided care, without lengthening the consultation duration.

Despite the availability of several EB PROM interventions across the world, the actual implementation of PROM interventions in clinical practice remains limited [14‐17]. There is a critical gap in behavioral medicine between what we know can optimize patient health and care outcomes and what gets implemented in everyday practice [1]. If EB PROM interventions are not successfully implemented in clinical practice, then intended effects are not reached, which limits the impact on patients’ health outcomes [18, 19]. Traditional randomized controlled trials study the effectiveness of PROM interventions under ideal circumstances. Yet for the implementation of PROMs in clinical practice, a different, more flexible approach is needed. Often, a “voltage drop” (a dramatic decrease in effectiveness) is seen once interventions get implemented in clinical practice [20]. Implementation research is defined by the National Institute of Health as the “scientific study of the use of strategies to adopt and integrate evidence-based health interventions into clinical and community settings in order to improve patient outcomes and benefit population health” [21]. Therefore, a scientific approach to the change process is crucial. In order to know what drives successful implementation of PROMs in clinical practice, we need to study the mechanisms that influence implementation outcomes [17, 22, 23]. Implementation science models, theories, or frameworks support in identifying factors that influence an implementation process or outcome.

In general, three overarching aims of theoretical approaches and five categories of theories, models, and frameworks used in implementation science can be distinguished [24]: (1) guiding the process of translating research into practice (process models), (2) understanding and/or explaining what influences implementation outcomes (determinant frameworks, classic theories, and implementation theories), and (3) evaluating implementation (evaluation frameworks). Specifically, determinant frameworks are useful in understanding or explaining what influences implementation outcomes and to support the design of implementation strategies or maximizing the use of enablers to implementation [24].

A widely cited and comprehensive determinant framework in the implementation science literature in health is the Consolidated Framework for Implementation Research (CFIR). Damschroder et al. [22] aimed to develop a framework that comprises common constructs from published implementation theories and includes, therefore, missing key constructs in other theories. It contains 39 constructs which are organized in five general domains: (1) intervention characteristics (e.g., evidence, complexity, adaptability, costs), (2) outer setting (e.g., peer pressure and external policies), (3) inner setting (e.g., structural characteristics, implementation climate, and culture), (4) characteristics of individuals (e.g., knowledge about the intervention and self-efficacy), and (5) implementation process (e.g., planning, engaging stakeholders, champions, and execution), see Fig. 3. Determinant frameworks, such as CFIR, are specifically useful in understanding or explaining what influences implementation outcomes and to support the design of implementation strategies or maximizing the use of enablers to implementation [24]. This paper aimed to (A) retrospectively describe the most prominent determinants and reasons of successful KLIK PROM implementation using CFIR and (B) use the CFIR-ERIC Implementation Strategy Matching tool to identify current barriers of the KLIK PROM portal implementation and match implementation strategies that address the identified barriers. In our specific study context, the CFIR framework seemed particularly useful as it covers a wide range of implementation constructs and domains and it allowed us to use a standardized framework to explain the influence of each domain on the implementation outcomes of an evidence-based PROM portal. With years of experience in the development and implementation, the KLIK PROM portal is now in a phase of understanding what barriers and facilitators have already been resolved and determining what major determinants are currently of influence to move to the next area of implementation: sustainability.

This paper aimed to retrospectively describe the most prominent determinants of successful KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR) and to identify current barriers and matching implementation strategies for the KLIK implementation using the CFIR-ERIC Implementation Strategy Matching Tool.

This retrospective evaluation shows that the strength of the KLIK PROM portal implementation lies particularly in the following CFIR domains: intervention characteristics (e.g., easy to use, direct feedback), characteristics of individuals (e.g., motivated clinicians), and process of implementation (e.g., support of the KLIK expert team). In addition, the climate of the outer setting is changing and patient-reported outcomes are more valued, which facilitates the implementation of the KLIK PROM portal. On the other hand, barriers in the implementation lie mainly in the domain of the inner setting and the intervention characteristics. Regarding the inner setting, involving and motivating all stakeholders at various levels of the multidisciplinary teams and hospitals is challenging. Regarding the intervention characteristics, mainly the tension field of providing optimal support of the KLIK expert team and the use of the KLIK PROM portal on the one hand and keeping low costs on the other hand is difficult. These findings are in line with another study discussing PROM implementation [38], where the authors describe the same relevant CFIR domains. This implies that the CFIR domain ‘outer setting’ might be less relevant than the other four domains when describing PROM implementation. However, a recent study on PREM implementation did find relevant outcomes regarding the outer setting, or macro level [39], and other literature on PROMs in palliative care also conclude that all CFIR domains need consideration for effective implementation [40].

Most CFIR domains were applicable to implementation of the KLIK PROM portal, showing that CFIR can be used in the context of implementing PROMs. However, the framework is not specifically developed for this context, resulting in insufficient attention for specific parts of the PROM implementation. For example, the content, length, and psychometric properties of PROMs are important factors for successful implementation of PROMs in clinical practice and are not addressed by the CFIR framework.

The CFIR is a comprehensive framework based on various published implementation theories [22], resulting in a very extensive framework consisting of many constructs, which can make it complicated to use. The five domains of the framework are intertwined and interacting, making it hard to determine where points of attention can be placed without iteration. In particular, the domain inner setting consists of many overlapping subdomains with intangible concepts. In addition, a recent systematic review on implementing e-health interventions shows blind spots in current literature about contextual factors (such as the organization), which makes it difficult for clinicians and researchers to understand these concepts and to translate it to clinical practice [41]. In previous literature, other weaknesses of CFIR are mentioned. In their systematic review on PROM implementation, Foster et al. identified the importance of different stages of the implementation process, which is not captured by CFIR [1].

The CFIR can be described as a determinant framework [24]. Determinant frameworks specify which factors (determinants) have a facilitating or inhibiting effect on the implementation. These frameworks thus describe the influence of processes on the implementation outcomes, but do not address these implementation outcomes, in contrary to evaluation frameworks. Therefore, it would be useful to use the CFIR in combination with another type of model. For instance, a widely used model on implementation outcomes is the “conceptual model of implementation research”, as described by Proctor and colleagues [18]. In order to improve outcomes for patients, it is important to be able to determine which determinants relate to which specific implementation outcomes. Only then can be reliably concluded which specific strategies work for which implementation outcomes.

The CFIR-ERIC Implementation Strategy Matching tool provided implementation strategies for the identified CFIR barriers [30]. Some of the suggested implementation strategies can be explored and used in the KLIK PROM portal implementation in the upcoming years. For example, assess key stakeholders for readiness is an ongoing process and still a challenge. By conducting individual interviews with the more reluctant clinicians underlying resistance can be better understand and addressed. In addition, identifying expected barriers and facilitators in the implementation process by actively discussing these topics in multidisciplinary team meetings in a more structured way is necessary. Also, incentives for patients in using the KLIK PROM portal could be explored further by increasing patient engagement.

However, not all suggested strategies by the matching tool provided new insights as they were directly linked to the perceived barrier (e.g., identify and prepare champions for the barrier ‘champions’ and access new funding for the barrier ‘cost’) and therefore were already known by the KLIK expert team. In addition, some strategies are currently being worked on (e.g., tailoring strategies, inform local opinion leaders, and identify barriers in the implementation process). Though, these strategies are difficult to implement and the tool underlines the need to pay more attention to these important strategies.

To further improve the KLIK implementation process in daily clinical practice, both the identified current barriers as well as the strategies extracted from the CFIR-ERIC tool can be used, to provide some examples:

At the time the implementation of the KLIK PROM portal in clinical practice started, a variety of implementation frameworks (including CFIR) and instruments to monitor and evaluate the implementation process from the start were not yet available. Just as we have evolved as a group, implementation science has evolved over the past decade as well. Implementation of the KLIK PROM portal was therefore essentially a process of “learning by doing”. Each time a specific multidisciplinary team showed interest in using KLIK, novel challenges appeared. As a result, a wide range of implementation strategies were used to tackle these particular issues. Notably, without realizing it at the time, many of the principles and strategies that are outlined in the CFIR tool were applied.

We recommend groups starting to implement PROMs in their setting to use an implementation science framework, like the CFIR, as knowing which factors need to be taken into account can lead to a more successful implementation in a specific context. The CFIR authors have developed an Interview Guide Tool (https://​cfirguide.​org/​tools/​) that can help researchers to question constructs of the CFIR that apply for the specific context. As every individual implementation process is different, also the constructs that are applicable differ.

Strengths of this study include the broad view of the retrospective description; multiple populations and multicenter experiences have been taken into account. In addition, the description is based on long-term experience and on published literature. However, this paper has several limitations. First, although a deliberate choice, no standardized qualitative research methods were used in this paper as the aim of this paper was to give a retrospective description of the KLIK PROM implementation process using the CFIR framework with the overarching purpose to create more awareness for the use of implementation science in PROM research. Second, the determinants and barriers for successful KLIK PROM implementation were described based on the experiences of the KLIK expert team (existing of members from two different centers) and this could have led to a selective view from the KLIK expert team. However, the KLIK expert team works closely with a variety of stakeholders on a day-to-day basis, including clinicians, patients, and parents. They furthermore provide opportunities for stakeholders to provide feedback during regular evaluation meetings. In addition, recently two evaluation studies were carried out to gain more insight into the perspectives of clinicians [35], and pediatric patients and parents [44]. Thus, even though other stakeholders were not literally represented as co-authors, it can be assumed that their opinions are represented throughout this study.

In conclusion, this retrospective approach showed that the CFIR provides clinicians and scientists guidance during a healthcare implementation process and can be used in all phases of implementation, although it is a quite extensive and complex framework with some overlapping constructs. For example, the CFIR can be used retrospectively, reflected in this article, to describe the implementation process and its determinants and to identify remaining barriers. An advantage of using this theoretical framework prior to start of implementation is that clinicians become aware of the possible facilitating determinants and barriers for implementation. Using an implementation science framework, like the CFIR, is recommended for groups starting to use PROMs in clinical care as knowing which factors need to be taken into account can lead to a more successful implementation in a specific context.