The application of the tracer method with peer observation and formative feedback for professional development in clinical practice: a scoping review

The tracer method was introduced in 1973 as a tool to assess the quality of care provided in a healthcare system [1]. Over the years, the tracer method has gained increasing worldwide popularity as an evaluation and assessment method to document a patient’s experience, a healthcare process or product, resulting in both summative and formative feedback for accreditation of health services [2‐5]. Carrying out this form of auditing is associated with improved patient experiences and observed safety on hospital wards, with no adverse outcomes on safety culture and team climate [6]. It has been proposed that the tracer method could also be a useful strategy to support continuous professional development of healthcare professionals in clinical practice [7, 8]. During a tracer, a professional quality assessor or peer assessor (colleague of equal status) observes a healthcare professional during daily practice and provides feedback on a number of pre-established indicators [9]. Using the method in this way can be viewed as a form of peer observation and formative feedback. A systematic review provides theoretical support for the use of audit and feedback in professional practice and healthcare, by showing that feedback is more likely to be effective if the feedback provider is a supervisor or a colleague, and when it is provided more than once, delivered in verbal and written formats, and when it includes explicit targets and an action plan [10]. However, a thorough overview of the evidence of the effects of the tracer method with peer observation and merely formative feedback, carried out by a colleague of equal status, primarily focused on continuous professional development purposes in healthcare, is lacking. Recent reviews of the tracer method in healthcare quality management found little evidence of using the tracer method outside the scope of accreditation of health service organizations [8, 9]. Thus, we conducted this scoping review based on the following research question: How, by whom, and with what effect is the tracer method with peer observation and formative feedback applied as an instrument for professional development of healthcare professionals?

We conducted a scoping review according to the guidance of Arksey and O’Malley, Levac, Grant, and the JBI Guide for scoping reviews [11‐14]. The first and second author conducted all the steps of the review, the others critically appraised the research process and provided feedback. Following the methodological framework of conducting a scoping review, every next step was only taken after achieving consensus by the whole research team. We report our findings in accordance with the PRISMA Extension for Scoping Reviews (PRISMA-ScR) [15].

During the first stage, the key aspects of the study objective were translated into the research question as described. The research question guided the search strategy and the following steps of the review.

In January 2020, with an update in January 2021, we searched four databases (CINAHL, Cochrane, Embase and MEDLINE) from the inception, in collaboration with a university librarian, using the search terms along with their most important synonyms and alternative definitions, as generated from the research question (See Appendix 1 of the Electronic Supplementary Material for the full search strategies).

We searched Google Scholar for grey literature in February 2020 limiting our inclusion to the first 100 hits, because the relevance of retrieved studies declined sharply afterwards. In addition, we asked JCI, Qmentum (the two global market leaders on identifying, measuring and sharing best practices in quality and patient safety in hospitals) and an expert in the field (P. Lalleman, the Netherlands) what they considered to be the most important literature on the topic.

Articles were screened on relevance by title and abstract by two independent reviewers (RS and MM), using prespecified inclusion and exclusion criteria. As peer observation programs globally do not always refer to the tracer method, we transformed our comprehensive set of relevant terms into an extensive set of criteria. Only studies with full text available were included if (a) healthcare professionals, peers or colleagues were the subject of the study; (b) when shadowing, tracing, direct observation, feedback peer review and/or peer evaluation was used, and (c) where the purpose was development of competencies, performance and/or quality improvement. Studies with an English abstract and the main text in German, French or Spanish were also included using translation software. Studies were excluded if they took place in an educational setting, if simulation, masked/secret observation and/or video observation was used, where there was a dependent or hierarchical relationship between observer/feedback provider and the observed healthcare professional/feedback recipient, and where the purpose was an audit, certification, assessment and/or examination.

These criteria were chosen to ensure inclusion of studies with a formative purpose (i.e., aiming at continuous professional development or performance improvement) by healthcare professionals. Studies on the use of the tracer method for accreditation purposes or for summative assessment by a supervisor in a hierarchical relation were excluded. Differences in opinion of the two reviewers were resolved by discussion, and when needed consensus was obtained by consulting a third research team member. Then, the full text of the selected articles was screened against the inclusion and exclusion criteria independently by the first and second author (RS and MM), and the reasons for excluding articles were recorded. Covidence software was used to support the study selection process [16].

From the selected articles, the following predefined data were extracted by two reviewers (RS and MM): author(s), year of publication, study location, population, the objective of the study, type of intervention, methodology, results, impact on professional development, perceived barriers and facilitators, and conclusion(s). We did not appraise the quality of the evidence of selected articles, because a scoping review provides a preliminary assessment of the potential size and scope of available research and it aims to identify the nature and extent of research evidence [13, 14].

The extracted data were analyzed based on the research questions, leading to collating, summarizing, and reporting of results. Data from qualitative or mixed methods studies were subjected to thematic analysis, following guidelines from the literature [17, 18], by line-by-line coding by two reviewers (RS and MM) to identify themes. These were included in the results when consensus was reached between reviewers. If needed, a third research team member was consulted. Types of evidence were classified according to Kirkpatrick’s model, a useful method for evaluating training outcomes, consisting of four successive levels of learning effects: reactions, learning, behavior, and results [19].

We invited all first authors of the included studies to join this panel, in order to discuss and validate our findings and to identify any missing information on the topic. After accepting the invitation, participants in the panel were sent the draft version of the review in preparation, together with a zoom link for the set date and time of the panel meeting. Three authors (PvdW, MM, RS) prepared the panel meeting by drawing up an agenda and a topic list. The meeting was chaired by PvdW and introduced on content by RS. The audio recording of the meeting was transcribed and thematically analyzed by the two first authors (RS and MM). Conclusions thus drawn were sent to participants for a member-check. Convening and consulting the expert panel was exempt from medical ethical review under Dutch law. All panel members participated voluntarily.

We identified 2300 potential studies of which we included eight ([20‐27]; see the Fig. 1 and Table 1 of the Electronic Supplementary Material). All included studies used the tracer method in the form of peer observation and feedback. Three studies used a standardized feedback instrument [20, 21, 24]. Seven of the eight included studies took place in a hospital [20, 22‐27], and one in general practice [21]. Overall, the study population of included studies comprised 228 pediatricians (mainly from one study in which 198 pediatricians participated), five other medical specialists, 16 nursing managers, three specialist nurses and two general practitioners. In these studies, the tracer method was used to assess its use in clinical practice (feasibility) and to assess its impact on professional development (effectiveness). The four feasibility studies described healthcare professionals’ experiences with the tracer method as a form of peer observation and feedback [22, 23, 26, 27].

Five studies applied qualitative research methods (interviews or surveys with open-ended questions) [20, 21, 23, 25, 26], two used mixed methods (questionnaires with open and closed questions and the use of the I‑PASS (Illness—Patient—Action—Situation—Synthesis) mnemonic) [22, 24], and one study used quantitative data (survey with questions scored on a Likert scale) [27].

The effectiveness studies aimed to evaluate the change in professional attitude or behavior (n = 3) [21, 23, 26] or to investigate whether the instrument used impacted learning and development (n = 2) [21, 26]. Secondary study aims included investigating the efficacy of the method, characterizing the practice of peer observation and feedback, and identifying preferences for the use of the method.

All the studies but one reported positive perceived effects: participants described the tracer method as being valuable (n = 6) [20, 22‐24, 26, 27] and worthy of being repeated (n = 2) [22, 24], as an innovative, interesting and effective training supporting professionals’ ongoing learning (n = 2) [21, 22], as a tool to strengthen the work culture/collectivity (n = 2) [22, 26], as a promoter of growth through collaboration (n = 2) [23, 24] and as an instrument to stimulate an investigative attitude among professionals (n = 2) [23, 26]. One study concluded that the application of the tracer method did not lead to strong learning effects [25].

One study, covering all four levels of Kirkpatrick’s model, concluded that direct peer observations with feedback strengthened the workplace culture, promoted growth through collaboration, and allowed acceptance and success of future projects involving peer observations and feedback [24]. Two studies covered three levels from reactions to behavior [21, 26], most studies examined only the first two levels of reactions and learning [20, 22, 23, 25], and one study covered only the level of reactions [27].

This scoping review shows that the application of the tracer method with peer observation and formative feedback for continuous professional development has been studied mainly in hospital settings to assess its feasibility and impact. In all included studies—five qualitative studies, two mixed methods studies, and one quantitative study—the researchers used the peer observation and formative feedback, by medical specialists and general practitioners, and by nurses and nursing managers. The application of the tracer method addressed all four levels of Kirkpatrick’s model (reactions, learning, behavior, results) in only one study. Participants valued applying the tracer method and found it useful for their professional development.

We propose further research should focus on the design and conduct of more extensive, and rigorous studies on the evaluations of the tracer method in continuous professional development in healthcare, especially if the observed facilitators and barriers are sufficiently considered. A good starting point would be to generate more complex evaluation designs resulting in quantitative and qualitative data on the method to gather more robust evidence of its effects. It is conceivable to undertake this research not only in clinical practice but also, for example, in education of healthcare professionals so that already at this stage the basic principles of continuous learning are taught, for which the tracer method with peer observation and formative feedback can be an important basis. Therefore, we argue for tailoring the design and implementation of the instrument to the specific context of healthcare professionals or students. Because direct observation and formative feedback are familiar to most healthcare professionals and students, and the term ‘tracer method’ has a growing reputation through the use of globally applied quality systems such as JCI and Qmentum, existing knowledge and experience in this field could be applied to use the tracer method as a quality improvement instrument for professional performance as well.

In comparison to the literature, this study demonstrates that only a few studies have examined the tracer method as a tool for direct peer observation and formative feedback, applied in a non-dependent and non-hierarchical relationship and for professional development purposes. It has been shown that direct observation of health professional trainees is valid and representative in assessing a broad spectrum of skills and competencies [28]. However, the literature on such direct observation is potentially influenced by the fact that it is unclear whether the direct observation is intended as assessment (summative assessment) or as a source for formative feedback. A growing body of research suggests that this distinction is crucially important [29]. Particularly also the question of whether the learner (observed) perceives the direct observation and feedback as an exam or as an opportunity to learn and grow. Therefore, we deliberately limited ourselves to studies that described that they were aimed at promoting growth and development (formative) and excluded studies that made summative judgments.

In most studies, the tracer method has been used in the context of quality assessment and as an accreditation tool for healthcare organizations [20, 21, 23‐27]. Other forms of peer observation and feedback, for instance through indirect observation via video, have been used to improve the quality of healthcare, for example in improving hand hygiene and medical administration [30‐34]. Our results agree with these findings, and with those of studies on the feasibility of peer observation and feedback in clinical practice [35, 36]. Our observation that participants consider the application of the method to be valuable has also been confirmed in two studies [37, 38].

Although Cheung et al. did not mention the tracer method explicitly in their study, they identified key barriers and enablers to direct observation and feedback in clinical practice and proposed that discordant intentions between observers and observed, together with social expectations that the observed should be responsible for ensuring that observations occur, may lie at the root of why direct observation and feedback tend to occur so infrequently in practice [39]. Veloski et al. stated that the effects of formal (summative or formative) assessment and feedback on physician performance are positively influenced by the reliability of the source and duration of the feedback [40]. These barriers and facilitators correspond to our findings. Both the evidence and the experience in the field of graduate and postgraduate medical education highlight the need to distinguish feedback being used to guide learners towards growth and development (formative feedback), from assessment that is being used for summative judgement [29]. The results of our thematic analysis suggest that the same applies to using the tracer method as a tool to promote development of professional competencies.

Our study has several limitations. Although we applied a deliberately sensitive search strategy and consulted experts in the field, a scoping review may always miss relevant studies. Where scoping reviews tend to be used to map more extensive bodies of literature to find gaps in existing knowledge, we found only eight studies that met our inclusion criteria. Thus, our conclusions regarding the scope of our review should be interpreted with caution. Following the recommendations of the methodological framework for scoping reviews, we refrained from methodological appraisal of the included studies, because a scoping review provides a preliminary assessment of the potential size and scope of available research and it aims to identify the nature and extent of research evidence, so that the validity of the retrieved evidence was not formally assessed [13, 14].

Application of the tracer method with peer observation and feedback holds promise as a tool to promote professional development of healthcare professionals because participants value the method to stimulate their learning. Although the evidence is scarce and robust quantitative data are lacking, particularly on the effect of the method on healthcare professionals behavior, the use of the tracer method as a professional development tool by healthcare professionals of equal status tentatively indicates a potential usefulness of the tracer method as a quality improvement instrument.

Because the body of evidence is small and largely limited to the hospital setting, the scope for further research in the early stages of this field should be on the design and conduct of further, more extensive, and rigorous studies on the evaluations of the tracer method in continuous professional development in healthcare, especially if the observed facilitators and barriers are sufficiently considered.