Role modelling in the training of hospital-based medical specialists: a validation study of the Role Model Apperception Tool (RoMAT)

Medical education largely takes place in clinical practice where clinical teachers play a pivotal role [1, 2]. Evidence supporting this emerged in the previous century, when the social learning theory was developed. This theory states that learning is a cognitive process in a social setting where behaviour can be learned through observation. Imitativeness, a component of behavioural learning, consists of an individual imitating a behaviour consciously depending on the response the behaviour evoked [3]. Albert Bandura demonstrated through a series of experiments the importance and effectiveness of modelling for attaining new behaviour [4, 5]. In medical education, workplace-based learning forms an important component of the training program and therefore should be used effectively [6]. Role modelling is a key component of this, which fits the framework of the social learning theory[7] and has been described in the literature as the process in which ‘faculty members demonstrate clinical skills, model and articulate expert thought processes and manifest positive professional characteristics’ [1]. The resident is influenced, positively or negatively, by the clinical teacher through his or her behaviour during daily practice [1, 7‐10]. An individual is a positive role model when he or she exhibits excellent teaching skills, clinical skills and personal factors such as compassion, integrity and honesty [2]. Role modelling is a dynamic process where the resident observes, judges and consciously and subconsciously decides if what is observed will be implemented into a personal style [10]. Role modelling exists in all phases of medical training and influences professional behaviour and identity and shapes career choices, indicating its power [11‐13]. Interestingly, clinical teachers and residents are not always aware of this process, risking transmission of negative behaviour [14‐16]. Awareness of being a role model improves role model behaviour [17]. For this reason, Jochemsen-van der Leeuw et al. identified characteristics of role model behaviour [11], and developed and subsequently validated the Role Model Apperception Tool (RoMAT) in the primary care training setting [18]. The purpose of this tool is to help residents assess positive and negative role modelling and serve as feedback to the clinical teacher on role model behaviour.

In the Netherlands, postgraduate medical education in primary care and secondary (hospital-based) care is arranged differently. Training takes 3 years in primary care and 5–6 years in secondary care. In primary care training, supervision in patient care work is usually done by a single general practitioner teacher and healthcare is provided locally in the community (2100 patients per full-time general practitioner). In the training of medical specialists in secondary care, supervision is more fragmented, with clinical teachers supervising multiple residents simultaneously, and residents being supervised by multiple clinical teachers. Moreover, healthcare is given on a larger scale in hospitals. These differences in the context of training may cause variation in the way role model behaviour is expressed by clinical teachers and perceived by residents in primary versus secondary care.

Currently no tool is used to assess the various components of role modelling as a specific entity in the training of medical specialists. Instruments to assess the quality of hospital-based clinical teachers are being used and contain some items on role modelling, but lack specificity to identify important characteristics of good or bad role models [19]. Furthermore we do not know whether observations on these characteristics differ per trainee. Workplace-based learning is interactive and based on observation and imitation[6] (as with role modelling), and behaviour, good or bad, is transmitted consciously and subconsciously [14‐16]. We believe that to improve role modelling[17] and to avoid transmission of negative behaviour, awareness of role modelling in the training of medical specialists must be enhanced. The intended use of the RoMAT for this new context is to help residents assess role model behaviour (good and bad), to serve as feedback to clinical teachers and to compare clinical teachers regarding role model behaviour.

Psychometric tool characteristics, such as validity, are not merely a reflection of the tool alone, but are also dependent of the context of application [20‐23]. We therefore aimed to validate the RoMAT and to investigate the possibility to compare clinical teachers regarding role model behaviour in the hospital-based training setting.

We validated the RoMAT in the training of medical specialists. The items of the tool were internally consistent and supporting evidence for construct validity was found. However, the ICC was low. Through PCA we extracted two components explaining 67% of the variation: ‘Caring Attitude’ and ‘Effectiveness’.

Compared with the primary study, one difference was the method of administration (paper-based versus web-based). Previous studies showed no differences in psychometric properties between paper-based and web-based administration of health measuring tools [38, 39]. We therefore believe that the method of administration did not influence our results. An advantage is that web-based surveys can reduce the risk of missing or ambiguous data by allowing one option to be chosen and prohibiting participants from proceeding when an item is not filled in (SurveyMonkey Inc., San Mateo, CA, USA).

In spite of the scree plot with a parallel analysis suggesting a one component structure, we extracted two components based on the underlying theory from the primary study. With eigenvalues greater than one, this was in accordance with the Kaiser criterion [40]. ‘Caring Attitude’ is defined as ‘the cluster of items reflecting relationship characteristics between clinical teachers to their patients, residents and others’, whereas ‘Effectiveness’ is defined as ‘the cluster of items relating to the ability of clinical teachers to provide their patients and residents with what they need’. Compared with the primary study, both components consisted of the same items, except for two [18]. Firstly item 7: ‘my clinical teacher demonstrates enthusiasm for his work’ fitted both components. We did not omit this item as it addresses a key aspect of role modelling [41]. Secondly, item 17: ‘my clinical teacher is professionally competent in difficult clinical situations and able to cope with adversity’, fitted the ‘Caring Attitude’ component in our study whereas it belonged to the ‘Effectiveness’ component in the primary study [18]. We found that this item did not seem to fit both components very well, as is illustrated by its low rotated factor loadings (Tab. 1). In the previous study low rotated factor loadings were also found [18]. This bad fit of item 7 and 17 can be explained by the differences between these items and both components. Both items reflect traits of the clinical teacher that are very personal in nature. On the other hand, both components are defined to reflect characteristics of the interaction between the clinical teacher and his/her personal environment. This discrepancy could explain our results. The good accordance between both studies regarding the underlying structure of the tool supports its construct validity and could imply that role model behaviour expressed by clinical teachers and perception of this by residents might be similar in primary care and secondary care training settings.

Internal consistency was high, reflected by a Cronbach’s alpha 0.94 and 0.93 for factor 1 and 2 respectively. In the literature it has been stated that values >0.9 may signal overlapping and thus redundant items [25], which may have been the case with the RoMAT. However, we have deliberately decided not to omit any items because of the need to address all specific aspects of role modelling reflected by all 17 items of the RoMAT.

We found a strong correlation between the single role model question and the RoMAT (Spearman r = 0.62), supporting evidence of construct validity. We found that residents with more prior experience gave lower scores (i.e. better role model behaviour) in the ‘Effectiveness’ component compared with residents with less experience. This is in accordance to previous literature[31, 33]. This correlation was not found in the primary study of the RoMAT[18]. Jochemsen-van der Leeuw studied residents in family medicine, where Côté et al.[42] found that these residents tend to choose role models with a more patient and colleague centred approach to care and focus less on clinical expertise compared with other specialties. This suggests that in hospital-based training characteristics of role models reflecting the ‘Effectiveness’ component are considered more important than characteristics from the ‘Caring Attitude’ component, and may explain this discrepancy. In contrast to the primary study[18] we found that younger year residents (years 1 to 3) gave higher scores (i.e. worse role model behaviour) on the items on the ‘Caring Attitude’ component compared with older year residents (years 4 to 6), although differences were small. This discrepancy might be caused by two factors. Firstly, although hospital-based clinical teachers nowadays become a clinical teacher voluntarily and receive more and better education regarding training, clinical teachers in general practice still undergo more comprehensive and frequent training [43]. Secondly, in hospital a multitude of residents and clinical teachers work at the same time, which makes interaction more fragmentary and less personal compared with training in general practice where supervision is done in a one-on-one fashion most of the time. In our study, this could have caused younger year residents to miss personal attention resulting in higher scores on the ‘Caring Attitude’ component. Moreover, as residents gain more experience this need for attention may be reduced and their focus shifts from personal to professional qualities, causing the ‘Caring Attitude’ scores to be lower for older year residents as they are less affected by fragmentary and less personal supervision. In accordance with previous studies we found no differences between males and females [18, 32].

In the current study, the majority of the scores were in the top three score categories (Fig. 1). Several factors may have caused this. Research has shown that in studies investigating questionnaires or self-reported measures, respondents may give socially desirable answers. Usually this phenomenon is present when participants are asked about sensitive subjects [44]. Moreover, in ‘Agree/Disagree’ formulated questions, respondents may have the tendency to answer Agree regardless of their opinion. This phenomenon is called acquiescence bias and is of uncertain origin [45]. On the other hand, the high and relatively homogeneous scores may well reflect the overall satisfaction of residents with the role model behaviour of their clinical teacher, possibly due to the strong focus on ongoing faculty development and clinical teaching skills of clinical teachers in the Netherlands, which is monitored during 5‑yearly recertification procedures of each training program [46]. In clinimetrics, it has been shown that sufficient variation in responses is necessary to attain high ICC values [47, 48]. We therefore believe that the poor IRR with low ICC are possibly a result of this homogeneity in responses and reflect our study population instead of tool characteristics. Nevertheless, this implies that the RoMAT might not be suitable to compare clinical teachers based on their scores.

Resident training is largely workplace-based[6] and learning occurs through observation and imitation[3] and subsequently through role modelling [10]. Literature has shown that students might benefit from early awareness of role model behaviour and clinical teachers have stated that they rarely receive feedback on the impact of their role modelling and lack awareness [7]. Awareness of role modelling improves role model behaviour [17]. This is where the RoMAT is of great use and fills in the current educational gap: it helps residents gain insight into their needs from their clinical teacher as a role model and serves as a source of feedback to clinical teachers.