Factors influencing students’ receptivity to formative feedback emerging from different assessment cultures

We purposively selected three medical schools known to have different assessment and feedback systems. When considering which schools to include, we considered published reports as to whether the overall assessment programme was intended to promote ‘assessment for learning’ or ‘assessment of learning’. If the programme was predominantly one of ‘assessment of learning’, we nevertheless wanted an institution that explicitly provided feedback to students as this was a fundamental aspect of our research question. Cleveland Clinic Lerner College of Medicine, USA (School A) has a programmatic approach to assessment with narrative feedback. Keele University School of Medicine, UK (School B) has a mix of formative and summative assessments with both numerical and narrative feedback. The Physician-Clinical Investigator Programme at Maastricht University, Netherlands (School C) has a programmatic approach to assessment with both numerical and narrative feedback. More details of each school’s approach are listed in Table 1 and have been published elsewhere [10, 20, 25‐29].

We chose focus groups in order to seek a range of views and to enable expressed ideas to be developed through interaction between participants. We conducted six focus groups in April to June 2014 (two at each school) to gather the perceptions of the student cohort on their feedback at different points in their studies. A faculty member from the local school, unconnected with the research project, recruited students by email. A convenience sampling approach was taken based on the students’ availability at pre-determined times. Incentives to participate were not offered. To encourage discussion of potentially challenging areas, each group deliberately consisted of a small number of students from a single year. More details of the groups are shown in Table 2. The order in which the groups took place was determined by timetabling constraints: B1, A1, A2, C1, C2, B2. Logistical issues prevented the organization of further groups in order to attempt to reach data saturation on each site. All students from Schools A and B spoke English as a first language; students at School C spoke English as a second language, but all were fluent and had no language difficulties.

The focus groups took place in the respective institutions with a single facilitator (CH) and lasted 60–120 min. A semi-structured approach was underpinned by open-ended questions designed to elicit students’ perceptions of their institution’s assessment system, their interpretation and use of feedback and any effect of grades on their aspiration to excellence. Questioning evolved according to the participants’ responses. We recorded and transcribed discussions maintaining student anonymity.

We used a constructivist grounded theory approach. This approach to qualitative data analysis encompasses the notion that interpretation of the data is co-constructed by both researchers and participants [30]. By studying the experiences and perspectives of the participants described in the transcripts, we aimed to identify thematic categories of factors that aid or hinder the uptake of feedback. Consistent with grounded theory, analysis occurred alongside data collection and was able to inform the questioning in the later groups. The lead author (CH) coded all transcripts in order to organize the data and identify key themes and concepts. A second researcher (VW) separately coded three transcripts (one from each school). There was close agreement; minor discrepancies were discussed and could be quickly resolved. At regular intervals, the research team conducted Skype teleconferences to refine the conceptual analysis. As our analysis framework assumes that data are co-constructed by interactions between researchers and participants, we provide the following contextual information: CH, LS, VW are medical doctors with a major involvement in medical education research and development; CvdV, KK have backgrounds in psychology and ED in sociology with a major involvement in medical education research and development.

Factors within the assessment systems across all schools promoted, or hindered, students’ personal agency, their capacity to act and make choices using their personal abilities within the constraints and possibilities of their context. Agency was promoted by providing choice, either within compulsory assessments or by providing optional assessments, which enabled students to demonstrate knowledge acquisition to themselves and their tutors. In contrast, compulsory multiple-choice question papers reduced personal agency by preventing students from fully demonstrating their knowledge; the questions were perceived to focus on specific ‘random’ facts.

I’m often frustrated with the questions because I think, ‘Well, I’m actually good in this field but this one particular question, I don’t know.’ (School C, Focus Group (FG)1)

Students discounted much of the feedback as irrelevant for future learning or assessments. Feedback methods enabling students to act independently were more conducive to supporting personal agency than those restricting autonomy. For example, online questions and feedback, combined with the ability to revisit these at any time, were preferred to didactic tutor-led feedback sessions delivered in a group setting without provision to revisit later. Providing significant autonomy within the assessment and feedback systems fostered an aspiration towards excellence:

It [the School] really lets you….truly reach and seek that excellence because it gives you that time that you can invest in whatever you deem to be important. (School A, FG1)

By contrast, an assessment system dominated by high-stakes assessments discouraged aspiration to improve:

I think sometimes exams can be limiting because you say, ‘Oh, I don’t need to know that for an exam.’ ……that stifles learning sometimes. (School B, FG1)

Students felt in control if feedback from low-stakes assessments appeared predictive of future performance in high-stakes assessments. The feeling was the opposite if low-stakes assessments were seen as too dissimilar from the ‘real thing’.

Institutional requirements for assessments (e. g. formatting rules for completing a portfolio) limited agency and provoked much frustration and risked devaluing learning. Students sometimes tried to subvert the marking criteria to maximize learning, demonstrating tension between student agency and institutional control.

Instead of looking at what they wanted I just went on a tangent and wrote. And then that for me I was like, I know I might not get a good mark. I might not even pass, but this is actually reflective. That’s going to be useful for me in the future to look back at. (School B, FG1)

Similarly, agency was threatened when institutions set criteria for selecting specific items from overall aggregated feedback to demonstrate progress. Students perceived being forced to include critical feedback, at the expense of detailed, personal and positive feedback, even if it was unhelpful, bland and generic. They felt forced to play the ‘assessment game’.

I had feedback, ‘You’re not saying enough.’ But on the other hand I got feedback, ‘But the quality of what you’re saying is very high.’ But I was always forced to make some goals from, ‘You’re not saying enough,’ so I was forced to say more. (School C, FG1)

Students recognized that autonomy should have limits. The safety of future patients necessitated that assessment processes identify students’ weaker areas or flaws in self-assessment. Standardization of assessments was generally accepted, even though it limited autonomy, but acceptance was much lower if it was perceived to lack immediate relevance to their future role as doctors.

If we’re going to do a critical appraisal of qualitative research, why not get people to find their own piece? So if you were doing paediatrics at the time, you could have found a paediatric paper that was relevant rather than writing about bottled water because it’s not got any clinical relevance to what we’re doing. (School B, FG1)

Grades and comparative student rankings within the cohort had mixed effects on receptivity to feedback. Grades reassured students they were achieving the standards required for qualification:

I think it’s like comparing my grades to what they think is necessary. So the necessary level of knowledge to be a doctor, is my knowledge lower than that? Is it higher than that? Is it about right? (School B, FG1)

Grades provided some clarity on the expected level of performance and avoided the potential for narrative feedback to mislead:

Well, I mean, you can get a warm and fuzzy description and be four out of ten. If you get a four then you know you need to do something to get to the six that is required to make the grade. I would much rather know that I need to work harder than have a warm and fuzzy feeling. (School B, FG2)

Information comparing performance with peers was an additional stimulus to take feedback seriously:

If I got a ranking [position relative to peers] of 50 on my MCQs and a ranking of 20 on my KFPs [Key Feature Problems], then I know I need to go and work on my MCQs. (School B, FG1)

Grades following summative, end-of-year assessments gave sufficient information; more detailed feedback was seen as superfluous:

So what was on our Year 2 paper – most of it is irrelevant to what’s going to be on our third-year paper. So to give us feedback is pointless. We’ve got a grade. That’s all we need to know. (School B, FG1)

At other times within a programme, grades without narrative feedback failed to provide sufficient information or motivation to stimulate improvement. At one school, so many students were graded ‘at the level expected’ that it was felt futile to aspire towards excellence.

And for these reports, ….. I don’t really feel stimulated to do my very best to get a good grade because I know it will just be on the threshold and I’ll be fine. (School C, FG1)

The transition from a grade-based system in a previous programme to one based on narrative feedback without grades presented a challenge for students to then find ways of checking they were ‘on track’.

You don’t have that oh, I got an A. Okay, you know, I can feel good about myself. It’s just…you don’t have that external validation. (School A, FG2)

As students adjusted to this new assessment model, they became less dependent on validating labels and focussed more on preparation for practice:

You can see that this is preparing you for real life when you don’t get an A every time you go. (School A, FG2)

As the transition continued, the absence of grading or ranking forced students to take a more nuanced view of their peers’ strengths and weaknesses. This gave them more freedom in using the feedback while preserving their own self-esteem. The lack of reassurance from ‘good enough’ grades incentivized students to aim for excellence:

We don’t know what the bare minimum is. You better have a slam-dunk to make sure. (School A, FG1)

The tendency for tutors at all schools to provide vague positive feedback frustrated students; it failed to meet aspirations to improve performance.

It means very little to me to always get these ‘great job, great job, great job’ versus someone who is trying to find ways to help me get better. (School A, FG2)

It was also commonly perceived that tutors feared giving critical feedback. Students at School A described how they had actively challenged tutors for more constructively critical feedback. There was little evidence of feedback-seeking behaviour at the other sites:

They have to be prompted because they’re afraid to write something bad in our feedback, but it’s necessary. (School A, FG2)

In recent decades behaviourist approaches to learning, relying on passive knowledge acquisition, have moved towards constructivist approaches and active learning strategies [32]. By contrast, the fundamental approach towards assessment has remained behaviourist. Students are rewarded for passing or punished for failing. Many institutions, including those with active learning approaches, have maintained assessments that reward superficial learning strategies, which students unsurprisingly adopt [33, 34]. Although more constructivist principles for assessment have been requested [35‐37], implementation has been limited. Our study demonstrates the potential benefits of three key constructivist principles: (i) improving the authenticity of the assessment, (ii) empowering students with a more active role and (iii) gradual descaffolding to enable transformation from a performance orientation towards a learning orientation. Active involvement in assessment and feedback processes is, however, neither easily achieved nor a panacea, and the role of the authenticity of assessments and the credibility of the mentor/coach as a knowledgeable other are important factors in this transformation. There are clearly patient safety risks if students are given excessive choice within assessments too early in the curriculum and the ensuing liberty to ignore uncomfortable feedback. Nevertheless, we argue that promoting and gradually developing a culture of receptivity to feedback will ultimately benefit patient safety.

A long-term mentoring relationship appears essential for feedback provision to stimulate effective learning. It enables trust to develop in a safe environment in which learners can be challenged. Although such relationships are common in other professions, medical education often involves multiple short-term clerkships [19, 31]. As with the transition to problem-based learning, mentors may struggle to adopt facilitative rather than paternalistic approaches [38].

The study was not designed to directly compare the programmatic approach to assessment with the more traditional summative assessment approach. Within each school’s assessment system we identified both helpful and counterproductive elements which impacted on receptivity to feedback. The generalization of our findings therefore does not come from stable factors that are supposed to be true in all contexts; rather our findings serve to better understand the driving forces to enable adaptability of this knowledge in the design of assessment in various contexts. Designers of assessment and feedback programmes could benefit from incorporating the more helpful elements in their programmes.

Our study has some limitations. Only the participants’ perspectives could be interpreted by the authors to co-construct meaning which may mean that certain perspectives may be over-represented and others may be under-represented. We do not claim to have achieved data saturation over the six focus groups. So although we think that the themes emerging from our study are important and make good sense, we are confident that there are other interesting themes that would emerge from future replications of our study. As such the combination of such studies would serve to complete the whole picture. A further support for the sensibility of our co-constructed conclusions is that they resonate with the existing feedback literature and yet allow a more focused ‘lens’ on the interactional factors between the learner and their learning context. The decision to sample different year groups was made on good grounds, but the differences between curricula may also have led to under-representation of certain perceptions, so again we want to stress that we do not assume to have been able to provide a complete picture of all themes. To triangulate students’ self-reported perceptions, it would be important in future studies to explore the perceptions of mentors and other faculty regarding the students’ receptivity to feedback.

Our results suggest several opportunities for further investigation. While it may be clear that there is a need to gradually shift the control over the assessment process from teacher to learner it is important to better understand the factors that could slow down or accelerate this process. Future studies seeking to understand these factors would be highly informative for assessment design. As stated above it would also be important to replicate our study in a different context to establish whether important themes regarding the uptake of feedback would be detected that our study did not find. Finally, research into the conceptualizations of faculty is of paramount importance to allow for triangulation of these themes.