Phase 1: Qualitative
Twenty-one students (14 women, 7 men; mean age: 24 years, range 21–31) participated in three focus groups (3, 8, and 10 participants). Nine (45%) had previous experience creating learning goals, in earlier education or work. Focus group results addressed three themes: information access, information interpretation, and coaching for change.
Most students described using the dashboard primarily to view score reports from written examinations and other performance reports that were automatically uploaded. One student explained: ‘I think of it as the assessment help, and for any type of assessment I can find out how I did there.’ (Group 2) While students in each group initially described obtaining score reports in the dashboard, each group then characterized other purposes of the dashboard, including capturing individual and longitudinal performance information. One student noted: ‘It’s one of the few sites where I feel like it’s personal to me.’ (Group 3)
Students had difficulty interpreting all information housed in the dashboard independently. Viewing information about their performance seemed to provide them some information while also raising many questions. One student observed: ‘It’s still a little bit vague, but that’s the information we have on how we’re doing.’ (Group 3) They desired more feedback, more longitudinal performance views, transparency in grading, and individualized comments on all assessments. Many students shared technical questions about particular dashboard features or suggested features that could be added. Many wanted more detailed information about their performance particularly relative to peers. The dashboard raised concerns for some about how the school might use performance information in ways they might not know to compare them with peers and school expectations.
Coaching for change:
Students praised their coaches highly as supports and mentors, yet expressed mixed opinions of their experience discussing learning planning with coaches. For some students, the process of setting learning goals for posting in the dashboard and discussing with the coach seemed productive. Coach discussions enhanced students’ perceptions of their ability to interpret their own progress:
I’ve actually found it really helpful to have time one on one to talk about academic things and SMART goals because I feel like sometimes the feedback from tests is super generic. You either met expectations, you were borderline, or you didn’t, and I often don’t really know what does that mean for what I’m doing and what I need to change? (Group 3)
In required progress meetings with coaches, some students viewed the dashboard together with coaches and found this step enlightening. One student appreciated that the coach advised: ‘See, by itself, this may be not the most useful thing in the world, but over time you can see a trend.’ (Group 1) Another student described the coach providing clarification about information in the dashboard: ‘With her eye, she can also make sense of the information and tell me so this is how it can be helpful for you.’ (Group 1).
Students perceived learning plan development with coaches to be more effective when goals were meaningful to the student rather than seeming to be dictated by data in the dashboard. They appreciated longitudinal coach relationships for individualized guidance and reassurance. They shared tentative thoughts that they might be developing a habit of mind through reflecting and articulating goals. One student characterized progress meetings: ‘By giving us this time almost off they’re forcing us to zoom out and step back and say, hey, wait, what am I working towards?’ (Group 1).
However, others reported that coaches did not steer them to use the dashboard together; those students trusted that their coaches had checked the dashboard in advance to ensure performance met expectations. Students valued agenda flexibility to address issues most salient to the student when meeting with coaches. Students reported that coaching meetings focused on varying topics including general academic support, career guidance, and students’ personal wellbeing:
I think coaching time is to just talk about what you want. People in our sessions have talked about very personal issues, people have talked about the curriculum, people have talked about SMART goals …just that openness … that’s a really helpful setting. (Group 2)
Multiple students struggled with figuring out how creating learning goals would benefit them. Some felt they already used effective learning strategies:
It’s not helpful to my learning or my long-term career goals because often what I’m doing is taking things that I’m already doing and then somehow squishing them into this format that I can just put on this dashboard. (Group 1)
Another barrier was the perception that completed learning experiences cannot be changed: ‘The dashboard is post-exam; it’s like, Well, there’s nothing I can do about that now.’ (Group 3).
Some students were still considering how to engage in the process of using performance review to identify and implement goals. One student explained:
I don’t always go into meetings with my coach with clear hopes or goals, but actually for my last session I had no ideas, and she was really helpful in talking to me and figuring out where to improve.’ (Group 3)
Many students similarly described reviewing their performance without having previously engaged in planning as part of a cyclic process.
Phase 2: Quantitative
Overall, 114 of 152 first-year students completed the survey (75%). Tab. 1
provides survey responses.
114 first-year medical student responses to a survey on dashboard usage, coaching and learning planning, University of California, San Francisco, School of Medicine
Factor 1: SMARTb goals
(α = 0.88)
I regularly set SMART goals for myself to improve my learning
SMART goals are a useful tool
I find it useful to create SMART goals related to my academic performance
I find it useful to create SMART goals related to my career planning
I find it useful to create SMART goals related to my well-being
I keep track of the progress of my SMART goals
Factor 2: Dashboard usage
(α = 0.82)
I usually choose which goal to create based on data on my performance from the dashboard
I am able to review my individual performance readily using the dashboard
It is helpful to review my performance compared with other students using the dashboard
The breadth of data in the dashboard is sufficient for me to assess my academic performance
The exam grade reports in the dashboard help in assessing my performance
I find it easy to track my exam scores over time to monitor improvement in my performance
Reviewing my dashboard motivates me to improve my performance
The information about my performance in the dashboard is accurate
The information about my performance in the dashboard reflects my progression as a medical student
Factor 3: Learning strategy
(α = 0.81)
I use effective learning tools and strategies
When I study, I test myself to check my understanding of what I’ve studied
I know how to identify the best learning resources to make progress in an area that I need to improve in academically
I have the skills required to be a lifelong learner
I am adequately supported to implement my SMART learning goals
I currently know how to interpret feedback and data/scores to improve my academic performance
Factor 4: Coaching for improvement (α = 0.71)
My coach helps me create or adjust my SMART goals
Working with my coach helps me understand how to use performance data and feedback to create SMART learning goals
My coach helps me to assess and reflect on the progress I have made on my goals
Factor 5: Reflection
(α = 0.70)
When I receive feedback, I often reflect and set new goals
I direct my learning based on my individual learning needs
I mainly use sources other than the dashboard to review how I am doing in medical school
If I’m not on track with my goals, I try to make adjustments and find resources to achieve those goals on my own
When I receive feedback, I use that information to change my behaviour
Factor analysis yielded five factors explaining 57% of the variance characterizing students’ perception of the various Master Adaptive Learner behaviours and infrastructure components. Cronbach’s alpha for the five factors ranged from 0.70 to 0.88. Factor 1 represented items related to learning goals development (6 items, factor mean 3.25, SD 0.91). Factor 2 addressed dashboard usage (9 items, mean 3.36, SD 0.64). The third factor addressed employment of learning strategies (6 items, mean 3.72, SD 0.64). The fourth factor focused on coaching (5 items, mean 3.67, SD 0.79). The fifth factor addressed reflection (5 items, mean 3.68, SD 0.64). Students’ perception of learning strategies (p < 0.000), coaching (p < 0.000), and reflection (p < 0.000) were all significantly higher compared with dashboard usage and learning goals development. Three individual items with the most positive responses (mean >4) pertained to students’ endorsement of their own skills in self-directed and lifelong learning, and incorporation of feedback.