Abstract
Demonstrating the impact of faculty development, is an increasingly mandated and ever elusive goal. Questions have been raised about the adequacy of current approaches. Here, we integrate realist and theory-driven evaluation approaches, to evaluate an intensive longitudinal program. Our aim is to elucidate how faculty development can work to support a range of outcomes among individuals and sub-systems in the academic health sciences. We conducted retrospective framework analysis of qualitative focus group data gathered from 79 program participants (5 cohorts) over a 10-year period. Additionally, we conducted follow-up interviews with 15 alumni. We represent the interactive relationships among contexts, mechanisms, and outcomes as a “mandala” of faculty development. The mandala illustrates the relationship between the immediate program context, and the broader institutional context of academic health sciences, and identifies relevant change mechanisms. Four primary mechanisms were collaborative-reflection, self-reflection and self-regulation, relationship building, and pedagogical knowledge acquisition. Individual outcomes, including changed teaching practices, are described. Perhaps most interestingly, secondary mechanisms—psychological and structural empowerment—contributed to institutional outcomes through participants’ engagement in change leadership in their local contexts. Our theoretically informed evaluation approach models how faculty development, situated in appropriate institutional contexts, can trigger mechanisms that yield a range of benefits for faculty and their institutions. The adopted methods hold potential as a way to demonstrate the often difficult-to-measure outcomes of educational programs, and allow for critical examination as to how and whether faculty development programs can accomplish their espoused goals.
Similar content being viewed by others
References
Abhyyankar, P., Cheyne, H., Maxwell, M., Harris, F., & McCourt, C. (2013). A realist evaluation of a normal birth programme. Evidence Based Midwifery, 11(4), 112–119.
Ashford, S. (2003). Reflections on the looking glass: A review of research on feedback-seeking behavior in organizations. Journal of Management, 29(6), 773–799.
Astbury, B. (2013). Some reflections on Pawson’s science of evaluation: A realist manifesto. Evaluation, 19(4), 383–401.
Astbury, B., & Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363–381.
Baldwin, C. K., Hutchinson, S. L., & Magnuson, D. R. (2004). Program theory: A framework for theory-driven programming and evaluation. Therapeutic Recreation Journal, 38(1), 16–31.
Blamey, A., & Mackenzie, M. (2007). Theories of change and realistic evaluation: Peas in a pod or apples and oranges? Evaluation, 13(4), 439–455.
Boud, D. (2001). Using journal writing to enhance reflective practice. New Directions for Adult and Continuing Education, 2001(90), 9–18.
Box, G., & Draper, N. (1987). Empirical model-building and response surfaces. New York, NY: Wiley.
Butts, M. M., Vandenberg, R. J., DeJoy, D. M., Schaffer, B. S., & Wilson, M. G. (2009). Individual reactions to high involvement work processes: Investigating the role of empowerment and perceived organizational support. Journal of Occupational Health Psychology, 14(2), 122–136.
Centra, J. A. (1978). Types of faculty development programs. The Journal of Higher Education, 49(2), 151–162.
Chen, H., & Rossi, P. H. (1983). Evaluating with sense: The theory driven approach. Evaluation Review, 7(3), 283–302.
Chen, H., & Rossi, P. H. (1989). Issues in the theory-driven perspective. Evaluation and Program Planning, 12(4), 299–306.
Chen, H., & Rossi, P. H. (1992). Using theory to improve program and policy evaluations. Santa Barbara, CA: Greenwood Press.
Davis, P. (2005). The limits of realist evaluation: Surfacing and exploring assumptions in assessing the best value performance regime. Evaluation, 11(3), 275–295.
Dey, I. (1999). Grounding grounded theory: Guidelines for qualitative inquiry. San Diego: Academic Press.
Greenhalgh, T., Humphrey, C., Hughes, J., Macfarlane, F., Butler, C., & Pawson, R. (2009). How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Quarterly, 87(2), 391–416.
Haji, F., Morin, M.-P., & Parker, K. (2013). Rethinking programme evaluation in health professions education: Beyond “did it work?”. Medical Education, 47(4), 342–351.
Hetzner, S., Heid, H., & Gruber, H. (2012). Change at work and professional learning: How readiness to change, self-determination and personal initiative affect individual learning through reflection. European Journal of Psychology of Education, 27(4), 539–555.
Hutchinson, L. (2003). Educational environment. BMJ (Clinical research ed.), 326(7393), 810–812.
Kanter, R. M. (1993). Men and women of the corporation. New York, NY: BasicBooks.
Karoly, P. (1993). Mechanisms of self-regulation: A systems view. Annual Review of Psychology, 44(1), 23–52.
Knol, J., & Van Linge, R. (2009). Innovative behaviour: The effect of structural and psychological empowerment on nurses. Journal of Advanced Nursing, 65(2), 359–370.
Laschinger, H., Finegan, J., Shamian, J., & Wilk, P. (2001). Impact of structural and psychological empowerment on job strain in nursing work settings: Expanding Kanter’s model. The Journal of Nursing Administration, 31(5), 260–272.
Laschinger, H., Finegan, J., Shamian, J., & Wilk, P. (2004). A longitudinal analysis of the impact of workplace empowerment on staff nurses’ work satisfaction. Journal of Organizational Behaviour, 25, 527–545.
Leslie, K., Baker, L., Egan-Lee, E., Esdaile, M., & Reeves, S. (2013). Advancing faculty development in medical education: A systematic review. Academic Medicine, 88(7), 1038–1045.
Lieff, S. (2009). Evolving curriculum design: A novel framework for continuous, timely, and relevant curriculum adaptation in faculty development. Academic Medicine, 84(1), 127–134.
Lyso, I. H. (2010). Managerial learning as co-reflective practice: Management development programs—Don’t use it if you don’t mean it. Dissertation, Norges Teknisk-Taturvitenskapelige Universitet.
McEvoy, P., & Richards, D. (2003). Critical realism: A way forward for evaluation research in nursing? Journal of Advanced Nursing, 43(4), 411–420.
McLeod, P. J., & Steinert, Y. (2010). The evolution of faculty development in Canada since the 1980s: Coming of age or time for a change? Medical Teacher, 32(1), e31–e35.
Monaghan, M. (2012). An Interview with Ray Pawson. Evidence and Policy, 8(4), 511–515.
Moon, J. A. A. (2004). Handbook of reflective and experiential learning: Theory and practice. London: Routledge Falmer.
Nesbit, P. L. (2012). The role of self-reflection, emotional management of feedback, and self- regulation processes in self-directed leadership development. Human Resource Development Review, 11(2), 203–226.
Ng, S. L., Kinsella, E. A., Friesen, F., & Hodges, B. (2015). Reclaiming a theoretical orientation to reflection in medical education research: A critical narrative review. Medical Education, 49(5), 461–475.
O’Sullivan, P. S., & Irby, D. M. (2011). Reframing research on faculty development. Academic Medicine, 86(4), 421–428.
Parker, K., Burrows, G., Nash, H., & Rosenblum, N. D. (2011). Going beyond Kirkpatrick in evaluating a clinician scientist program: It’s not “if it works” but “how it works”. Academic Medicine, 86(11), 1389–1396.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.
Pawson, R. (2013). The science of evaluation: A realist manifesto. London: Sage Publications.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage.
Ritchie, J., & Spencer, L. (1994). Qualitative data analysis for applied policy research. In A. Bryman & R. Burgess (Eds.), Analysing qualitative data (pp. 173–194). London: Routledge.
Rogers, P. J., Hasci, T. A., Petrosino, A., & Huebner, T. A. (Eds.). (2000). Program theory in evaluation: Challenges and opportunities. New directions for evaluation (No. 87). San Francisco: Jossey-Bass.
Rushmer, R., Kelly, D., Lough, M., Wilkinson, J. E., & Davies, H. T. O. (2004). Introducing the learning practice-II. Becoming a learning practice. Journal of Evaluation in Clinical Practice, 10(3), 387–398.
Singh, T. (2013). Medical education faculty development : Why it fails to impress us. The National Medical Journal of India, 26(2), 96–99.
Snape, D., & Spencer, L. (2003). The foundations of qualitative research. In J. Ritchie & J. Lewis (Eds.), Qualitative research practice: A guide for social science students and researchers. London: Sage Publications.
Sorinola, O. O., Thistlethwaite, J., Davies, D., & Peile, E. (2015). Faculty development for educators: A realist evaluation. Advances in Health Sciences Education, 20(2), 385–401. doi:10.1007/s10459-014-9534-4.
Spreitzer, G. M. (1995a). An empirical test of a comprehensive model of intrapersonal empowerment in the workplace. American Journal of Community Psychology, 23(5), 601–629.
Spreitzer, G. M. (1995b). Psychological empowerment in the workplace: Dimensions, measurement, and validation. Academy of Management Journal, 38(5), 1442–1465.
Spreitzer, G. M. (2009). Taking stock: A review of more than twenty years of research on empowerment at work. In S. R. Clegg & C. L. Cooper (Eds.), The SAGE handbook of organizational behaviour (Vol. 2). Thousand Oaks, CA: SAGE.
Stame, N. (2004). Theory-based evaluation and types of complexity. Evaluation, 10(1), 58–76.
Steinert, Y. (2010). Faculty development: From workshops to communities of practice. Medical Teacher, 32(5), 425–428.
Steinert, Y. (2014). Faculty development in the health professions: A focus on research and practice. New York: Springer.
Thomas, K. W., & Velthouse, B. A. (1990). Cognitive elements of empowerment: An “interpretive” model of intrinsic task motivation. Academy of Management Review, 15, 666–681.
Ward, D. J., Furber, C., Tierney, S., & Swallow, V. (2013). Using Framework Analysis in nursing research: A worked example. Journal of Advanced Nursing, 69(11), 2423–2431.
Watkins, K. E., Lyso, I. H., & deMarrais, K. (2011). Evaluating executive leadership programs: A theory of change approach. Advances in Developing Human Resources, 13(2), 208–239.
West, S. G., & Aiken, L. S. (1997). Toward understanding individual effects in multicomponent prevention programs: Design and analysis strategies. In K. J. Bryant, M. Windle, & S. G. West (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research (pp. 167–209). Washington, DC: American Psychological Association.
Young, J. E., Williamson, M. I., & Egan, T. G. (2015). Students’ reflections on the relationships between safe learning environments, learning challenge and positive experiences of learning in a simulated GP clinic. Advances in Health Sciences Education,. doi:10.1007/s10459-015-9611-3.
Yukawa, J. (2006). Co-reflection in online learning: Collaborative critical thinking as narrative. International Journal of Computer-Supported Collaborative Learning, 1(2), 203–228.
Zibrowski, E. M., Weston, W. W., & Goldszmidt, M. A. (2008). ‘I don’t have time’: Issues of fragmentation, prioritisation and motivation for education scholarship among medical faculty. Medical Education, 42(9), 872–878.
Acknowledgments
The authors would like to acknowledge Jasmine Sheehan for her coordination of the ESP program, Amy Dionne for her project management and support of the project, and Farah Friesen and Daniel Miller for their editorial support during manuscript preparation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors of this study work at the Centre for Faculty Development.
Ethical standard
The study was approved by St. Michael’s Hospital Research Ethics Board (REB 07-182).
Appendices
Appendix 1: Logic model for longitudinal faculty development program
Appendix 2: Semi-structured interview protocol for longitudinal follow-up with program graduates
-
1.
Please describe your current role(s) in education and what it involves.
-
Probes
-
How does that (your role) affect the people you work with internally/externally?
-
-
2.
We sent you the last reflection paper you wrote while in the program for review. We are interested in exploring your experiences since that point in time.
-
Probes
-
Please describe your journey since your completion of the program?
-
Have there been any changes for you since that point in time?
-
What has your involvement been in education (in the academic health professions education system) since the program?
-
Have you had any influence in your role/capacity/system? In what ways have you influenced your surroundings?
-
-
3.
Please describe any successes you have had since completing the program
-
Probes
-
In what contexts have you had successes?
-
How do you define success?
-
Why do you think you were successful?
-
What are the factors (individual or contextual) that influenced your success?
-
Are there areas where you feel you haven’t achieved what you wanted to achieve?
-
Why do you think you were unsuccessful?
-
What would success look like for you over the next 5 years?
-
-
4.
Who do you talk to about teaching and education?
-
Probes
-
Why (do you talk to these people)?
-
Have the people you talk to (engage with) about education changed since your completion of the program? Please explain
-
Has your approach toward them changed? Please explain
-
Rights and permissions
About this article
Cite this article
Onyura, B., Ng, S.L., Baker, L.R. et al. A mandala of faculty development: using theory-based evaluation to explore contexts, mechanisms and outcomes. Adv in Health Sci Educ 22, 165–186 (2017). https://doi.org/10.1007/s10459-016-9690-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-016-9690-9