Practices and challenges in educational program evaluation in the Asia-Pacific region: Results of a Delphi study
Introduction
As the demand for accountability continues to increase, educational program evaluation (EPE) has become more important. To understand where the field is, numerous studies have been conducted in Western countries (Weiss, 1998; Worthen, Sanders, & Fitzpatrick, 2004), but little is known about EPE in the Asia-Pacific region (Mohandas, Wei, & Keeves, 2002). To that end, we used the Delphi technique to study what is happening now and what might take place in the future in this part of the world as determined by a knowledgeable panel of experts.
Section snippets
Background
The emergence of modern EPE in the Asia-Pacific area is traced to the mid-1970s (Lunt & Trotman, 2005; Sharp, 2003) when it was heavily affected by the experiences and writings of American and European evaluators. They were the first ones to undertake large-scale evaluations and to develop evaluation as a discipline (Mohandas et al., 2002; Straton, 2001).
One example of this was apparent in Australia where two forces shaped the field. It first took on the objectivist and positivist stance of
Methodology
The study was exploratory and descriptive. Its goals were to collect perceptions about the current status of and challenges in EPE in selected Asia-Pacific countries as well as what the future might hold. A classical Delphi technique with three iterations via web delivery was utilized. The Delphi is a well-recognized group communication process that solicits judgment through iterative sequential questionnaires interspersed with summary and feedback of opinions derived from previous responses (
Results and discussion
Twenty-four out of 27 panelists responded to the final round, an 89% return rate. Forty-seven out of the total of 106 statements reached consensus. Higher agreement was noted for the concept of EPE as compared to current and future statuses. The pattern is evident in Table 3. Major findings are summarized below followed by discussion. They are based on quantitative results and what was gleaned from the comments on the initial round and from open-ended items on the other ones.
Lessons learned
Our findings are suggestive of the current nature of and challenges for EPE in the Asia-Pacific region. They should be helpful to evaluators, researchers, and decision makers for policy formation, the implementation of evaluation, and for the training of evaluators. They may also generalize to other parts of the world where evaluation is more in nascent or early stages of development. While the diversity of opinion for current status, strengths, weaknesses, and challenges of EPE demonstrated
Yi-Fang Lee Ph.D. is an Assistant Professor at National Chi Nan University in Taiwan. She has published and presented in the area of needs assessment and Science, Technology, Engineering, and Mathematics retention with underrepresented minorities.
References (31)
- et al.
Evaluation traditions in Australia: Distillation of the old, wellspring of the new
Evaluation and Program Planning
(1991) Evaluation theory development
Evaluation Comment
(1969)- APQN. (2008). About APQN. Retrieved January 10, 2008, from...
- et al.
Evaluation standards and their application to indigenous programs in Victoria, Australia
Evaluation Journal of Australasia
(2003) Delphi technique: Expanding applications
North Central Association Quarterly
(1979)- et al.
An experimental application method to the use of experts
Management Science
(1963) - et al.
Group techniques for program planning
(1975) The educational imagination: On the design and evaluation of school programs
(1975)Flexner, accreditation, and evaluation
National evaluation standards for Australia and New Zealand: Many questions but few answers
New Directions for Evaluation
(2004)
Fourth generation evaluation
Justice in evaluation
The program evaluation standards
The Delphi method: Techniques and applications
A stagecraft of New Zealand evaluation
Evaluation Journal of Australasia
Cited by (18)
Iran's university admission programme at change: Policies, prospects and pitfalls from planners’ perspectives
2018, Studies in Educational EvaluationCitation Excerpt :In total, the panel included 15 members, all PhDs except one with MA degree, and with age range of 40–55. Inspection of the literature on Delphi studies documents a structured process of collecting and synthetizing knowledge from a group of experts through a series of interviews and questionnaires that are accompanied by subsequent controlled feedback (see Adler and Ziglio, 1996; Geist, 2010; Lee et al., 2008). Following this process, the present study included three main rounds (or a three-phase strategy): a qualitative generative round that was consequently followed by rounds of survey ratings (see Table 1).
Further considerations of evaluation competencies in Taiwan
2013, Evaluation and Program PlanningCitation Excerpt :The field of educational program evaluation (EPE) is progressing toward a profession around the world. This is true for the Asia-Pacific region but at the same time concerns have arisen there about the quality of evaluations, shortages of experienced, trained evaluators, and identification of the skills or competencies for qualified persons (Garden, 2010; Hay, 2010; Hung, Altschuld, & Lee, 2012; Kumar, 2010; Lee, Altschuld, & Hung, 2008). Accordingly in 2011, Lee et al. (2012) conducted research in the region related to the topic.
A qualitative method proposal to improve environmental impact assessment
2013, Environmental Impact Assessment ReviewCitation Excerpt :In order to establish a system capable of accurately calculating the importance of the activities subject to EIA, we used the Delphi method, which is a structured communication technique that relies on a panel of experts. It has been widely applied and validated in research carried out all over the world (Dalkey and Helmer, 1963; Kamal and Bashar, 2009; Lee et al., 2008; Thompson and Schaffer, 2002; Yu-Chun et al., 2007). Kontic (2000) regards expert opinions as the backbone of all EIA processes.
Essential competencies for program evaluators in a diverse cultural context
2012, Evaluation and Program PlanningCitation Excerpt :This is especially relevant in Taiwan where there has been progress toward professionalization of evaluation, but at the same time, shortages of experienced evaluators are apparent, stable career opportunities in the field are lacking, and formal preparation programs need to be developed (Lee, Altschuld, & Hung, 2008). Evaluation has been influenced by the US (Lee et al., 2008) yet cultural forces shape what an evaluator does and what constitutes meaningful evaluation work. These were major considerations in this study.
Exploring training needs of educational program evaluators in the Asia-Pacific region
2012, Evaluation and Program PlanningCitation Excerpt :There is much interest in EPE globally but little is known about it in this area of the world (Mohandas, Wei, & Keeves, 2002; Owen, 2003). This paper is based on a further analysis of data from a Delphi study examining EPE development across 11 Asian countries (Hung, Altschuld, & Lee, 2008; Lee, Altschuld, & Hung, 2008). The results presented here enhance a deeper understanding about the current status of and challenges in EPE, training professionals in the field, and problems that might be encountered in Asia-Pacific nations.
Indicators of Child Victimization in High-conflict Divorce: Interprofessional Consensus Based on a Delphi Panel
2024, Anuario de Psicologia Juridica
Yi-Fang Lee Ph.D. is an Assistant Professor at National Chi Nan University in Taiwan. She has published and presented in the area of needs assessment and Science, Technology, Engineering, and Mathematics retention with underrepresented minorities.
James W. Altschuld is Professor Emeritus of Quantitative Research, Evaluation and Measurement in Education at The Ohio State University. His research and writing interests are in needs assessment and the training of evaluators.
Hsin-Ling Hung is Research Associate at the Evaluation Services Center, University of Cincinnati. Her research interests include educational program evaluation, needs assessment, and research on the special needs population.