Practices and challenges in educational program evaluation in the Asia-Pacific region: Results of a Delphi study

doi:10.1016/j.evalprogplan.2008.08.003

Evaluation and Program Planning

Volume 31, Issue 4, November 2008, Pages 368-375

https://doi.org/10.1016/j.evalprogplan.2008.08.003 Get rights and content

Abstract

While educational program evaluation has become more important in recent years because of increasing governmental demands for accountability, little is known about the development of and issues in regard to this topic in the Asia-Pacific region. The findings from a Delphi study conducted in a number of relevant countries are provided in this article. Thirty panelists participated in three rounds of iterations to identify current status, challenges, and concerns in the near future. The results should be of value to educational evaluators, researchers, and decision makers.

Introduction

As the demand for accountability continues to increase, educational program evaluation (EPE) has become more important. To understand where the field is, numerous studies have been conducted in Western countries (Weiss, 1998; Worthen, Sanders, & Fitzpatrick, 2004), but little is known about EPE in the Asia-Pacific region (Mohandas, Wei, & Keeves, 2002). To that end, we used the Delphi technique to study what is happening now and what might take place in the future in this part of the world as determined by a knowledgeable panel of experts.

Section snippets

Background

The emergence of modern EPE in the Asia-Pacific area is traced to the mid-1970s (Lunt & Trotman, 2005; Sharp, 2003) when it was heavily affected by the experiences and writings of American and European evaluators. They were the first ones to undertake large-scale evaluations and to develop evaluation as a discipline (Mohandas et al., 2002; Straton, 2001).

One example of this was apparent in Australia where two forces shaped the field. It first took on the objectivist and positivist stance of

Methodology

The study was exploratory and descriptive. Its goals were to collect perceptions about the current status of and challenges in EPE in selected Asia-Pacific countries as well as what the future might hold. A classical Delphi technique with three iterations via web delivery was utilized. The Delphi is a well-recognized group communication process that solicits judgment through iterative sequential questionnaires interspersed with summary and feedback of opinions derived from previous responses (

Results and discussion

Twenty-four out of 27 panelists responded to the final round, an 89% return rate. Forty-seven out of the total of 106 statements reached consensus. Higher agreement was noted for the concept of EPE as compared to current and future statuses. The pattern is evident in Table 3. Major findings are summarized below followed by discussion. They are based on quantitative results and what was gleaned from the comments on the initial round and from open-ended items on the other ones.

Lessons learned

Our findings are suggestive of the current nature of and challenges for EPE in the Asia-Pacific region. They should be helpful to evaluators, researchers, and decision makers for policy formation, the implementation of evaluation, and for the training of evaluators. They may also generalize to other parts of the world where evaluation is more in nascent or early stages of development. While the diversity of opinion for current status, strengths, weaknesses, and challenges of EPE demonstrated

Yi-Fang Lee Ph.D. is an Assistant Professor at National Chi Nan University in Taiwan. She has published and presented in the area of needs assessment and Science, Technology, Engineering, and Mathematics retention with underrepresented minorities.

References (31)

R. McTaggart et al.
Evaluation traditions in Australia: Distillation of the old, wellspring of the new
Evaluation and Program Planning
(1991)
M.C. Alkin
Evaluation theory development
Evaluation Comment
(1969)
APQN. (2008). About APQN. Retrieved January 10, 2008, from...
L. Berends et al.
Evaluation standards and their application to indigenous programs in Victoria, Australia
Evaluation Journal of Australasia
(2003)
K.W. Brooks
Delphi technique: Expanding applications
North Central Association Quarterly
(1979)
N.C. Dalkey et al.
An experimental application method to the use of experts
Management Science
(1963)
A.L. Delbecq et al.
Group techniques for program planning
(1975)
E.W. Eisner
The educational imagination: On the design and evaluation of school programs
(1975)
R.E. Floden
Flexner, accreditation, and evaluation
D. Fraser
National evaluation standards for Australia and New Zealand: Many questions but few answers
New Directions for Evaluation
(2004)

E.G. Guba et al.

Fourth generation evaluation

(1989)

E.R. House

Justice in evaluation

Joint Committee on Standards for Educational Evaluation

The program evaluation standards

(1994)

H.A. Linstone et al.

The Delphi method: Techniques and applications

(1975)

N.T. Lunt et al.

A stagecraft of New Zealand evaluation

Evaluation Journal of Australasia

(2005)

Cited by (18)

Iran's university admission programme at change: Policies, prospects and pitfalls from planners’ perspectives
2018, Studies in Educational Evaluation
Citation Excerpt :
In total, the panel included 15 members, all PhDs except one with MA degree, and with age range of 40–55. Inspection of the literature on Delphi studies documents a structured process of collecting and synthetizing knowledge from a group of experts through a series of interviews and questionnaires that are accompanied by subsequent controlled feedback (see Adler and Ziglio, 1996; Geist, 2010; Lee et al., 2008). Following this process, the present study included three main rounds (or a three-phase strategy): a qualitative generative round that was consequently followed by rounds of survey ratings (see Table 1).
Testing-based reforms are perceived as the leverage for promoting quality of learning and teaching. However, such reform policies have failed as often as they have succeeded. As reasons for such a failure were mostly studied after implementation or at completion phase of a reform project, there remain theoretical and analytical gaps regarding exploring policy-making phase. Designed within an evaluation framework, the present study adopted a qualitative-quantitative Delphi method to examine the recent reform policies of Iran’s state University Entrance Examinations (UEEs). Panelists were educational policymakers (N = 15) generating their perspectives in iteration rounds that finally identified ‘a conceptual picture of consequences’, ‘paradigm shift towards edumetric culture’, and ‘ulterior challenges and problems’. Findings reveal that challenges of this change intersect with the policies that still envisage the same selection function for the UEE’s alternative. This would thwart the success rate of the ideals aimed by the directing function of the new policy.
Further considerations of evaluation competencies in Taiwan
2013, Evaluation and Program Planning
Citation Excerpt :
The field of educational program evaluation (EPE) is progressing toward a profession around the world. This is true for the Asia-Pacific region but at the same time concerns have arisen there about the quality of evaluations, shortages of experienced, trained evaluators, and identification of the skills or competencies for qualified persons (Garden, 2010; Hay, 2010; Hung, Altschuld, & Lee, 2012; Kumar, 2010; Lee, Altschuld, & Hung, 2008). Accordingly in 2011, Lee et al. (2012) conducted research in the region related to the topic.
A list of evaluator competencies (Stevahn, King, Ghere, & Minnema, 2005) was adapted to fit the Taiwanese context by Lee, Altschuld, & Lee (2012). It was studied as to how it generalized to a large sample in Taiwan. Likert and Fuzzy surveys with needs assessment formats (importance and competence) were mailed via random assignment to two groups of participants. The questions for the study were: do the modified competencies relate country-wide to Taiwan, did the investigation uncover training needs for evaluators, and were there convergent rating patterns across the two forms of the instrument? The results supported a fit of the modified competencies to the context and convergent validity was observed but strong competency needs were not apparent. Reasons for the findings and implications for future research are discussed.
A qualitative method proposal to improve environmental impact assessment
2013, Environmental Impact Assessment Review
Citation Excerpt :
In order to establish a system capable of accurately calculating the importance of the activities subject to EIA, we used the Delphi method, which is a structured communication technique that relies on a panel of experts. It has been widely applied and validated in research carried out all over the world (Dalkey and Helmer, 1963; Kamal and Bashar, 2009; Lee et al., 2008; Thompson and Schaffer, 2002; Yu-Chun et al., 2007). Kontic (2000) regards expert opinions as the backbone of all EIA processes.
In environmental impact assessment, qualitative methods are used because they are versatile and easy to apply. This methodology is based on the evaluation of the strength of the impact by grading a series of qualitative attributes that can be manipulated by the evaluator. The results thus obtained are not objective, and all too often impacts are eliminated that should be mitigated with corrective measures. However, qualitative methodology can be improved if the calculation of Impact Importance is based on the characteristics of environmental factors and project activities instead on indicators assessed by evaluators. In this sense, this paper proposes the inclusion of the vulnerability of environmental factors and the potential environmental impact of project activities. For this purpose, the study described in this paper defined Total Impact Importance and specified a quantification procedure. The results obtained in the case study of oil drilling in Colombia reflect greater objectivity in the evaluation of impacts as well as a positive correlation between impact values, the environmental characteristics at and near the project location, and the technical characteristics of project activities.
Essential competencies for program evaluators in a diverse cultural context
2012, Evaluation and Program Planning
Citation Excerpt :
This is especially relevant in Taiwan where there has been progress toward professionalization of evaluation, but at the same time, shortages of experienced evaluators are apparent, stable career opportunities in the field are lacking, and formal preparation programs need to be developed (Lee, Altschuld, & Hung, 2008). Evaluation has been influenced by the US (Lee et al., 2008) yet cultural forces shape what an evaluator does and what constitutes meaningful evaluation work. These were major considerations in this study.
Essential evaluator competencies as identified by Stevahn, King, Ghere, and Minnema (2005) were studied in regard to how well they generalize to an Asian (Taiwan) context. A fuzzy Delphi survey with two iterations was used to collect data from 12 experts. While most competencies fit Taiwan, there were a number of unique ones. A complete set of results is provided along with the implications of the findings and what they might mean for evaluation in Taiwan particularly in relationship to the professionalization of evaluation.
Exploring training needs of educational program evaluators in the Asia-Pacific region
2012, Evaluation and Program Planning
Citation Excerpt :
There is much interest in EPE globally but little is known about it in this area of the world (Mohandas, Wei, & Keeves, 2002; Owen, 2003). This paper is based on a further analysis of data from a Delphi study examining EPE development across 11 Asian countries (Hung, Altschuld, & Lee, 2008; Lee, Altschuld, & Hung, 2008). The results presented here enhance a deeper understanding about the current status of and challenges in EPE, training professionals in the field, and problems that might be encountered in Asia-Pacific nations.
Training is critical for the growth of the evaluation profession as more qualified and competent individuals will be required to meet the increased demand for evaluation. This paper deals with the training needs of professionals and is based on a part of a study on the status and challenges of educational program evaluation in the Asia-Pacific region. A web-based Delphi was used with a needs assessment framework for examining the current nature of training, what should be done, and discrepancies (gaps) between the two. The results should be of value for academic training programs and those in non-academic settings.
Indicators of Child Victimization in High-conflict Divorce: Interprofessional Consensus Based on a Delphi Panel
2024, Anuario de Psicologia Juridica

View all citing articles on Scopus

James W. Altschuld is Professor Emeritus of Quantitative Research, Evaluation and Measurement in Education at The Ohio State University. His research and writing interests are in needs assessment and the training of evaluators.

Hsin-Ling Hung is Research Associate at the Evaluation Services Center, University of Cincinnati. Her research interests include educational program evaluation, needs assessment, and research on the special needs population.

¹: Tel.: +886 49 291 0960x2611; fax: +886 49 291 5306.

²: Tel.: +1 513 556 7004; fax: +1 513 556 5112.

View full text

Practices and challenges in educational program evaluation in the Asia-Pacific region: Results of a Delphi study

Abstract

Introduction

Section snippets

Background

Methodology

Results and discussion

Lessons learned

Evaluation and Program Planning

Evaluation theory development

Evaluation Comment

Evaluation standards and their application to indigenous programs in Victoria, Australia

Evaluation Journal of Australasia

Delphi technique: Expanding applications

North Central Association Quarterly

An experimental application method to the use of experts

Management Science

Group techniques for program planning

The educational imagination: On the design and evaluation of school programs

Flexner, accreditation, and evaluation

National evaluation standards for Australia and New Zealand: Many questions but few answers

New Directions for Evaluation

Fourth generation evaluation

Justice in evaluation

The program evaluation standards

The Delphi method: Techniques and applications

A stagecraft of New Zealand evaluation

Evaluation Journal of Australasia