Academic Course Evaluation
Theoretical and Empirical Distinctions Between Self-Rated Gain in Competences and Satisfaction with Teaching Behavior
Abstract
This article contributes to the conceptual and empirical distinction between (the assessment of) appraisals of teaching behavior and (the assessment of) self-reported competence acquirement within academic course evaluation. The Bologna Process, the current higher-education reform in Europe, emphasizes education aimed toward vocationally oriented competences and demands the certification of acquired competences. Currently available evaluation questionnaires measure the students’ satisfaction with a lecturer’s behavior, whereas the “Evaluation in Higher Education: Self-Assessed Competences” (HEsaCom) measures the students’ personal benefit in terms of competences. In a sample of 1403 German students, we administered a scale of satisfaction with teaching behavior and the German version of the HEsaCom at the same time. Using confirmatory factor analysis, the estimated correlations between the various scales of self-rated competences and teaching behavior appraisals were moderate to strong, yet the constructs were shown to be empirically distinct. We conclude that the self-rated gains in competences are distinct from satisfaction with course and instructor. In line with the higher education reform, self-reported gains in competences are an important aspect of academic course evaluation, which should be taken into account in the future and might be able to restructure the view of “quality of higher education.” The English version of the HEsaCom is presented in the Appendix.
References
(2004). Using learning outcomes. A consideration of the nature, role, application, and implications for European education of employing “learning outcomes” at the local, national, and international levels. Edinburgh: UK Bologna Seminar.
(1973). The Illinois Course Evaluation Questionnaire: A description of its development and a report of some of its results. Educational and Psychological Measurement, 33, 669–684.
(1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182.
(2005). Simulation study on fit indexes in CFA based on data with slightly distorted simple structure. Structural Equation Modeling, 12, 41–75.
(1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1. New York: McKay.
. (2005). A framework for qualifications of the European Higher Education Area. Greena/Copenhagen: Grefta Tryk A/S. Retrieved on August 22, 2009, from www.bologna-bergen2005.no/Docs/00-Main_doc/050218_QF_EHEA.pdf
(2008). Das Berliner Evaluationsinstrument für selbstgeschätzte studentische Kompetenzen (BEvaKomp) [
Competence-based course evaluation: Berlin Evaluation Instrument for students’ competences ]. Diagnostica, 54, 30–42.(2006). Ergebnisorientierte Lehrveranstaltungsevaluation [
Outcome-oriented course evaluation ]. In , Qualitätsentwicklung an Hochschulen – Erfahrungen und Lehren aus 10 Jahren Evaluation [Quality development in higher education institutions – 10 years of experience ] (pp. 60–67). Bonn: HRK.(2001). Lifelong learning for equity and social cohesion a new challenge to higher education. Strasbourg: Final Conference Report.
(1986). Measuring student’s perceptions of teaching: Dimensions of evaluation. Educational and Psychological Measurement, 46, 63–9.
(1995). Student ratings of teaching: The research revisited. IDEA Paper No. 32. Manhattan: Kansas State University, Center for Faculty Evaluation and Development.
(1998). The development of competence in favorable and unfavorable environments: Lessons from research on successful children. American Psychologist, 53, 205–220.
(2001). A comparison of inclusive and restrictive strategies in modern missing-data procedures. Psychological Methods, 6, 330–351.
(1995). Mapping prior knowledge: A framework for discussion among researchers. European Journal of Psychology of Education, 10, 225–242.
(2003). Einführung [
Introduction ]. In , Handbuch Kompetenzmessung. Erkennen, Verstehen und Bewerten von Kompetenzen in der betrieblichen, pädagogischen und psychologischen Praxis [Handbook of competence assessment. Identification, comprehension, and evaluation of competences in job-related, pedagogical, and psychological practice ] (pp. 9–19). Stuttgart: Schaeffer-Poeschel.(2006). Analyzing structural equation models with missing data. In , Analyzing structural equation models with missing data (pp. 313–342). Greenwich, CT: Information Age.
(2004). Using an em covariance matrix to estimate structural equation models with missing data: Choosing an adjusted sample size to improve the accuracy of inferences. Structural Equation Modeling, 11, 1–19.
(2005). The relationship between the need for closure and support for military action against Iraq: Moderating effects of national attachment. Personality and Social Psychology Bulletin, 31, 621–632.
(2000). Performance of modified test statistics in covariance and correlation structure analysis under conditions of multivariate nonnormality. Structural Equation Modeling, 7, 356–410.
(1998). Do people know how they behave? Self-reported act frequencies compared with on-line codings by observers. Journal of Personality and Social Psychology, 74, 1337–1349.
(1998). How useful are student ratings? American Psychologist, 53, 1228–1229.
(1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52, 1182–1186.
(1998). The development of self-concept and interests. In , Interest and learning (pp. 105–125). Kiel: IPN.
(2006). Relations among student effort, perceived class difficulty appropriateness, and student evaluations of teaching: Is it possible to “buy” better evaluations through lenient grading? College Student Journal, 40, 588–596.
(1993). Testing structural equation models. In , Testing structural equation models (pp. 294–317). Newbury Park: Sage.
(2002). Fächerübergreifende Kompetenzen [
Interdisciplinary competences ]. In , Leistungsmessungen in der Schule [Measurements of achievement in schools ] (pp. 203–218). Weinheim: Beltz.(1999). Student ratings of instruction in the USA: Hidden assumptions and missing conceptions about “good” teaching. Studies in Higher Education, 24, 27–42.
(1990). Methodological explanations for the modest effect of feedback from student ratings. Journal of Educational Psychology, 82, 232–241.
(1976). The nature and causes of job satisfaction. In , Handbook of industrial and organizational psychology (pp. 1297–1349). Chicago: Rand McNally.
(2006). Global self-assessment. In , Handbook of multimethod measurement in psychology (pp. 29–42). Washington, DC: American Psychological Association.
(1984). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76, 707–754.
(1994). Using the national longitudinal study of 1988 to evaluate theoretical models of self-concept: The self-description questionnaire. Journal of Educational Psychology, 86, 439–456.
(1988). Goodness of fit in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391–410.
(2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11, 320–341.
(1997). Making students’ evaluations of teaching effectiveness effective. American Psychologist, 52, 1187–1197.
(2000). Effects of grading leniency and low workload on students’ evaluations of teaching: Popular myth, bias, validity, or innocent bystanders? Journal of Educational Psychology, 92, 202–228.
. (1999). Measuring student knowledge and skills. A new framework for assessment. Paris: OECD Publishing.
(2001). Lehrevaluation. Einführung und Überblick zu Forschung und Praxis der Lehrevaluation an Hochschulen mit einem Beitrag zur Evaluation computerbasierten Unterrichts [
Course evaluation: Introduction and overview on research into and practice of course evaluation in higher education, including a contribution on the evaluation of computer-based teaching ]. Landau: Verlag Empirische Pädagogik.(1998). Workload, grades, and students’ evaluations of teaching: Clear understanding sometimes requires more patient explanations. American Psychologist, 53, 1230–1231.
(1997). Analysis of incomplete multivariate data. London: Chapman and Hall/CRC Press.
(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177.
(1995). Theory-guided design of a rating scale for course evaluation in problem-based curricula. Teaching and Learning in Medicine, 7, 82–91.
(1982). Asymptotic intervals for indirect effects in structural equations models. In , Sociological methodology (pp. 290–312). San Francisco: Jossey-Bass.
(2000). Perspective on personality: The relative accuracy of self versus others for the prediction of emotion and behavior. Journal of Personality, 68, 837–867.
(2005). A note on testing perfect correlations in SEM. Structural Equation Modeling, 12, 551–577.
(2000). The Bologna Declaration: Enhancing the transparency and competitiveness of European higher education. Higher Education in Europe, 25, 205–310.
(1999). Social-motivational processes and interpersonal relationships: Implications for understanding students’ academic success. Journal of Educational Psychology, 91, 76–97.
(2001). Concept of competence: A conceptual clarification. In , Defining and selecting key competences (pp. 17–31). Göttingen: Hogrefe & Huber.
(1996). Identifikation und Erfassung von Komponenten der Studienzufriedenheit [
Identification and measurement of elements of students’ satisfaction ]. Psychologie in Unterricht und Erziehung, 43, 1–22.(1998). Bewertung von Lehrveranstaltungen und Studienbedingungen durch Studierende: Theorieorientierte Entwicklung von Fragebögen [
Students’ evaluation of courses and study conditions: Theory-oriented development of questionnaires ]. Empirische Pädagogik, 12, 133–166.
