The purpose of this study was to develop an effective evaluation tool to review course syllabi as a method for ensuring effective outcomes for interdisciplinary personnel preparation. A well-written course syllabus reflects the quality of the academic program in which the coursework is a part. A review of the syllabi across interdisciplinary programs is an attempt to align the quality of the courses and the graduate student competencies as set forth by each program. A syllabus evaluation form was developed and used systematically by three reviewers who gave each syllabus in the program a rating based on the three project target competencies and the listed indicators. For this project, three common components of syllabi were reviewed: learning outcomes, readings, and assignments. The findings suggest that a systematic syllabi review can be used to determine individual course alignment and for overall program evaluation. Findings also suggest that a syllabi evaluation form can be an effective tool to use in systematic syllabi reviews.
Opmerkingen
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Well-prepared personnel are needed to work with young children with disabilities in high-need communities and in diverse least restrictive environments. National, state, and local data clarify the need for personnel who understand the effects of poverty on young children’s development, work effectively with families and other agencies, address resource access, and improve services to ensure the developmental progress and academic success of children (Coleman et al., 2020; Magnuson & Waldfogel, 2005; Reed, 2012). Together, these factors require early intervention/early childhood special education (EI/ECSE) personnel to be culturally competent while also skilled in working directly with infants and young children with disabilities in varied community settings. EI/ECSE personnel need to be well-prepared to assess, identify, and provide culturally responsive supports and resources to serve diverse children and families. They need to be equipped with knowledge, skills, and dispositions in designing learning environments that promote optimal progress for all young children, in linking assessment to instruction and monitoring programming effectiveness, and in collaborating with families and professionals from diverse backgrounds.
Interdisciplinary collaboration is a recommended practice for infants and young children with disabilities (Division for Early Childhood, 2014). In the field of special education, team collaboration is an essential requirement of the Individuals with Disabilities Education Act (IDEA), which is also well documented in the literature (Bruder et al., 2019; Dettmer et al., 2009; Friend & Cook, 2010; Stayton, 2015). Interdisciplinary team collaboration is also emphasized by EI/ECSE standards (Division for Early Childhood [DEC], 2020) and clinical guidelines (Adams et al., 2013). Furthermore, ensuring special education services delivered in the least restrictive environments requires effective teamwork to support desired outcomes for young children with disabilities, including those with high-intensity needs (DEC, 2020; National Association for the Education of Young Children (2019)) It is challenging to provide high-quality services to children with severe medical, behavioral, and emotional disabilities (Chen et al., 2009). Providing high-quality services means learning when and how to rely on both discipline-specific and shared interdisciplinary expertise.
Review of Literature
The syllabus has an important role in the teaching-learning interaction as it sets the parameters for what will be taught and how. Richmond et al. (2014) define the syllabus as a document that communicates content coverage and important student requirements; Parkes and Harris (2002) view the roles of a syllabus as three pronged-as a contract, as a permanent record, and as a learning tool. The role of the syllabus as a learning tool is very relevant especially when we look at the teacher preparation courses. For teacher preparation, the syllabus becomes a crucial document- what is it that we focus on and what are we excluding? These choices will determine what gets transacted in the teacher education program and what skills and competencies are possessed by the teachers who graduate from these programs. A review of the syllabus can thus be a starting point for analyzing the quality of teacher preparation.
Recently, more attention is being given to the alignment of professional courses with the requirements set out by professional bodies in the field. Sayeski and Higgins (2014) point out the federal requirements of having highly qualified teachers (HQTs) in special education as in general education, instruction that defines standards and expects students to master those standards, and the evolving requirements for accreditation in higher education as the push factors for revamping the curriculum as well as the syllabus. They further acknowledge that the trend towards placement of students with ‘high–incidence disabilities’ in general classrooms makes the teachers accountable to these students. This fixes the responsibility of the quality of special educators trained on teacher education programs further necessitating such curriculum and syllabus restructuring.
Two US professional bodies, have been engaged in setting standards for Early Childhood and Early Childhood Special Education teacher preparation programs: The National Association for the Education of Young Children’ (NAEYC) and the Division of Early Childhood (DEC) of the Council for Exceptional Children (CEC). (DEC, 2020; NAEYC, 2019; Stayton et al., 2023). These bodies ensure that teachers know and understand the diverse needs of children, their families, and communities so that they can merge meaningful quality teaching into the learning environments (Lim & Able-Boone, 2005).
Syllabus Reviews
Special education teacher preparation programs have been evaluated with respect to various competencies considered to be important for working with special needs populations. Oliver and Reschly (2010) assessed the emphasis on classroom organization and behavior management techniques in a review of course syllabi of 26 teacher preparation programs. They found that most course syllabi do not cover these skills adequately which has implications for the teaching of children with emotional and behavioral disorders. In their review of special education teacher preparation syllabi, they stated that “inadequate teacher preparation hinders inclusion efforts” (p. 189). Gregori-Giralt and Menéndez (2021) propose syllabi evaluation as a form of assessing professors’ instructional practices that is both inexpensive and convenient.
Syllabi have also been reviewed for examining their learner-centeredness, pupil diversity (Kratochvílová1 et al. (2022); Sayeski & Higgins, 2014), inclusion (Servizzi, 2015), culture of teaching and learning (Stanny et al., 2015), teaching culture promoted by institutions and learning outcomes of general education (Doolittle & Siudzinski, 2010; Eberly et al., 2001). Syllabi are also reviewed to ensure the alignment of student learning outcomes with those identified by a professional society for the discipline (e.g. Grauerholz & Gibson, 2006) and to enhance information literacy in the use of library resources (Beouy & Boss, 2019).
Methods of Reviewing
Common methods are found in the literature for conducting syllabi reviews. These commonalities include using core rubrics, training reviewers, and using a process to come to a consensus over disagreements. Researchers also often use predetermined elements to score the rubrics and an agreed-upon scoring system. For example, in a series of four reviews of courses in the University of West Florida, Stanny et al. (2015) used a rubric for conducting a syllabus review for the quality of the syllabus and culture of teaching and learning by focusing on the ‘required components’ and ‘best practices.’ The rubric elements were scored as either absent or present by the raters. ‘Required components’ relate to the syllabi elements based on institutional policy, and ‘best practices’ refer to components related to course design and syllabus available in literature. Required components included basic information like course number and title, contact information, recommended readings, course description, and student learning outcomes. The best practice components included details of class meeting time and location, ISBN of required textbooks, instructor goals for the course, or how the course fits in with program requirements or prepares students for other courses, instructor professional details, software or technological skills required, and strategies for effective study and success in the course (Stanny et al., 2015).
The raters used the rubric in the Stanny et al. (2015) study to come up with two composite scores for rating the quality of the syllabus on the basis of the frequency of ‘required components’ and ‘best practice’ components as described above. For scoring abstract qualities like student engagement, the rubric was used to assess the presence of specific, clear-cut components contributing to the abstract concept. The abstract qualities of the syllabus in this case refer to evidence-based practices for enhancing student engagement and making the class more learning-centered. Some examples of evidence of enhancing student engagement are using incentives to form study groups, providing team skills guidelines and activities for developing those, assigning group projects, and grading the final product through peer evaluation, encouraging student involvement in cultural performances, incorporating multiple modes of teaching along with lecture.
Reviewers in the Stanny et al. (2015) study were trained for the work by evaluating a sample of training syllabi. All of the syllabi were scored by each of the reviewers followed by a computation of pair-wise inter-rater agreement with reviewers being assigned to pairs randomly. In case of any disagreements, consensus was arrived at and the training syllabi were rescored. When reviewers achieved the target of at least 75% agreement across rubric elements, they began scoring independently.
Coleman et al. (2020) carried out a systematic syllabi review of 13 EI/ECSE courses under an Office of Special Education Program (OSEP) funded project that aimed for community-based, evidence-based, and culturally responsive instruction in personnel preparation programs. Specific components of the syllabus, namely, learning objectives, readings, and assignments, were evaluated for consistency with the goals of the project. A rubric was used to assess established syllabi components. The syllabus was examined by three reviewers who followed a process for consensus-building. After reviewing each syllabi individually, the reviewers found variations in their ratings. To overcome this, they created indicator definitions to ensure reliability. Once the syllabi were evaluated and scored by each reviewer independently using the indicator definitions, they met to compare scores. If the scores differed, they discussed and came to mutual agreement to give the syllabi the final scores.
Beouy and Boss (2019) developed a rubric based on the 2015 framework of American College and Research Libraries’ to analyze departmental syllabi for opportunities to ‘scaffold information literacy instruction’. They examined 369 syllabi from three different departments of a university and used a rubric consisting of a three-indicator scale to examine the presence of six specified themes. The reviewers created norming sessions in which each reviewer coded the same syllabi followed by a discussion of the process, any discrepancies among them, and increasing the process validity. This rubric allows instruction librarians to add instruction on information literacy to a disciplinary curriculum by providing a method of obtaining and analyzing data on course syllabi. Though most syllabi reviews have made use of rubrics, some other methods of evaluating syllabi include descriptive content analysis (Pieterse et al., 2009), and qualitative analysis (Fröberg & Lundvall, 2022; Newman et al., 2021).
OSEP Program
Three objectives for the interdisciplinary preparation are addressed through a grant project funded by the U.S. Department of Education Office of Special Education Programs (OSEP): (1) To improve the quality of personnel working with young children with high-intensity needs; (2) To increase the knowledge, skills, and dispositions of pre-service professionals in evidence-based strategic interprofessional practices; and (3) To promote cultural humility and cultural competence to better support the diverse needs of children with disabilities and their families.
The target project model identifies 18 core competencies that emphasize the necessary knowledge, skills, and dispositions for EI personnel from the professional standards set by the DEC Recommended Practices (DEC, 2014), NAEYC’s Developmentally Appropriate Practices (NAEYC, 2019), and the Council on Social Work Education (CSWE, 2015). Achieving these competencies will enable project scholars to support positive learning and developmental outcomes for infants, toddlers, and preschool-age children, including those with significant disabilities and high-intensity needs.
The identified 18 core competencies are expected to be incorporated into each of the target curricula and reflected by core course syllabi, which are structured documents to describe course-specific expectations and to communicate university and unit policies to students (Goodwin et al., 2018). A course syllabus can also provide insight about the instructor’s teaching philosophy and approach to teaching the course content and interacting with students (Cullen & Harris, 2009; Richmond et al., 2019). Therefore, course syllabi evaluation can serve as a valuable and effective tool for examining content expectations and learning outcomes as well as the overall effects of instructional methods. This study focuses on the quality of course syllabi as a method for ensuring the aforementioned outcomes for interdisciplinary project scholars. The goal of this specific project was to evaluate all of the courses across ECSE and Social Work (SW) programs to help ensure the quality of the teacher preparation program. A course syllabus is frequently used as one of the tools to document the content and quality of academic coursework (Willingham–McLain, 2011).
Purpose of the Present Study
Despite a wide range of differences in formats and contents, almost all syllabi include course objectives, assignments, and readings that reflect academic program quality and ensure that course expectations are clearly communicated with students (Stanny et al., 2015). A well-written course syllabus also reflects the quality of the academic program in which the coursework is a part. A review of the syllabi is an attempt to align the quality of the courses and the graduate student competencies as set forth by each program. Although the importance, of course syllabi is recognized by all teacher education programs and best practice components can be identified in any given course syllabus, less has been done to systematically review the quality of syllabi of the core content courses in an academic program. Further, no study has been done to review course syllabi in terms of the alignment with target competencies that are based on professional standards. This course syllabi evaluation study has been undertaken to fill both of these gaps as a component of the federally funded OSEP target project model.
An evaluation form was developed for this study to consistently evaluate course syllabi for interdisciplinary OSEP projects as detailed above. The evaluation form started with the foundational components such as the learning objectives of each course syllabus that align with professional standards and subject matter contents. Building upon the foundations were “constructional materials” to help students learn and comprehend, which were readings and resources. Finally, students would be guided to take actions to demonstrate their knowledge and skills targeted by a specific course, illustrated by specific assignments. These three primary components of a course syllabus would be coherently connected to support the overall course description. However, not all course syllabi were explicitly organized to show this connection, which could lead to miscommunication between the instructor and students. The goal of this project was to evaluate all of the project model’s courses across EI/ECSE and SW programs in order to help ensure the quality of the teacher preparation program. For this interdisciplinary personnel preparation project, professional standards from the Council on Social Work Education (2015) were also included to help identify the 10 core competencies across ECSE and SW. A review of the syllabi is an attempt to align the quality of the syllabi with the graduate student competencies as set forth by each department. As a result of this project, the syllabus evaluation form that was project-developed and tested can be used by stakeholders in other universities to examine their own OSEP programs.
The following research questions guided this review: (1) How does each course align with the interdisciplinary project goals? (2) How do the syllabi combined across the disciplines align with the project goals in order to ensure quality teacher preparation? (3) How can this syllabi review process help inform other syllabi reviews? The current study uses constructivism as a theoretical framework to answer the research questions. Constructivism suggests that learners construct knowledge through their experiences and interactions with the learning environment. A syllabus serves as one artifact in a learning environment whereas the syllabus facilitates active learning, provides opportunities for student engagement, and aligns the learning objectives with objectives with assessment.
Methods
Measures
Similar to the Coleman et al. (2020) study, a syllabus evaluation form was developed and used to analyze each syllabus and give each syllabus a rating for its coverage of the three project target competencies: (1) Working with children and families in high-needs communities, (2) Providing intensive interventions for children with significant disabilities, and (3) Developing interdisciplinary teamwork for optimal child and family outcomes. The developer of the initial syllabus evaluation form was a Full Professor with over 20 years research and teaching experience in EI/ECSE. After initial reviews and feedback from the EI/ECSE faculty team, the form was piloted by a research team using a different but related OSEP project. Results of the pilot led to further revision of the items and refinement of the rating scale. In total, the research team reviewed ten syllabi. Five of these ten syllabi were core courses from the Master of Education in Special Education-Early Childhood program and five other syllabi were part of the concentration courses of the Master of Social Work program. All course syllabi were first reviewed by program faculty in each of the programs by comparing the target competencies. Then the faculty from both programs worked as a team to select the most relevant courses from each of the programs to be included in the final review. See Fig. 1 for the full evaluation form.
Fig. 1
Syllabi Evaluation Form
×
Rubric Development
As described previously, the original developer of the syllabus evaluation form was a Full Professor in EI/ECSE with over 20 years of research and teaching experience as well as program evaluations. The form was initially piloted by a research team using a related OSEP project with similar target competencies. Results of the pilot helped inform revisions of the current form. The initial rating scale was 0 (not mentioned), 1 (mentioned once), 2 (mentioned more than once). After the current research team reviewed the piloted data, the rating scale was modified to increase the variability slightly, however, the team did not want to make the variability too large due to the limited nature of the information that a syllabus provides and because the purpose of this tool is for screening only. Similar to the process used by Coleman et al. (2020) in which they created indicator definitions, the research team further refined the rating rubric with operational definitions to quantify the scoring criteria. The final rating scale that was used is as follows: 0 = not mentioned; 1 = briefly mentioned (1–2 times); 2 = included, but inconsistent (3–4 times); 3 = consistently addressed (5 or more times).
Coding Procedures
Three of the authors served as the syllabi reviewers. One of the reviewers is a doctoral student in special education at a different university than the one where the study took place. She is a licensed early childhood education teacher. A second reviewer is a doctoral student in the school of education where the study took place and has a background in literacy. A third reviewer is an Assistant Professor in Psychology at an international university and has a diploma in Early Childhood Care and Education. She is also a coordinator for a certificate course in Early Childhood Care and Education and served as a Fulbright Humphrey Fellow at the university where the study was conducted.
The reviewers assessed 10 syllabi across the two project programs to investigate how they aligned to the project’s goals. Similar to the way other researchers have conducted syllabi reviews, prior to beginning the official syllabi analysis, the review team analyzed a practice syllabus (i.e. one not included in this review) and met to refine coding procedures (Beouy & Boss, 2019; Coleman et al., 2020; Stanny et al., 2015). During this discussion, the team noted that the evaluators used a different approach to scoring and that the rating scale limited the amount of possible variability in each syllabus. Several cumulative scores were calculated for each syllabus. The syllabi evaluation form was divided into three target competencies, each with six indicators. For each syllabus, the reviewers assigned points for the indicators based on whether or not they were mentioned in the syllabi components: learning objectives, readings, and assignments. Using this scoring scale explained previously, each section of indicators was given a subtotal, with the highest possible score of 54 points (e.g., 9 points per syllabi component × 6 indicators). The three subtotals were then combined for a total score, the highest number of points possible for each syllabus was 162.
For each syllabus, the researchers followed the same coding procedures. First, the project indicators (see Fig. 1) were compared to the syllabi components (learning objectives, readings, and assignments). For each match, the rater provided a tally mark in the corresponding column. For example, the learning objective “Student will demonstrate respect for cultural differences” would receive one tally under the indicator “understand multicultural perspectives.” Each syllabus component could be scored for more than one indicator. For the readings subcomponent, only required readings were rated, as optional readings may or may not be accessed by any individual student in the target course. For consistency and feasibility, only titles of articles or chapters were coded for target competencies, not the entire text.
Reliability
In order to ensure a high percentage of final consensus, all three reviewers coded the first three syllabi. Similar to the “norming sessions” in the Beouy and Boss (2019) study, after each reviewer scored the syllabi independently, the team met to discuss disagreements and come to a consensus. For scores in which two of the three reviewers initially agreed, that score was taken as the final score. For scores on which all three reviewers initially disagreed, the team discussed their thinking and clarified interpretations. In some cases, this led to consensus, and in other cases, the reviewers agreed to disagree and the best of the three scores was used. The next two syllabi were coded by two reviewers in the same manner. Subsequently, a high level of agreement was consistently achieved, and as a result, the team felt confident in the validity of the process so the remaining five syllabi were coded by one reviewer each.
Results
Out of the 10 syllabi reviewed, the mean score was 48.3 points out of a total of 154 possible points. The range of those 10 syllabi was 69 points and the range spanned from a low of 18 points to a high of 87 points. See Fig. 2 for a breakdown of how many points each individual syllabus scored. The results in the following sections are organized by reviewer agreement, the breakdown of target competencies, syllabi components, and discipline. In addition to a total score for each syllabus, the scores for three target competencies in each syllabus were also calculated. The total points possible for each competency to earn was 54.
Fig. 2
Syllabi Scores Across Competencies
×
Reviewer Agreement
In addition to the total number of points scored by each syllabus, the percentages of both the initial and final agreement among the reviewers were also recorded. The mean for the percentage of initial agreement was 49% with a range of 67. The minimum score of the initial agreement was 18% and the maximum was 85%. The mean for the percentage of final agreement was 90%, with a minimum final agreement score of 74% and a maximum final score of 100%.
Working with Children and Families
The competency working with children and families consistently scored the highest of the three competencies with a mean score of 19.5 and a range of 19. The minimum score for this competency among syllabi was 8 and the maximum score was 27. It was present in every single course reviewed, indicating a high overall level of interdisciplinary programmatic coverage of working with children and families.
Developing Interdisciplinary Teamwork for Optimal Child and Family Outcomes
The competency developing interdisciplinary teamwork for optimal child and family outcomes had an average score of 18.4 with a range of 25 points. The minimum score for this competency across syllabi was 9 and the highest score was 34.
Providing Intensive Interventions for Children with Significant Disabilities
The competency providing intensive interventions for children with significant disabilities showed the lowest total scores across the syllabi reviewed. The mean of this competency was 10.4 with a range of 28. The minimum score reported was zero and the maximum was 31. Scoring the lowest on average of the three competency areas (10.4 vs 19.5 and 18.4), providing intensive interventions for children with significant disabilities was the least represented of the competencies. However, it was highly represented in certain courses (e.g., SEDP 533 scored 31 points in this area) suggesting that a handful of courses more specifically targeted this competency rather than having across-the-board representation.
Results by Syllabi Component
In general, the syllabi component that consistently scored the highest was the course objectives. This was due to the fact that there were almost always less course readings and assignments on any given syllabus than there were course objectives. For example, in one syllabus that was reviewed, there were a total of fourteen-course objectives, 4 required readings, and 7 assignments. Similarly, in another example, there were 7 objectives, two required readings, and 4 assignments.
Results by Discipline
There was variability in the results by discipline. The early childhood special education (ECSE) syllabi consistently scored the highest with a range of 52–87 points (note that due to a change in discipline titles, the ECSE and the SEDP acronyms both refer to early childhood special education courses). The syllabi in the social work (SLWK) discipline consistently scored lower than the syllabi in the ECSE discipline with a range of 18–37. For example, the course SLWK 710 (Social Work Policy) scored the lowest across all three competencies. The findings from this syllabi review do not suggest that this course lacks anything, only that its focus was less directly relevant to working with young children with disabilities as operationalized by the present syllabi scoring rubric. However, social work syllabi had a relative strength in the competency working with children and families in high-needs communities. As expected, social work courses less frequently covered competencies relating to Providing intensive interventions for children with significant disabilities. As the goal of social work is to “help people prevent and cope with problems in their everyday lives” (Bureau of Labor Statistics, n.d.) the community focus of these syllabi make sense. In sum, this indicates that an interdisciplinary preparation model can build on the strengths of multiple departments. It is very important to note here, however, that the goal of an interdisciplinary review such as this is not to compare disciplines but rather to ensure that all competencies are being met across the disciplines.
Discussion
The purpose of this federally funded OSEP model subproject was twofold. One purpose was to evaluate the two Master’s programs across the project to compare syllabi and competencies for scholars to determine if the examined programs are meeting the target competency goals to ensure the quality of the teacher education program. The second purpose was to offer a data-driven systematic syllabi review process that others can use for the same purpose mentioned above.
The results of this syllabus evaluation project suggest that course instructors can benefit from the feedback of an external reviewer (Gregori-Giralt and Menéndez (2021)). In alignment with prior research on syllabi reviews, this feedback can offer perspective and remind the instructors of the importance of considering how their course may or may not be consistently demonstrating the core competencies (Doolittle & Siudzinski, 2010; Eberly et al., 2001). The lack of consistency among scores within each target competency suggests a well-scattered result when reviewing the combination of programs overall. No one program or course is expected to cover all of the indicators for each competency. However, it is the expectation of the project model that within each program, the competencies will be covered through a combination of the required coursework.
Course Alignment
One of the questions this study sought to answer was, “How does each course align with the interdisciplinary project goals?” To answer this question, each course syllabus was given an overall score based on a Likert scale rating for each component within the core competencies. This score allows the professor of the course to understand to what depth their course may or may not be supporting their students by aligning competencies (Sayeski and Higgins, 2014). The results suggest that each course evaluated in this study can benefit from further scrutiny and revision by the course facilitator. With the overall possible score that a syllabus could score being 154, and the highest scoring syllabus receiving 87 points with most of the syllabi scoring closer to a 30 to 50 point range, it is evident that each course facilitator can use their own individual results to bulk up their syllabus in the competency areas in which it may be lacking. The evaluation form can provide a clear view and purposeful direction for each facilitator to intentionally accomplish this task.
Program Evaluation
The second research question for this study was, “How do the syllabi combined across the disciplines align with the project goals in order to ensure quality teacher preparation?” When many syllabi are rated and compared across programs, such as the 10 syllabi that were reviewed across the two programs, program teams can meet to discuss how the syllabi combined across the disciplines align with the project goals in order to ensure quality teacher preparation (Gregori-Giralt and Menéndez (2021)).
Results from this study show that improvement is needed in this program overall across the syllabi reviewed. The total points possible for each competency to earn was 54. The competency working with children and families consistently scored the highest with a mean of 19.5 and a range of 19 (8–27). The competency developing interdisciplinary teamwork for optimal child and family outcomes had an average score of 18.4 with a range of 25 (9–34). The competency providing intensive interventions for children with significant disabilities showed the lowest, with a mean of 10.4 and a range of 28 (3–31). The review team noted that this competency tended to score the lowest due to the lack of practicum experience evident in any of the syllabi that were reviewed. It is difficult to meet the requirement of “providing intensive interventions” without the opportunity to interact with children and their families directly. This finding is meaningful because it suggests that a practicum or field experience component is critical to target this competency. Actually, practicum is a part of both ECSE and SW programs. Therefore, future syllabi review should include the practicum and fieldwork course syllabus to provide adequate evidence that the program does indeed allow for this type of experience. Additionally, the program leadership should meet together to discuss how to strengthen each course individually to ensure that the program as a whole is covering the competencies consistently and effectively (Willingham–McLain, 2011).
Evaluation Form as a Tool
Research question 3, “How can this syllabi review process help inform other syllabi reviews?” was answered by way of the design of the form itself and in the validity of the review process. The results of the study suggest that an effectively designed evaluation form contains the competencies and the indicators within each competency that the program expects for the outcomes of graduates. The three universal syllabi components including learning objectives, readings, and assignments, should each be examined for the competencies and indicators (Coleman et al., 2020). Additionally, when several reviewers use the evaluation form to systematically review each syllabus in an agreed-upon manner, the tool can be effective for ensuring quality and effective teacher preparation. (Beouy & Boss, 2019; Coleman et al., 2020; Stanny et al., 2015). It is important to keep in mind that due to the limited nature of information that a syllabus can provide (i.e. when only citations of assigned readings are given rather than a summary or abstract, it is difficult to know all that that reading will entail), this evaluation form is a tool for screening rather than for in-depth scrutiny. Findings of this syllabi review process can help instructors better organize course syllabus contents with coherence to enhance communication between teachers and learners leading to effective outcomes. However, further research needs to be conducted to validate this process.
Overall, the information gained through the process of this study can guide program facilitators in supporting strong syllabi development with the overall goal of improving the quality of teacher preparation programs in general (Coleman et al., 2020; Magnuson & Waldfogel, 2005; Reed, 2012). More specifically, special education teacher preparation programs can be strengthened through interdisciplinary program evaluation (Bruder et al., 2019; Dettmer et al., 2009; Friend & Cook, 2010; Stayton, 2015). By working together across ECSE and social work or other related services programs, the interdisciplinary program team can identify shared competencies as recognized by each discipline-specific professional standards, including DEC recommended practices (2020), NAEYC developmentally appropriate practices (2019), and CSWE (2015). It is necessary that these competencies be placed front and center when creating and revising course syllabi.
Implications and Next Steps
Similar to other research using rubrics for reviewing syllabi, the findings of this study suggest that useful data can be obtained through a syllabi review process using a rubric to inform interdisciplinary programs on whether or not their combined coursework is meeting the requirements for professionals’ competencies (Beouy & Boss, 2019; Coleman et al., 2020). This may also provide implications when developing or reviewing other interdisciplinary preparation programs to ensure alignment and coverage across all competency areas. For example, although certain courses in the present syllabi analysis addressed competencies fewer times than others, solid representation was evident across the entire course program sequence.
There are also a number of implications regarding the measurement rubric itself. Three reviewers seem adequate to produce solid recommendations for a program when a specific process is followed for the syllabi review. Reviewers should meet before the process begins to clarify definitions for the rating scale that will be used for scoring. In the case described above, the reviewers modified the initial scale slightly to increase the variability. However, keeping in mind the limited nature of the information that any given syllabus offers, the variability should not be too large as this is a tool for screening rather than for the purpose of a detailed, well-scrutinized evaluation. To increase the variability too much could skew the data that the process of a screening tool is meant to offer. The results of this study also suggest that beginning with a sample syllabus may help the review team come to an agreement with how the scoring should occur. A review of a sample syllabus followed by a robust conversation among reviewers can ensure better inter-rater reliability and robust results.
This project serves as an example of how standards for a program can be evaluated within each course. A rubric such as this could be used in interdisciplinary training programs to rate syllabi before the courses are taught (Goodwin et al., 2018; Richmond et al., 2019; Servizzi, 2015). The implications of this study suggest that in projects specific to interdisciplinary syllabi reviews, a possible way to strengthen the utility of the reviews would be to determine a way to show how the standards are covered across all of the courses in a program. It is lofty to assume that any one course will fully contain all of the competency expectations for a given program overall. It may be useful to find a way that once all of the syllabi are reviewed, the ratings can be used to determine the competency alignment of the program overall. Possibly a similar screening tool could be developed to rate and evaluate teacher preparation programs as a whole.
Additionally, it should be noted that this study is quantitative in nature. Future research that builds on this work could include a qualitative approach. Researchers could facilitate focus groups with faculty across interdisciplinary programs to provide more narrative to support the quantitative data (Bochner, 2018; Maxwell, 2013).
Limitations
There were several limitations to this project that are worth noting. First, each researcher brings their own subjectivity to the process based on their background knowledge, field of expertise, and personal experiences. Therefore, even with the well-worded indicators and a formal rating scale, the review for each syllabus will still be up to each individual reviewers’ interpretation. Additionally, given the limited nature of the information that a syllabus provides (i.e., the title of articles rather than an abstract, short description of assignments), the reviewers are required to give scores based on what they project any given assignment or reading might entail completely. It is also worth noting the point that courses with more learning objectives, readings, and/or assignments will likely score higher than courses with less of the components. Along this same vein, courses with more articles in the readings category of the syllabus may be rated higher than classes with a textbook focus.
Areas to Improve
For this particular project, it is important to draw attention to the disparity of scores between the EI/ECSE course syllabi and the syllabi for the SW program. The language in the EI/ECSE courses is more inherently in line with the language and terminology used on the syllabus evaluation form. One of these reasons for this disparity is likely because the evaluation form was initially developed using the EI/ECSE courses as examples. Although the target competencies were evident in the curriculum of MSW, the language was less explicit in specific course syllabi. Therefore, the social work syllabi in general received lower scores than the syllabi in the EI/ECSE program. This is not a limitation, per se, but important to consider in future syllabi review projects. Perhaps the language of particular programs should be considered before a future review is undertaken. The syllabi evaluation form can be refined by program faculty from interdisciplinary program teams. When this process is done systematically and comprehensively, the evaluation form can be generalized into other interdisciplinary personnel preparation programs in EI/ECSE and related services such as speech and language pathology, physical therapy, and occupational therapy.
Conclusion
This study details how an evidence-based syllabus evaluation form was developed and tested for the purpose of strengthening an interdisciplinary OSEP program. The paper offers information on how to use a systematic syllabus review form as a measure to document and disseminate the quality of a teacher preparation program. This study suggests the importance of consistency when target competencies across programs were reviewed. Findings of this review also indicate that systematic syllabi evaluation can be an effective approach to documenting the quality of an interdisciplinary personnel preparation program.
Compliance with Ethical Standards
Conflict of Interest
The authors declare no competing interests.
Ethical Approval
Human subjects were not used and IRB approval was not required for this study.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Met BSL Psychologie Totaal blijf je als professional steeds op de hoogte van de nieuwste ontwikkelingen binnen jouw vak. Met het online abonnement heb je toegang tot een groot aantal boeken, protocollen, vaktijdschriften en e-learnings op het gebied van psychologie en psychiatrie. Zo kun je op je gemak en wanneer het jou het beste uitkomt verdiepen in jouw vakgebied.
