Design of a three-level evaluation of the Dutch Healthy School program

The conceptual framework distinguishes three levels, namely that of the student, the school, and the region. The school and the region are considered to be so-called complex adaptive systems [7‐15]. This means they are systems made up of many components, which are constantly interacting with each other. These components include the people in the system (the actors) and the factors and characteristics of the system. Changes in one part (e.g., implementation of an HS initiative) lead to changes in other parts or are resisted, in an effort to stay balanced as a whole [8, 12, 14]. For this purpose, there is constant feedback between the components [12]. This process is unpredictable since small efforts may have large effects and vice versa [8, 14]. Therefore, in each system, i.e., in every school or region, there is a unique context [7‐14]. Based on this theoretical vision, (the preliminary version of) the conceptual framework provides a visualization of the expected process of professional support at the regional level, its implementation at the school level, and outcomes at the student level (Fig. 1).

The school is central to the framework and is visualized by the circle in the middle. The lower part represents the implementation of the HS program, as part of the context of the complex adaptive school system. The upper part shows the actors, factors, and characteristics of the school context that are important for implementation of the program. Circular arrows indicate continuous feedback between the two parts to keep the school system in balance [7, 16]. The wavy line in the school system shows that there is no strict boundary between the parts, but that they flow into each other (for example, when an HS activity is organized by a group of teachers). Actors in the school context include teachers, students, parents, management, and care coordinators. Relevant factors may be directly related to these actors (e.g., students’ work ethic or teachers’ pedagogical competencies for the HS program), but also to the school as an organization (e.g., the number of hours available for the HS program and coordination by management) and the sociopolitical environment (laws and regulations regarding HS) [16]. In addition, general school characteristics (e.g., school size) and, if applicable, HS program characteristics (e.g., which topic certificates have been obtained) determine the school context [17].

The PHS region is shown at the top of the framework. It is assumed that the implementation process of the HS program in schools does not only depend on factors in the school context, but also on the professional support schools receive [18‐21]. Top-down input from the national HS program, consisting of financial and practical support, enters the complex adaptive PHS regional system. This reaches the school through an HS adviser from the PHS or directly via the website [22]. The lower part of the regional system shows the level of support that has been offered to schools. The upper part shows the actors, factors, and characteristics of the context. Circular arrows again indicate constant feedback between these two parts to maintain balance in the system. The wavy line shows that the parts overlap (for example, when a PHS professional supports a school). Furthermore, the overlap between the regional and school systems assumes reciprocal influence between the regional support and the school context, represented by circular arrows. Actors in the regional context include the PHS, municipalities, Youth Health Care Services, local sports companies, environmental organizations, and addiction prevention. Four categories of contextual factors were selected based on existing literature [18, 23‐25]. These are factors related to the professional (e.g., attitude toward HS program support), the organization (including available hours for support), the broader context (e.g., shared vision on regional prevention policies), and the collaboration (e.g., method of communication). The regional context is also determined by general characteristics (e.g., the number of collaborative partners) and HS-related characteristics (e.g., the year of adoption of HS program support).

The framework further assumes a relationship between the HS program and student-level outcomes. Indicators of these outcomes are based on the objectives of the program: 1) healthy lifestyle, 2) health and personal development, and 3) educational performance and school absenteeism [26]. This is shown in the bottom section of the framework. It is hypothesized that the HS program is associated with lifestyles related to the topics of the program, including fruit and vegetable consumption, physical activity, substance use, and sexual risk behavior [6, 27]. This might in turn improve general and psychosocial health [28‐30]. Previous research further demonstrates that unhealthy lifestyles and deteriorated health may be associated with poorer educational performance, such as lower grade point averages, increased likelihood of grade repetition, and school absenteeism [2, 31‐33]. Therefore, it is hypothesized that the HS program also leads to improvements of the other indicators via improvements in lifestyle [6, 26, 29, 33]. This is a complex process of reciprocal influence on lifestyle, health, and educational performance that is shaped by the home, the school, and the broader environment [34‐36]. This reciprocal influence is represented by circular arrows between the three indicators. In addition, the impact of the HS program at the student level may vary by school, in part due to differences in HS program implementation [37]. However, even schools that implement the program in similar ways may vary in outcomes, as these may also be influenced by the factors in the school context described earlier [38, 39]. The impact of these possible moderators is visualized by the arrow on the left side of the framework.

To answer the first research question, two substudies will be conducted (Fig. 1 and Tab. 1). For substudy 1.1, retrospective analyses of the development of student outcomes and characteristics of schools and students that contribute to explaining the differences found between schools will be conducted. For this purpose, existing datasets from, among others, participating PHSs and Statistics Netherlands (CBS) that contain indicators regarding lifestyle, health and personal development, and/or educational performance/school absenteeism of students will be used. Usable datasets consist of student-level data from both schools with and without HS program certification and include an indicator to determine which students were in the same school during that period. For primary schools, in particular, digitalized data from the PHSs of ~5- and 10-year-olds from multiple birth cohorts will be used. This mainly involves two types of data: 1) growth data (height and weight), which are used to calculate BMI z‑scores, and 2) outcomes of the Strengths and Difficulties Questionnaire (SDQ), which is used to measure psychosocial problems in children and adolescents [40]. For secondary schools, data from the national Youth Health Monitor for 2015 and 2019, filled out by second and fourth graders, will be used primarily [41]. These include data on health and lifestyle (including dietary intake, physical activity, substance use, and sexual risk behavior), personal development (including SDQ), and school absenteeism. Data on educational performance for primary and secondary schools will be obtained through the Education Executive Agency (DUO) and Statistics Netherlands, among others. For secondary vocational schools, the possibilities are still being investigated.

For substudy 1.1, methods will be applied that allow reasonable statements to be made about associations between the HS program and the intended outcomes. These could include multilevel regression analyses or quasi-experimental methods. The final choice of quantitative analysis strategies will depend on the characteristics of the available data on students and schools. We will first identify the variation across schools for the intended outcomes and then examine to what extent differences can be explained by school characteristics (e.g., school type), school population (e.g., students’ educational performance), and HS program characteristics (e.g., which topic certificates were obtained). To analyze data from different sources as a whole, data harmonization techniques will be used. These are techniques that allow different data sources to be combined prior to statistical analyses [42]. To deal with missing data, both at the individual and school level, multiple imputation will be used [43]. To account for the nested structure of the data, we will use multilevel imputation [44].

Next, for substudy 1.2, school conditions for student outcomes will be examined in more detail. Based on data from substudy 1.1, a total of ~50 schools for primary, secondary, and secondary vocational education that perform relatively better or worse on student outcomes will be selected. Within this selection, matching will be applied based on school population characteristics, such as the educational track in secondary schools, where possible. The follow-up analysis will consist of predominantly qualitative research. Data on school characteristics (e.g., facilities), school population (e.g., teacher attitudes), and HS program characteristics (e.g., the degree of implementation) will be collected through interviews with school staff, parents, and students, document analysis, observations, and—if not yet filled out by the school—the developed HS implementation questionnaire (see substudy 2.1). All data will be analyzed with Qualitative Comparative Analysis (QCA) to explain differences between schools. QCA is a method by which complex relationships between many factors of different groups (in this case, schools) can be examined [45].

To answer the second research question, two substudies will be conducted (Fig. 1 and Tab. 1). Substudy 2.1 will map the degree of HS program implementation in schools. For this purpose, a questionnaire will be developed for school staff. During the development process, experts in school health promotion (n = 10–15) from academia, PHSs, schools, and the HS program will be asked to define the degree of HS implementation in semi-structured interviews. This will be done using seven concepts: 1) adherence to the principles (by HS topic), 2) the dosage of HS program implementation at the school, 3) the quality of delivery, 4) the degree to which actors are involved, 5) the degree to which the HS program contains unique elements to make a difference, 6) the degree of HS program adaptation to fit the school context, and 7) the degree to which the HS program is part of their DNA (the routines, norms and identity) [46‐51]. Using the interview data, a questionnaire is drafted, which is then further developed during an online expert consultation (n = ~40 experts in school health promotion) and a pretest among school staff (n = ~10). The final questionnaire will be distributed to all primary, secondary, and secondary vocational schools in the nine participating PHS regions (n = ~3000) and filled out by the person who is most knowledgeable about the school’s health promotion practices. This results in quantitative data, which can be used to differentiate schools for each of the seven dimensions (concepts).

Substudy 2.2 aims to understand factors in the school context that may influence the degree of HS program implementation. Based on the results of substudy 2.1, ~50 schools will be selected, stratified by the degree of implementation and education sector. Although these schools will be selected based on different criteria than those used in substudy 1.2, the selections may overlap. At these schools, data on factors related to the actors, school organization, and sociopolitical environment will be collected, as well as general school and HS program characteristics (also see “Conceptual Framework” and Fig. 1). Factor selection will be guided by the Measurement Instrument for Determinants of Innovations (MIDI) [52]. Furthermore, the degree of HS program implementation, as found in substudy 2.1, will be verified. Data stemming from a combination of semi-structured (group) interviews, observations, registration systems, and document analysis will be collected. The interviews will be conducted with teachers, principals, the care team, parents, and students. Using the predominantly qualitative data, a descriptive analysis of the school context will be performed.

Next, differences in the degree of HS program implementation (substudy 2.1) will be explained by the school context (substudy 2.2) and regional support (see substudy 3.1 and 3.2). Data consisting of information on explanatory factors will be analyzed using QCA [45]. Data concerning actors’ perceptions of the association between explanatory factors and the degree of HS program implementation will be analyzed descriptively.

To answer the third research question, three substudies will be conducted (Fig. 1 and Tab. 1). Substudy 3.1 provides insight into the regional professional support received by schools. The study sample will be the selection of schools (n = ~50) from substudy 2.2. At the schools, insight into regional support will be obtained through staff interviews by inquiring, for example, about the type and intensity of HS program support received. In substudy 3.2, the offer from the participating PHS regions (n = 9) will be mapped. These PHS regions were selected for participation based on variation in location and willingness to participate. For each region, data from semi-structured (group) interviews, observations, document analysis, and registration systems will be collected. This includes, for example, the hours and financial resources available to provide HS program support. Initial participants are coordinators of HS advisers from the respective regions, after which other relevant stakeholders are identified by the snowball method [53]. In substudy 3.3, information on factors in the regional context related to the stakeholders and their organizations, the broader context and collaboration between the organizations, and general and HS program characteristics will be collected in each region (also see “Conceptual Framework” and Fig. 1). These data will be collected simultaneously with those for substudy 3.2 and entail information from the semi-structured (group) interviews with stakeholders, observations, document analysis, and registration systems.

The predominantly qualitative data from substudies 3.1, 3.2 and 3.3 will first be analyzed descriptively. Next, differences in the level of regional professional support (substudies 3.1 and 3.2) will be explained by the school context (see substudy 2.2) and regional context (substudy 3.3). Data consisting of information about the explanatory factors will be analyzed with QCA [45], whereas data regarding actors’ perceptions of the association between the explanatory factors and regional support will be analyzed descriptively.

In semi-annual meetings of the so-called Community of Practice, researchers and practice professionals exchange information and knowledge. This improves the connection between research and daily practice and increases the knowledge and skills of school health promotion professionals. Participants are (coordinators of) HS advisers and epidemiologists from PHSs in the Netherlands, employees of the national professional support system for the HS program, employees of other relevant organizations in the health and education sectors, and scientists.