Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet administration compared with the standard paper version

In 2003, 1,071 students in 55 classes of various educational levels in the 3rd year of seven secondary schools (13–7 years old) in the area of Vlaardingen (metropolitan area) and Harderwijk (rural area), The Netherlands, were invited to complete the Child Health Questionnaire Child Form (CHQ-CF). The parents and students each received written information about the study several weeks before data collection; parents could refuse their child’s participation, and participation by the students was voluntary.

The CHQ-CF consists of 87 items with 4, 5, or 6 response options divided over 10 multi-item scales and two single-item scales (Table 1) [3]. To reduce respondent burden, the item “change-in-health–was not fielded in this study, and the CHQ-CF scales “role functioning-emotional–and “role functioning-behavioral–were combined into a single scale. The combination of the two role functioning scales is a departure from the CHQ-CF instructions that makes the test analogous to the parent form of the CHQ in this regard [3]. For each scale, items were summed up (some recoded/recalibrated) and transformed into a 0 (worst possible score) to 100 (best possible score) scale [3]. Items on standard socio-demographic variables and the prevalence of seven chronic conditions were included in the questionnaire. From the conventional paper format, using the same wording of the items and instructions, an internet version of the questionnaire was developed through a generic internet tool using PHP (4.0.1), MySQL (3.22), and JavaScript (1.3) [27]. The internet version of the questionnaire listed the items of each CHQ-CF scale on a separate web-page. The internet version did not allow the respondent to select more than one answer to each item of the CHQ-CF and it checked the questionnaire for missing answers before the respondent could “logout– If one or more of the items were not answered, the internet version prompted the respondent to go back to complete those items; but, if the user failed to “logout–properly, missing answers would remain.

Within each school class, students were randomly assigned to either the paper or the internet mode of administration using SPSS-generated random numbers. Students completed the questionnaires, either on paper or online in a classroom with computers linked to the internet, under the supervision of a research assistant; the students were allowed adequate privacy.

Preparatory secondary vocational education was labelled as “lower secondary education”; secondary schools that prepare students for higher professional training as "intermediate secondary education", and university preparatory secondary education as "higher secondary education". Differences between the characteristics of the participants allocated to the paper versus the internet versions of the questionnaires were tested by Student’s t-test and the χ² test. We assessed the frequency of missing answers to CHQ-CF items; the difference in the number of missing answers between the two formats was assessed by the Mann-Whitney U test. We assessed the distributions of the CHQ-CF scale scores to evaluate floor and ceiling effects (≥25% of the respondents having the lowest/highest score) for both modes of administration. Differences between CHQ-CF scale scores by format in the total sample were assessed by Mann-Whitney U tests. Additionally, after transforming the scale scores into ranks, ANOVA was applied to test whether the mode of questionnaire administration interacted with the variables gender (male n = 432; female n = 501) and age (13–4 year olds, n = 399; 15–7 year olds, n = 534). Cohen’s effect sizes, defined as d = [Mean(a) Mean(b)]/SD, where the denominator was the square root of [(n _a -1)SD _a ² + (n _b -1)SD _b ² ] / [(n _a -1) + (n _b -1)], were applied to indicate the relative magnitude of score differences between modes of administration. Here, the letters "a" and "b" refer to the subgroups administered the paper and internet forms of the test, respectively [28]. Following Cohen’s suggested guidelines, 0.20 ≤d < 0.50 indicated a “small effect– 0.50 ≤d < 0.80 a “medium effect– and d ≥0.80 a “large effect–[28]; Norman et al. have suggested that, in general, d = 0.50 can be considered as threshold for a “minimally important difference–(MID) [29]. Cronbach’s α was applied to evaluate the internal consistency reliability of CHQ-CF multi-item scales by format; α of 0.70 or higher was considered to indicate sufficient internal consistency reliability [30]. We applied statistical tests of the hypothesis that the Cronbach’s α reliability coefficients of CHQ multi-item scales in the sample administered the test on paper were equal to those administered the test online [31]. We applied item-level discriminant tests to evaluate whether the CHQ-CF items represent separate domains. For each mode of questionnaire administration, we evaluated whether (on average) correlation coefficients (Pearson-r correlation coefficients) between the items and their own scale score (without the item under consideration) were higher than the correlation coefficients between these items and any other scale. The average Pearson-r correlation coefficients were calculated by applying Fisher’s z transformations [32]; we tested whether the differences between the average Pearson-r correlation coefficients in the subgroup administered the paper form and in the subgroup administered the test online were statistically significant [33]. We assessed the CHQ-CF’s ability to discriminate between subgroups with 0, 1 or 2, and 3 or more chronic conditions, after having transformed the CHQ-CF scale scores into ranks, by ANOVA with the independent variables “number of chronic conditions– "mode of questionnaire administration– and the interaction term “number of chronic conditions–“mode of questionnaire administration– Cohen’s effect sizes d = [Mean(a) Mean(b)]/SD in the condition subgroup were calculated for 1 or 2 versus 0 conditions, and for ≥3 versus 0 conditions. The designations "a" and "b" refer to the subgroups without chronic conditions and those with chronic conditions, respectively [28].

All analyses were done using SPSS, Version 11.0.1. The medical ethical committee of the Erasmus MC-University Medical Center Rotterdam, approved the study.

The participation rate was high. Study group characteristics (age, gender, country of birth, and educational levels) were representative of those of the general population of Dutch adolescents [34]. Randomization to either the paper or internet mode of administration of the CHQ-CF was successful with respect to the evaluated characteristics.

We applied a randomized parallel group design that allows for the comparison of indicators of feasibility, reliability, and validity at the group level between a subgroup that completed a paper version and a subgroup that completed an internet version of the CHQ-CF. However, this did not allow an evaluation of whether the same person would provide equivalent or different answers to the same CHQ-CF questionnaire administered by the alternative mode, which would require a randomized crossover design [25, 35]. Such an evaluation at the individual level requires the respondent to forget all previously provided answers at the second assessment, e.g., by waiting 1 or 2 weeks between the two measurements. It also requires that there is no effect from having previously completed a CHQ-CF questionnaire by any mode, paper or internet, at the second assessment, and that scores by the same mode of administration after a relatively short interval, in the absence of changes in health status, are exactly the same. However, in an evaluation of retesting with the same paper version of the CHQ-CF after 2 weeks, 5 out of 10 CHQ-CF scales showed statistically significant, higher scores at the second measurement with Cohen’s effect sizes ranging from 0.25 to 0.40, while intraclass correlation coefficients between the first and second measurement ranged from 0.06 thru 0.84 [7]. Furthermore, in a randomized crossover design, “carry-over–effects may be present, i.e., completing an internet version before a paper version may have a different effect on the second assessment, than does completing a paper version before an internet version [35]. Despite the logistical and the above-mentioned methodological challenges, we recommend future studies comparing the paper and internet versions of the CHQ-CF applying a randomized crossover design to evaluate congruency of answers to CHQ-CF items at the individual level.

In this study, internet and paper questionnaires were completed in a controlled environment with adequate privacy and supervision. This may not be the case during future applications. We are unaware of the impact less privacy during completion of the questionnaires may have, but this would apply to both the paper and the internet versions of the questionnaire.

For both modes of questionnaire administration, we did not evaluate correlations between CHQ-CF scores and a relevant parent-rated questionnaire such as the CHQ-PF50 [2, 3]. Test-retest reliability of the CHQ-CF and its responsiveness and sensitivity to changes in health were not evaluated in the current study. The CHQ-CF has a relatively large number of items; therefore, we recommend developing a shorter version in the future.

The psychometric properties, with only a few exceptions, were equal between the two modes of questionnaire administration. The Cronbach’s α of the scale “physical functioning–in the subgroup administered the paper version of the questionnaire was just under 0.70, and the difference with the alpha in the subgroup administered the internet version was not statistically significant.

Compared with the paper version, the internet version was successful in reducing the quantity of missing data. Theoretically, differences in selective partial non-response between formats might have contributed to differences in scores that were reported in this study. In our study, in the subgroup (n = 86) that had at least one missing answer to a paper CHQ-CF item, all scale means were somewhat lower than in the subgroup (n = 389) with no missing answers, but these differences were not significant (P ≥0.05). Thus, missing answers are not a source of the observed score differences.

Recently, Ritter et al. found no statistically significant score differences between internet and paper modes of administration for 16 health-related measures, but the study was conducted in an opportunity sample retrieved from the internet, which limits its generalizability [23]. In a randomized internet-paper comparison among adolescents concerning various health measures other than the CHQ-CF, only one statistically significant score difference was reported among 21 topics [24]. In another randomized adolescent study, a medical consumption index and 11 indicators of fruit consumption and determinants of fruit consumption did not show statistically significant score differences between internet and paper administration of the questionnaire, except for one measure that showed small score differences between modes of administration [25]. The International Study of Asthma and Allergies in Childhood (ISAAC) questionnaire did not show statistically significant score differences between internet and paper administration in two randomized adolescent studies [25, 26].

In our study, in the whole sample, the paper version resulted in slightly, yet statistically significant, higher scores for 4 of 10 CHQ-CF scales compared with the internet version. One plausible explanation is chance, since it should be considered that given multiple comparisons, there is a 1-in-20 chance of a false association for each comparison (Type I error at α = 0.05) [36]. A commonly used Bonferroni correction for 10 comparisons would imply an adjusted α = 0.05/10 = 0.005 [36]; at α = 0.005, only one score difference (regarding the scale “general behavior– was significant. Furthermore, given Cohen’s suggested guidelines for the interpretation of effect sizes, three of the four statistically significant differences between modes of administration can be considered as negligible (d ≤0.12), and one difference regarding the CHQ-scale “general behavior–(d = 0.21) can be considered as small [28]; all effect sizes were far below d = 0.50 that was suggested as an approximate threshold for “minimally important differences–by Norman et al. [29]. This study provides no explanations for the established small score differences between paper and internet administration, or for the established statistically significant, but small interaction effects of administration mode with age in the case of four CHQ-CF scales.