Correction
26 Jan 2015: The PLOS ONE Staff (2015) Correction: Comparison of the Web-Based and Digital Questionnaires of the Spanish and Catalan Versions of the KIDSCREEN-52. PLOS ONE 10(1): e0116619. https://doi.org/10.1371/journal.pone.0116619 View correction
Figures
Abstract
Background
The objectives of the study were to develop web-based Spanish and Catalan versions of the KIDSCREEN, and to compare scores and psychometric properties with the paper version.
Methods
Internet and paper Spanish and Catalan versions of the KIDSCREEN-52 were included in a cross-sectional study in school-age children. Web-based and paper Spanish or Catalan versions of the KIDSCREEN-52 were administered to students aged 8 to 18 years from primary and secondary schools in Palafolls (Barcelona, Spain, n = 923). All students completed both web-based and paper versions during school time with an interval of at least 2 hours between administrations. The order of administration was randomized. The KIDSCREEN-52, the Strengths and Difficulties Questionnaire (SDQ), and sociodemographic variables were collected. Missing values, floor and ceiling effects, and internal consistency were compared between both versions, as well as mean score differences, level of agreement, and known groups and construct validity.
Results
Participation rate was 77% (n = 715). Web-based and paper versions showed low percentage of missing values and similar high ceiling effect (range 0 to 44%). Mean score differences showed an effect size (ES) lower than 0.2 in all dimensions. Internal consistency ranged from 0.7 to 0.88, and degree of agreement was excellent (Intraclass correlation coefficient [ICC] range 0.75 to 0.87). Expected differences were seen by sex, age, socioeconomic status and mental health status.
Citation: Rajmil L, Robles N, Rodriguez-Arjona D, Azuara M, Codina F, Raat H, et al. (2014) Comparison of the Web-Based and Digital Questionnaires of the Spanish and Catalan Versions of the KIDSCREEN-52. PLoS ONE 9(12): e114527. https://doi.org/10.1371/journal.pone.0114527
Editor: Valsamma Eapen, University of New South Wales, Australia
Received: July 4, 2014; Accepted: October 23, 2014; Published: December 5, 2014
Copyright: © 2014 Rajmil et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. Data are from the eKIDSCREEN version study whose authors may be contacted at email of corresponding author: lrajmil@gencat.cat
Funding: The study was financed by the Spanish Ministry of Health (contract number PI12/01296). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The use of information and communication technology (ICT) in healthcare services seems to be one of the most promising current areas of research. Innovation in the area of health in recent years is concentrated largely in ICTs [1]. Children and adolescents are population groups who could most benefit from the use of ICTs. Administration of standardized paper questionnaires of health-related quality of life (HRQOL) instruments addressed to children would be probably increasingly replaced by digital versions, specifically designed for children and teenagers [2]. This requires the development of questionnaires via web and the comparison to the original paper versions.
From the perspective of clinical practice and research the use of Internet avoids printing paper, data entry is done online, and further procedures may be employed to reduce the number of missing responses. In general, the mode of administration of the questionnaire can affect participation rate, the number of missing responses, psychometric properties, and scores [3]. Some studies showed minor differences between computer and paper versions of the same questionnaires [4], [5]. When the same questionnaire on paper and computer were compared and the contents were sensitive and/or committed such as drug use or sexual behavior, it was observed that administration via computer increased coverage because it was perceived as providing more intimacy than the paper format [2].
The KIDSCREEN is a HRQOL instrument addressed to children 8 to 18 years old developed in Europe and used worldwide [6]. Although several studies used electronic administration of the instrument, a few studies to date have been published on the psychometric properties of this version. A digital version adapted to deaf children has been developed [7]. An electronic version of the KIDSCREEN-27 has demonstrated acceptable psychometric properties without face to face comparison with the paper version [8]. However, it is widely accepted that measurement properties of any new administration form should be compared with that of the original version [9]. The objectives of the present study were to develop web-based Spanish and Catalan versions of KIDSCREEN-52 (eKIDSCREEN), and to compare scores and psychometric properties with the paper version.
Methods
From the conventional paper format, using the same wording of the items and instructions, an internet version of the questionnaire was developed through a generic internet tool using Ruby on Rails applications and MySQL database (http://rubyonrails.org). The design of the web version was done trying to present a visual format to participants as similar as possible to the paper version. The internet version did not allow the respondent to select more than one answer to each item and it checked the questionnaire for missing answers before the respondent could “logout”. A pilot test was carry out including approximately 20 simulations to detect inconsistencies, errors and problems with the use of the digital version.
Sample selection and procedures
All students were selected from 3rd to 6th course of Primary education (approximately 8 to 11 years old), 1st through 4th grade of Secondary education (12–16y), and High School (17–18y) of all 3 schools in Palafolls, a village from Barcelona province with approximately 9000 inhabitants (n = 923). All students completed both paper and digital versions during the same day with a minimum of 2 hours apart, following 1 hour class and also distracting activities during the interval between administrations. Both versions were administered in individual random order. Given the bilingual characteristics of the study population also the language (Spanish or Catalan) was individually randomized, and each student answered the web and paper version of the same language. The fieldwork was carried out between October and November 2013.
Ethics statement
All students whose parents signed consent to participate and they voluntarily agreed to participate were included in the study. All procedures were carried out following the data protection requirements of the European Parliament (Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data). The ethical and legal requirements in Spain were also adhered to, and the protocol was approved by the Clinical Research Ethical Committee of the Hospital del Mar, Barcelona.
Measures
The KIDSCREEN-52 is a self-reported, generic measure of HRQOL for use in children and adolescents [6]. The KIDSCREEN-52 measures HRQOL in ten dimensions: Physical Well-being (PH, five items); Psychological Wellbeing (PW, six items); Moods and Emotions (ME, seven items); Self-perception (SP, five items); Autonomy (AU, five items); Parent Relation and Home Life (PA, six items); Social Support and Peers (PE, six items); School Environment (SC, six items); Social Acceptance (bullying) (BU, three items); Financial Resources (FI, three items). The KIDSCREEN-52 item uses five-point Likert-type scales to assess either the frequency (never, seldom, sometimes, often, always) or intensity (not at all, slightly, moderately, very, extremely). The recall period is 1 week. Scores for each dimension are calculated using Rasch analysis and then transformed into T-values with a mean of 50 and a standard deviation (SD) of 10. Higher scores indicate better HRQOL. KIDSCREEN-52 T scores refer to the mean values and SD from a representative sample of the European general population [10]. The Spanish and Catalan paper versions of the KIDSCREEN-52 showed acceptable levels of reliability as well as construct and convergent validity. Cronbach's alpha values ranged from 0.74 to 0.86 for the different dimensions, and intraclass correlation coefficients ranged from 0.55 to 0.79. It fulfills requirements of the Rasch model in showing no differential item functioning by age group or sex [11]. As the KIDSCREEN-52 was only designed to give dimension scores, not an overall score, we also calculated KIDSCREEN-10 scores. The KIDSCREEN-10 was specifically designed to be used as an index score and meets all of the necessary criteria, including unidimensionality, good internal consistency (0.82), and satisfactory construct, concurrent and discriminant validity [12].
Other variables collected in the present study included age, sex, family socio-economic status, family type, and parental level of education. Socio-economic status was measured using the Family Affluence Scale (FAS) [13], which includes family car ownership, having their own unshared room, the number of computers at home, and how many times they spent on holidays in the past 12 months. FAS scores were categorized as low (0–3), intermediate (4–5), and high (6–7) affluence level. Sociodemographic information collected from parents included the highest family level of education categorized as primary education, secondary education, and university degree.
Children's mental health status was assessed using the Strengths and Difficulties Questionnaire (SDQ). The SDQ is a brief behavioural screening questionnaire for children and adolescents that asks about their mental health symptoms and positive attitudes [14]. The instrument consists of 25 items measuring 5 dimensions. Higher scores indicate more problems except on the pro-social behaviour dimension. Items in the 4 problem dimensions are summed to give a total difficulties score ranging from 0 (no problems) – 40 (maximum problems). The Spanish version has been shown to be reliable and valid [15].
Statistical analysis
Mean HRQOL scores were compared between web and paper versions using paired T Test and standardized mean differences were computed (effect size, ES) to evaluate the degree of differences. ES lower than 0.2 was considered as absence of differences, 0.21 to 0.5 as low, 0.51 to 0.8 as moderate and >0.8 as high [16]. It was expected that the Spanish and Catalan eKIDSCREEN would present similar scores to the paper version with no differences greater than 0.2 standard deviation (SD). Missing values, and floor and ceiling effect were estimated expecting less than 15% in all dimensions. The internal consistency of the KIDSCREEN-52 dimensions was calculated using Cronbach's alpha [17]. Alpha coefficients of 0.7 or higher were considered acceptable.
Agreement was analyzed through Intraclass correlation coefficients (ICC) [18], Bland and Altman plots, and 95% Confidence Interval (95% CI) was also calculated for the upper and lower limits of agreement [19]. An ICC lower than 0.4 was considered as very low, 0.4 to 0.74 as low to acceptable, and 0.75 or higher as excellent [20]. It was expected that the psychometric properties of the digital version would be similar to the paper version: it was expected similar floor and ceiling effects, all dimensions would show an internal consistency >0.7, agreement with the paper version would be acceptable (ICC at least 0.4 to 0.74). Additionally, mean score comparison was also carried out according to the language and order of administration.
Construct validity was assessed analyzing the KIDSCREEN-52 dimension scores following the known groups approach according to age, gender, and the SDQ total difficulties stratified by unlikely vs probable/possible case. A priori hypotheses about the relative magnitude of the differences were specified according to previous studies [6], [10], [11], and ES was calculated to estimate the magnitude of differences comparing web and paper versions. It was expected that the web version would present similar results in terms of construct validity than the paper version (according to age, sex, socioeconomic status, and mental health status).
All statistical analyses were repeated for the total sample and stratified by order of administration and language.
Results
Minor changes have been made to the internet version after the pilot test. Participation rate was 77% (n = 715). 54% of the sample were girls, mean age was 11.7y; 23% came from families with university degree, and 30% were in the low FAS (table 1).
Mean score comparisons between web and paper versions of the KIDSCREEN-52 and the KIDSCREEN-10 Index showed no differences, except for the dimension of Peers and social support (p = 0.04) (table 2). ES was lower than 0.2 in all dimensions. Percentage of missing values was slightly lower in the web version compared to the paper version (web version n = 2; paper version ranged from n = 7 in BU to n = 30 in the KS-10 Index). Ceiling effect was higher than 15% in 9 out of ten dimensions, and similar in both web and paper versions. The KIDSCREEN-10 Index showed low percentage of floor effect and no ceiling effect in both versions. Internal consistency in the web version (Cronbach's alpha ranged 0.7 to 0.88) was also similar to the paper version.
The degree of agreement of web and paper versions was excellent (ICC range = 0.75 to 0.87) (table 3). Lower and upper limits of agreement showed relatively high variability.
The level of agreement varied according to the HRQOL level (Figure 1). Agreement was better for worst values in almost all dimensions and the KIDSCREEN-10 Index.
Table 4 shows score comparisons according to the language version (Catalan or Spanish) and the order of administration (web or paper version first). ES lower than 0.2 were seen in case of language comparisons and minimal differences were found on Physical well-being (ES = 0.2) and Moods and Emotion (ES = 0.25) when the web version was administered after the paper version. Internal consistency was also similar comparing both language versions and the order of administration (data not shown).
Results from the web version showed that boys scored higher than girls with statistically significant differences in all dimensions (p<0.01) except on SC, BU and FI; younger children scored better than older adolescents in all dimensions except on BU and FI; statistically significant differences were also found according to the FAS with worse scores in all dimensions in those children reporting low family affluence. Similar results were found with the paper version (data not shown and available from authors upon request). Table 5 shows KIDSCREEN scores according to mental health status. Probable/possible cases show lower (worse) scores in all dimensions, with higher ES in the dimensions of Moods and Emotions (ES = 0.98 in both versions) and Psychological well-being (ES = 0.78 and 0.86 for web and paper versions, respectively).
No differences were found stratifying the sample according to the language. Slightly higher (better) score, although without statistically significant differences, were seen when web version was administrated after the paper version (data not shown and available from authors upon request).
Discussion
The present study demonstrates that both web and paper versions showed comparable scale scores, and psychometric properties, and also an excellent level of agreement. The results of the study support that web-based Spanish version of the KIDSCREEN-52 is feasible, and as reliable and valid as the paper version.
Some studies have demonstrated that online health questionnaires are feasible especially among adolescents [21], and showed similar scores and psychometric properties compared to the paper version [22]. A computer-based questionnaire offers a number of advantages and limitations comparing with paper-and-pencil versions. Missing values or incomplete data can be reduced by requiring completion of an item before the individual can move on to the next question, and out-of-range values can be eliminated. Personnel time can be saved by reducing the amount of time spent entering data and handling paper, which also increases the accuracy of data by reducing typing errors. In addition, computer software scores the patient responses immediately and creates summary information. A meta-analytic review concluded that extensive evidence indicates that paper and computer versions of self-reported questionnaires are equivalent [3]. Nevertheless, some differences between the web and paper version exist. In the paper version all items are visible all the time, while usually only one item or a set of items is visible at a time in the computer. When completing the paper version, one can create an overall impression of the questionnaire before choosing the appropriate option for each item. In addition, the paper version allows the opportunity to change/correct one of the responses if the subject so chooses. In the computer version, it is not possible to go back to change any previous answers. On the other hand, the subject must respond to each set of questions as the software used in the study did not allow the subject to move on without answering each set of questions.
The potential usefulness of measures of child HRQOL using ICTs are in the evaluation of the effectiveness of health interventions, and in daily clinical practice. One barrier to implement these measures in clinical practice is to find time and space for patients to answer the questionnaires before the consultation, the subsequent entry into database, and the need for statistical software to implement algorithms complex scoring. Instead digital versions are a good way to solve most of these barriers.
In the present study the psychometric properties were similar between the two modes of administration. Both versions showed a similar high ceiling effect and acceptable internal consistency. Both versions also showed a similar ability to discriminate between known groups and according to mental health status. One of the strengths of the study was that all participants answered both versions and also that both versions were randomly administered.
Some limitations of the study deserve comments. First, the sample selected in the present study to compare the web-based and paper versions included the whole school population of the village of Palafolls that is not necessarily representative of the Spanish and Catalan childhood population. Nevertheless, in this type of studies it is not necessary for the samples to be representative, but they should include a wide range of possible answers. This ensures and facilitates comparisons within the sample. Secondly, the fact that both versions have been administered on a single day may have caused some recall. Nevertheless, no clear evidence exists on the ideal time span for gathering test-retest reliability. According to some authors it could be 2 weeks [23]. A study carried out in adult population on test-retest reliability of health status instruments using 2-day or 2-week time frame between administrations did not affect the results of the reliability testing [24]. Another study in children using similar strategy than the present work, administering the questionnaires the same day with distracting activities before starting the second mode showed acceptable results [25]. A certain retest effect has been described with improvements in the second administration independently of the time between measures [26]. In the present study the order of completing web and paper versions was randomized and scores were slightly better when the web version was administered after the paper version. Nevertheless, these differences were minimal in terms of ES and only in two dimensions of the KIDSCREEN (ES = 0.2 and 0.25 respectively). These results reinforce the idea that, although there was some memory effect, this effect did not show an important impact on the level of agreement between the web and paper versions. Future studies could add more information on the appropriate time-interval for scores comparisons by administering the two versions within two hours at a later time-point (i.e. two weeks or two months apart). A matter of future studies would also be the use of techniques such as the item response theory (IRT) [27]. IRT might be useful to identify potential predictors of discrepancies at item level, i.e. items functioning differently for respondents from different languages or according to the order of administration”. Thirdly, the statistical methods used in this study have strengths and weaknesses. The ICC combines information about bias and association as it reflects both the degree of correspondence and agreement between ratings, takes account of the actual magnitude of the score and is sensitive to systematic bias in data. One disadvantage is that ICC is strongly influenced by the variance. The Bland and Altman method has advantages compared with ICC, such as easily-detected visual representation of the degree of agreement, easy identification of bias outliers, and easy detection of any relationship between variability of measure with the size of mean. The major disadvantages of Bland and Altman method are the complexity of interpretation compared with a single reliability index, and the need for a sample set over 50 subjects to avoid very wide 95%CI for limits of agreement. Finally, the average time to complete the web version was 9.15 minutes, and no comparable data were possible to collect with the paper version.
A promising area of research in which KIDSCREEN electronic is contributing to is in the development of the KIDS-CAT [28], a computer-adaptive test that is in the process of development in German speaking countries. They apply analogous system to the methods used by the US-wide patient-reported outcome initiative Patient Reported Outcomes Measurement Information System (PROMIS) [29]. The KIDS-CAT content is based on the KIDSCREEN-27 domain structure, and item banks including all KIDSCREEN items plus items used in other established pediatric health instruments. The development of the Spanish eKIDSCREEN allows to establish some comparisons with these instruments.
In summary, the results show that the eKIDSCREEN is equally reliable and valid than the paper version to be administered in children and adolescents. It will incorporate the child population in the assessment of HRQOL using a medium in which they are handled flexibly and autonomously, providing a more attractive format, and easier to manage and analyze than the paper versions.
Author Contributions
Conceived and designed the experiments: LR NR DRA HR URS. Performed the experiments: LR NR DRA MA FC. Analyzed the data: LR NR DRA. Contributed reagents/materials/analysis tools: LR NR HR URS. Wrote the paper: LR NR DRA MA FC HR URS.
References
- 1. Schreier G (2014) The internet of things for personalized health. Stud Health Technol Inform 200:22–31.
- 2. Haverman L, Engelen V, van Rossum MAJ, Heymans HSA, Grootenhuis MA (2011) Monitoring health-related quality of life in paediatric practice: development of an innovative web-based application. BMC Pediatrics 11:3.
- 3. Gwaltney CJ, Shields AL, Shiffman S (2008) Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 11:322–333.
- 4. Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL (1998) Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science 280:867–873.
- 5. Mangunkusumo RT, Duisterhout JS, de Graaff N, Maarsingh EJ, de Koning MJ, et al. (2006) Internet Versus Paper Mode of Health and Health Behavior Questionnaires in Elementary school: asthma and fruits as examples. J School Health 76:80–86.
- 6. Ravens-Sieberer U, Gosch A, Rajmil L, Erhart M, Bruil J, et al. (2008) The KIDSCREEN-52 Quality of Life Measure for Children andAdolescents: Psychometric Results from a Cross-Cultural Survey in 13 European Countries. Value Health 11:645–658.
- 7. Pardo-Guijarro MJ, Woll B, Moya-Martínez P, Martínez-Andrés M, Cortés-Ramírez EE, et al. (2013) Validity and reliability of the Spanish sign language version of the KIDSCREEN-27 health-related quality of life questionnaire for use in deaf children and adolescents. Gac Sanit 27:318–324.
- 8. Lloyd K (2011) Kids' Life and Times: using an Internet survey to measure children's health-related quality of life. Qual Life Res 20:37–44.
- 9. Scientific Advisory Committee of the Medical Outcome trust (2002) Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 11:193–205.
- 10.
The KIDSCREEN Group Europe (2006) The KIDSCREEN questionnaires. Handbook. Lengerich, Germany: Pabst Science Publishers.
- 11. Tebé C, Berra S, Herdman M, Aymerich M, Alonso J, et al. (2008) Fiabilidad y validez de la versión española del KIDSCREEN-52 para población infantil y adolescente. Med Clin (Barc) 130:650–654.
- 12. Ravens-Sieberer U, Erhart M, Rajmil L, Herdman M, Auquier P, et al. (2010) Reliability, construct and criterion validity of the KIDSCREEN-10 score: a short measure for children and adolescents' well-being and health-related quality of life. Qual Life Res 19:1487–1500.
- 13. Currie C, Molcho M, Boyce W, Holstein B, Torsheim T, et al. (2008) Researching health inequalities in adolescents: The development of the Health Behaviour in School-Aged Children (HBSC) family affluence scale. Soc Sci Med 66:1429e–1436e.
- 14. Goodman R (1997) The strengths and difficulties questionnaire: A research note. J Child Psychol Psychiatry 38:581–586.
- 15.
Pardo Cladellas Y (2007) El desenvolupament de la personalitat psicopàtica: pràctiques educatives parentals i altres variables familiars (Thesis). Barcelona: Universitat Autònoma de Barcelona.
- 16.
Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
- 17. Cronbach LJ (1951) Coefficient alpha and the internal structure of test. Psychometrika, 16, 297–334.
- 18. Prieto L, Lamarca R, Casado A (1998) La evaluación de la fiabilidad en las observaciones clínicas: el coeficiente de correlación intraclase. Med Clin (Barc) 110:142–145.
- 19. Bland JM, Altman DG (1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 8476:307–310.
- 20.
Fleiss JL (1986) The Design and Analysis of Clinical Experiments. New York: John Wiley & Sons, Inc.
- 21. Mangunkusomo RT, Moorman PW, Van Den Berg-de Ruiter AE, Van Der Lei J, De Koning HJ, et al. (2005) Internet-administered adolescent health questionaire compared with a paper versión in a randomized study. J Adolesc Health 36:70.e1–70.e6.
- 22. Raat H, Mangunkusomo RT, Landgraft J, Kloek G, Brug J (2007) Feasibility, reliability, and validity of the adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet administration compared with the standard paper version. Qual Life Res 16:675–685.
- 23.
Nunnally J, Bernstein I (1994) Psychometric theory. 3rd Edition. New York: McGraw-Hill.
- 24. Marx RG, Menezes A, Horovitz L, Jones EC, Warren RF (2003) A comparison of two time intervals for test-retest reliability of health status instruments. J Clin Epidemiol 56:730–735.
- 25. Mangunkusumo RT, Duisterhout JS, de Graaff N, Maarsingh EJ, de Koning HJ, et al. (2006) Internet versus paper mode of health and health behavior questionnaires in elementary schools: asthma and fruit as examples. J School Health 76:80–86.
- 26. Arrindell WA (2001) Changes in waiting-list patients over time: data on some commonly-used measures. Beware! Behavior Res Therapy, 39, 1227–1247.
- 27.
Embretson SE, Reise SP (2000) Item Response Theory for Psychologists. Mahwah, NJ: Erlbaum.
- 28. Ravens-Sieberer U, Herdman M, Devine J, Otto C, Bullinger M, et al. (2014) The European KIDSCREEN approach to measure quality of life and well-being in children: development, current application, and future advances. Qual Life Res 23:791–803.
- 29. Cella D, Riley W, Stone A, Rothrock N, Reeve B, et al. (2010) The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol 63:1179–1194.