Equivalence of paper and pencil vs Internet forms of the ZKPQ-50-CC in Spanish and French samples

Abstract

This study was designed to check for the equivalence of the ZKPQ-50-CC (Spanish and French versions) through Internet on-line (OL) and paper and pencil (PP) answer format. Differences in means and deviations were significant in some scales, but effect sizes are minimal except for Sociability in the Spanish sample. Alpha reliabilities are also very similar in both versions with no significant differences between formats. A robust factorial structure was found for the two formats and the average congruency coefficients were 0.98. The goodness-of-fit indexes obtained by confirmatory factorial analysis are very similar to those obtained in the ZKPQ-50-CC validation study and they do not differ between the two formats. The multi-group analysis confirms the equivalence among the OL–PP formats in both countries. These results in general support the validity and reliability of the Internet as a method in investigations using the ZKPQ-50-CC.

Introduction

From the mid 1990s onwards, researchers in psychology have given personality tests on the Internet, using the items of traditional paper and pencil administered inventories. There is currently a large scientific literature on-line versus paper and pencil answer formats in psychological tests (Buchanan, 2000, Buchanan, 2001, Buchanan, 2002, Buchanan et al., 2005, Buchanan and Smith, 1999, Hertel et al., 2002, Vazire, 2006) The advantages of the on-line-administration have been generally deemed as being of low cost and access to a greater number of subjects in much less time. In addition, data entry requirements are largely eliminated, and the potential for errors in data entry is largely reduced (Barak and English, 2002, Buchanan, 2006).

Most of the authors conclude that the on-line formats of personality questionnaires have good equivalence and similar psychometric properties to the traditional paper and pencil forms (Chuah et al., 2006, Davis, 1999, Gosling et al., 2004). However, equivalence between test formats needs to be established (American Educational Research Association, 1999). As noted by Chuah et al. (2006), it cannot be assumed that an Internet form has a similar validity and reliability as the paper and pencil form and its psychometric properties must be compared to those of the paper-and-pencil format. If there are mean differences in scales it is important to have norms for each format.

Psychologists have previously compared the equivalence of the two types of formats (Ferrando & Lorenzo-Seva, 2005), that have been approached from statistics relying on classical test theory (CTT), or from statistics relying on item response theory (IRT), although most works have been based on the former. Besides, CTT is based in the comparison of the following statistics in the two formats PP–OL: (a) the means and standard deviations, (b) scale internal consistency, and (c) factor structure with exploratory factor analysis or structural equation modeling (Buchanan and Smith, 1999, Steinberg et al., 2000). Mean differences, reliabilities and factor analysis can support the equivalence of Internet and paper-and-pencil administrations of personality tests (including ITR) (Chuah et al., 2006).

The Zuckerman–Kuhlman Personality Questionnaire (ZKPQ; Zuckerman, 2002, Zuckerman et al., 1993) was derived empirically through exploratory factor analysis of a number of personality questionnaires (Zuckerman et al., 1988, Zuckerman et al., 1991). The ZKPQ is based on a five-factor psychobiological model (Zuckerman, 2005). Aluja et al. (2006) developed a short version of the ZKPQ (ZKPQ-50-CC) selecting 10 items per scale: Impulsive Sensation Seeking (ImpSS), Neuroticism–Anxiety (N–Anx), Aggression–Hostility (Agg–Host), Activity (Act), and Sociability (Sy). This short version has equivalent factor structure and reliability in four language versions: English, French, German, and Spanish (Aluja et al., 2006). The ZKPQ-50-CC predicts scores on the Personality Disorders measured by the Millon Clinical Multiaxial Inventory-III. The predictive power was the same as that for the NEO-FFI-R (Aluja, García, Cuevas, & García, 2007). This ZKPQ version is for testing situations where it is necessary to use short protocols. Also, the abbreviated form can be used to obtain data through the Internet. For this reason, it is important to know if the on-line data and paper and pencil format have equivalent characteristics in similar samples.

The aim of this study was to check the equivalence between the paper and pencil and Internet formats of the ZKPQ-50-CC, comparing their psychometric properties. The properties compared between the two formats of administration were: (a) means and standard deviations of scales, (b) distribution measures: Skewness and kurtosis, (c) internal consistency alpha, (d) factorial structure and congruence coefficients inter-format, and (e) confirmatory factorial analysis for each format and multi-group analysis.