Standards for Internet-Based Experimenting
Abstract
This article summarizes expertise gleaned from the first years of Internet-based experimental research and presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples, generalizability, dropout, experimental control, identity checks, multiple submissions, configuration errors, control of motivational confounding, and pre-testing. Several techniques are explained, including “warm-up,” “high hurdle,” password methods, “multiple site entry,” randomization, and the use of incentives. The article concludes by proposing sixteen standards for Internet-based experimenting.
References
(2002). Integration von Wahrscheinlichkeiten: Verarbeitung von zwei Wahrscheinlichkeitsinformationen. [Integration of probabilities: Processing two pieces of probability information]. Unpublished master’s thesis, University of Zurich, Switzerland.
(2002). Can you see the real me? Activation and expression of the “true self” on the Internet. Journal of Social Issues, 58, 33– 48
(2000). SurveyWiz and FactorWiz: JavaScript Web pages that make HTML forms for research on the Internet. Behavior Research Methods, Instruments, and Computers, 32, 339– 346
(2001). A Web-based program of research on decision making. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 23– 55 Lengerich, Germany: Pabst Science.
(2002). Rape myth acceptance and judgments of vulnerability to sexual assault: An Internet experiment. Experimental Psychology, 49, 4 257– 269
(2001). Participation in non-restricted Web surveys: A typology and explanatory model for item non-response. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 193– 208 Lengerich, Germany: Pabst Science.
(2002). Online assessment: Desirable or dangerous?. Professional Psychology: Research and Practice, 33, 148– 154
(2001, October 10). Platform-dependent biases in Online Research: Do Mac users really think different?. In K. J. Jonas, P. Breuer, B. Schauenburg, & M. Boos (Eds.), Perspectives on Internet Research: Concepts and Methods. Retrieved December 27, 2001, from server3.uni-psych.gwdg.de/gor/contrib/buchanan-tom.
(1970). The relevance of laboratory studies to practical situations. In D. P. Schultz (Ed.), The science of psychology: Critical reflections. New York: Appleton Century Crofts.
(1997, June 30). Using the Internet for survey research. Sociological Research Online, 2, Retrieved June 16th, 2002, from www.socresonline.org.uk/2/2/2.html.
(2001). Reaction time measurement by JAVA-applets implementing Internet-based experiments. Behavior Research Methods, Instruments, and Computers, 33, 179– 186
(2002). Measuring differences in preactivation on the Internet: The content category superiority effect. Experimental Psychology, 49, 4 283– 291
(2001). Financial incentives, personal information, and dropout in online studies. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 209– 219 Lengerich, Germany: Pabst Science.
(1996). Proper methodologies for psychological and sociological studies conducted via the Internet. Behavior Research Methods, Instruments, and Computers, 28, 186– 191
(2002). Online longitudinal survey research: Viability and participation. Social Science Computer Review, 20(3), 250– 259
(2001). User-controlled photographic animations, photograph-based questions, and questionnaires: Three instruments for measuring drivers’ risk-taking behavior on the Internet. Behavior Research Methods, Instruments, and Computers, 33, 46– 58
(2001). Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European Journal of Social Psychology, 31, 177– 192
(1986). Response effects in the electronic survey. Public Opinion Quarterly, 50, 402– 413
(2000). On belief bias in syllogistic reasoning. Psychological Review, 107, 852– 884
(2000). Tell me, what did you see? The stimulus on computers. Behavior Research Methods, Instruments, and Computers, 32, 221– 229
(2001). Stimulus delivery on the Web: What can be presented when calibration isn’t possible. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 113– 130 Lengerich, Germany: Pabst Science.
(2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet 35– 60 San Diego, CA: Academic Press.
(2001). A Web experiment on color harmony principles applied to computer user interface design. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 131– 145 Lengerich, Germany: Pabst Science.
(2002). Psychological experimenting on the World Wide Web: Investigating content effects in syllogistic reasoning. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online Social Sciences 181– 212 Göttingen, Germany: Hogrefe & Huber.
(2000). A brief history of Web experimenting. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet 61– 88 San Diego, CA: Academic Press.
(2001). Methodological variables in Web-based research that may affect results: Sample type, monetary incentives, and personal information. Behavior Research Methods, Instruments, and Computers, 33, 226– 233
(1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776– 783
(2002). Hindsight bias around the world. Experimental Psychology, 49(4), 270– 282
(2001). Social influence in computer-mediated communication: The effect of anonymity on group behavior. Personality and Social Psychology Bulletin, 27, 1243– 1254
(1995). The Web experiment method. Retrieved January 6, 2002, from www.genpsy.unizh.ch/Ulf/Lab/WWWExpMethod.html.
(1997). Das psychologische Experimentieren im Internet [Psychological experimenting on the Internet]. In B. Batinic (Ed.), Internet für Psychologen 245– 265 Göttingen, Germany: Hogrefe.
(1999). Online research with children. In U.-D. Reips, B. Batinic, W. Bandilla, M. Bosnjak, L. Gräf, K. Moser, & A. Werner (Eds.). Current Internet science - trends, techniques, results. Aktuelle Online-Forschung - Trends, Techniken, Ergebnisse. Zürich: Online Press. Retrieved April 7, 2002 from dgof.de/tband99/.
(2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet 89– 114 San Diego, CA: Academic Press.
(2001). The Web Experimental Psychology Lab: Five years of data collection on the Internet. Behavior Research Methods, Instruments, and Computers, 33, 201– 211
(2002a). Internet-based psychological experimenting: Five dos and five don’ts. Social Science Computer Review, 20(3), 241– 249
(2002b). Theory and techniques of conducting Web experiments. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online Social Sciences 229– 250 Seattle: Hogrefe & Huber.
(2001). Dimensions of Internet Science. Lengerich, Germany: Pabst Science.
(2001). “Fünfe gerade sein lassen”: Listenkontexteffekte beim Kategorisieren. [“Letting five be equal”: List context effects in categorization]. Unpublished manuscript. Retrieved April 7, 2002 from www.psychologie.unizh.ch/genpsy/reips/papers/re_mo_me2001.pdf.
(2002). WEXTOR: A Web-based tool for generating and visualizing experimental designs and procedures. Behavior Research Methods, Instruments, and Computers, 34, 234– 240
(2001). Differential effects of Ecstasy and cannabis on self-reports of memory ability: A web-based study. Human Psychopharmacology: Clinical and Experimental, 16, 619– 625
(1966). Experimenter effects in behavioral research. New York: Appleton-Century-Crofts.
(1973). The effect of experimenter bias on the performance of the albino rat. Behavioral Science, 8, 183– 189
(1969). Artifact in behavioral research. New York: Academic Press.
(2001). Games as research tools for visual perception over the Internet. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 147– 158 Lengerich, Germany: Pabst.
(1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments, and Computers, 29, 274– 279
(2000). The server-side of psychology Web experiments. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet 285– 310 San Diego, CA: Academic Press.
(1966). Subject selection bias in psychological research. Canadian Psychologist, 7a, 115– 121
(2001). CGI versus JavaScript: A Web experiment on the reversed hindsight bias. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 75– 90 Lengerich, Germany: Pabst Science.
(n.d.) BUGTRAQ Vulnerability Database Statistics. Retrieved April 7, 2002, from www.securityfocus.com.
(2001). Online replication of Evolutionary Psychology evidence: Sex differences in sexual jealousy in imagined scenarios of mate’s sexual versus emotional infidelity. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 91– 112 Lengerich, Germany: Pabst Science.
(2001). Webdesign, Informationssuche und Flow: Nutzerverhalten auf unterschiedlich strukturierten Websites. [Web design, search for information, and flow: User behavior on differently structured Web sites]. Lohmar, Germany: Eul.
