Kelly Caine
ABSTRACT
We describe the primary ways researchers can determine the size of a sample of research participants, present the benefits and drawbacks of each of those methods, and focus on improving one method that could be useful to the CHI community: local standards. To determine local standards for sample size within the CHI community, we conducted an analysis of all manuscripts published at CHI2014. We find that sample size for manuscripts published at CHI ranges from 1 -- 916,000 and the most common sample size is 12. We also find that sample size differs based on factors such as study setting and type of methodology employed. The outcome of this paper is an overview of the various ways sample size may be determined and an analysis of local standards for sample size within the CHI community. These contributions may be useful to researchers planning studies and reviewers evaluating the validity of results.
- ACM SIGCHI 2015. Guide to a Successful Paper or Note Submission.Google Scholar
- ACM SIGCHI 2015. Papers Versus Notes Whats the Difference.Google Scholar
- Herman Aguinis and Erika Harden. 2009. Sample size rules of thumb: evaluating three common practices. In Statistical and methodological myths and urban legends: doctrine, verity and fable in the organizational and social sciences., Charles Lance and Robert Vandenberg Eds., NY, NY, 267--286.Google Scholar
- William Albert and Thomas Tullis. 2013. Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes. Google ScholarDigital Library
- Peter Bacchetti. 2002. Peer review of statistics in medical research: the other problem. BMJ: British Medical Journal 324, 7348, 1271.Google ScholarCross Ref
- Peter Bacchetti. 2010. Current sample size conventions: Flaws, harms, and alternatives. BMC Medicine 8, 1, 1--7. http://dx.doi.org/10.1186/1741--7015--8--17.Google ScholarCross Ref
- Peter Bacchetti, Steven G. Deeks, and Joseph M. McCune. 2011. Breaking Free of Sample Size Dogma to Perform Innovative Translational Research. Science Translational Medicine 3, 87 (2011-06--15 00:00:00), 87ps24--87ps24. http://dx.doi.org/10.1126/scitranslmed.3001628.Google Scholar
- Peter Bacchetti, Charles E. McCulloch, and Mark R. Segal. 2008. Simple, Defensible Sample Sizes Based on Cost Efficiency. Biometrics 64, 2, 577--585. http://dx.doi.org/10.1111/j.1541-0420.2008.01004_1.x.Google ScholarCross Ref
- Louise Barkhuus and Jennifer A. Rode. 2007. From Mice to Men - 24 Years of Evaluation in CHI. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA 2007), ACM, 2180963. http://dx.doi.org/10.1145/1240624.2180963. Google ScholarDigital Library
- Kathy Baxter, Catherine Courage, and Kelly Caine. 2015. Understanding Your Users: A Practical Guide to User Research Methods. Morgan Kaufmann. Google ScholarDigital Library
- Simone Borsci, Robert D. Macredie, Julie Barnett, Jennifer Martin, Jasna Kuljis, and Terry Young. 2013. Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation. ACM Trans. Comput.-Hum. Interact. 20, 5, 1--23. http://dx.doi.org/10.1145/2506210. Google ScholarDigital Library
- Paul Cairns. 2007. HCI... not as it should be: inferential statistics in HCI research. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 British Computer Society, 195--201. Google ScholarDigital Library
- Ed H. Chi. 2011. On the importance of Replication in HCI and Social Computing Research. In BLOG@CACM.Google Scholar
- Jacob Cohen. 1962. The statistical power of abnormalsocial psychological research: A review. The Journal of Abnormal and Social Psychology 65, 3, 145--153. http://dx.doi.org/http://dx.doi.org.libproxy.clemson.edu/ 10.1037/h0045186.Google ScholarCross Ref
- Paul D Ellis. 2010. The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.Google Scholar
- Barney G Glaser and Anselm L Strauss. 2009. The discovery of grounded theory: Strategies for qualitative research. Transaction Publishers.Google Scholar
- Henry A Glick. 2011. Sample Size and Power for Cost Effectiveness Analysis (Part 2). Pharmacoeconomics 29, 4, 287--296.Google ScholarCross Ref
- Saul Greenberg and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy2008), ACM, 1357074, 111--120. http://dx.doi.org/10.1145/1357054.1357074. Google ScholarDigital Library
- Scott A. Hale. 2014. Global connectivity and multilinguals in the Twitter network. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada2014), ACM, 2557203, 833--842. http://dx.doi.org/10.1145/2556288.2557203. Google ScholarDigital Library
- Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences 33, 2--3, 61--83. http://dx.doi.org/doi:10.1017/S0140525X0999152X.Google ScholarCross Ref
- Wonil Hwang and Gavriel Salvendy. 2010. Number of people required for usability evaluation: the 10±2 rule. Commun. ACM 53, 5, 130--133. http://dx.doi.org/10.1145/1735223.1735255. Google ScholarDigital Library
- Maurits Kaptein and Judy Robertson. 2012. Rethinking statistical analysis methods for CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM, 1105--1114. Google ScholarDigital Library
- Helena Chmura Kraemer and Christine Blasey. 2015. How many subjects?: Statistical power analysis in research. Sage Publications.Google Scholar
- Yong Liu, Jorge Goncalves, Denzil Ferreira, Bei Xiao, Simo Hosio, and Vassilis Kostakos. 2014. CHI 19942013: mapping two decades of intellectual progress through co-word analysis. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada2014), ACM, 2556969, 3553--3562. http://dx.doi.org/10.1145/2556288.2556969. Google ScholarDigital Library
- Jakob Nielsen. 1994. Estimating the number of subjects needed for a thinking aloud test. International journal of human-computer studies 41, 3, 385--397. Google ScholarDigital Library
- R Nuzzo. 2014. Statistical errors: P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature 506, 150, 52.Google ScholarCross Ref
- Robert J. Ploutz-Snyder, James Fiedler, and Alan H. Feiveson. 2014. Justifying small-n research in scientifically amazing settings: Challenging the notion that only "big-n" studies are worthwhile. Journal of Applied Physiology(2014-01-09 22:33:40). http://dx.doi.org/10.1152/japplphysiol.01335.2013.Google Scholar
- Jenny Preece, Helen Sharp, and Yvonne Rogers. 2015. Interaction Design-beyond human-computer interaction. John Wiley & Sons.Google Scholar
- Daniel Reed and Ed H. Chi. 2012. Online privacy; replicating research results. Commun. ACM 55, 10, 8--9. http://dx.doi.org/10.1145/2347736.2347739. Google ScholarDigital Library
- Robert Rosenthal. 1965. The volunteer subject. Human relations 18, 4, 389.Google Scholar
- Albrecht Schmidt and Tovi Grossman. 2014. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM, Toronto, Ontario, Canada, 4206.Google Scholar
- Wendie Wulff and Dick E Mahling. 1990. An assessment of HCI: issues and implications. ACM SIGCHI Bulletin 22, 1, 80--87. Google ScholarDigital Library
Index Terms
- Local Standards for Sample Size at CHI
Recommendations
Stratification and sample size of data sources for agricultural mathematical programming models
A comparison is made between the variance of the estimator of the total of a variable obtained from both a simple and a stratified random sampling, in which the sample sizes of some strata are equal to the stratum population size. It is shown that in ...
Sample Size Determination Algorithm for fingerprint-based indoor localization systems
Provision of accurate location information is an important task in the Internet of Things (IoT) applications and scenarios. This need has boosted the research and development of fingerprint based, indoor localization systems, since GPS information is ...
Density estimation-based method to determine sample size for random sample partition of big data
AbstractRandom sample partition (RSP) is a newly developed big data representation and management model to deal with big data approximate computation problems. Academic research and practical applications have confirmed that RSP is an efficient solution ...
Comments