Abstract
Crowdsourcing can be defined as a task, which is usually performed by an employee, that is given out as an open call to a crowd of users to be completed. Although crowdsourcing has been growing in recent years, its application to design research and education has only scratched the surface of its potential. In this chapter we first introduce the different types of crowdsourcing. Then, following the typical design cycle we present examples from literature and cases from an educational setting of how crowdsourcing can support designers. Based on these examples we provide a list of tips for utilizing crowdsourcing for design and user research activities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Annotated Bibliography
Kittur A, Chi EH, Suh B (2008) Crowdsourcing user studies with Mechanical Turk. In: Proceedings of the SIGCHI conference on human factors in computing systems, April. ACM, New York, USA, pp 453–456
The authors describe an experiment in which they investigate Amazon’s Mechanical Turk and how they can get quality responses from workers
Mason W, Suri S (2012) Conducting behavioral research on Amazon’s Mechanical Turk. Behav Res Methods 44(1):1–23
The paper demonstrates how to use Amazon’s Mechanical Turk website for conducting behavioral research and tries to lower the entry barrier for researchers who could benefit from this platform
McDonald P, Mohebbi M, Slatkin B (2012) Comparing Google consumer surveys to existing probability and non-probability based internet surveys. Google Whitepaper. Retrieved from http://www.google.com/insights/consumersurveys/static/consumer_surveys_whitepaper.pdf
Description of how Google Consumer Surveys is superior to current probability based Internet panels by using what is known as a “surveywall” to attract respondents
Aitamurto T, Leiponen A, Tee R (2013). The promise of idea crowdsourcing: benefits, contexts, limitations, September 25. Retrieved from http://www.academia.edu/963662/The_Promise_of_Idea_Crowdsourcing_Benefits_Contexts_Limitations
The authors review crowdsourcing for idea generation (‘idea crowdsourcing’) both from the perspective of academic literature and actual cases from businesses to understand how and when to use crowdsourcing and with which benefits and costs
References
4 Reasons you should consider crowdsourced design for your next big project (n.d.) Retrieved September 30, 2014, from http://michaelhyatt.com/crowd-sourced-design.html
Adepetu A, Ahmed KA, Al Abd, Y, Al Zaabi A, Svetinovic D (2012) CrowdREquire: a requirements engineering crowdsourcing platform. In: AAAI spring symposium: wisdom of the crowd, March
Albert W, Tullis T, Tedesco D (2009) Beyond the usability lab: conducting large-scale online user experience studies. Morgan Kaufmann
Bayus BL (2013) Crowdsourcing new product ideas over time: an analysis of the dell IdeaStorm community. Manag Sci 59(1):226–244, http://doi.org/10.1287/mnsc.1120.1599
Bjelland O, Wood R (2008) An inside view of IBM’s ‘innovation jam. MIT Sloan Manag Rev 50(1):32–40
Blattberg E (2011) Crowdsourcing industry landscape. Retrieved from http://www.crowdsourcing.org/editorial/november-2011-crowdsourcing-industry-landscape-infographic/7680
Brabham DC (2008) Crowdsourcing as a model for problem solving: an introduction and cases. Converg Int J Res New Media Technol 14(1):75–90, http://doi.org/10.1177/1354856507084420
Corney JR, Sanchez CT, Jagadeesan AP, Regli WC (2009) Outsourcing labour to the cloud. Int J Innov Sustain Dev 4(4):294, http://doi.org/10.1504/IJISD.2009.033083
Crowdsourcing your brand design: the math just doesn’t work out (n.d.) Retrieved from http://forty.co/crowdsourcing-your-brand-design-the-math-just-doesnt-work-out
Dow S, Gerber E, Wong A (2013) A pilot study of using crowds in the classroom. ACM Press, New York, USA, p227, http://doi.org/10.1145/2470654.2470686
Felstiner A (2011) Working the crowd: employment and labor law in the crowdsourcing industry (SSRN scholarly paper No. ID 1593853). Social Science Research Network, Rochester. Retrieved from http://papers.ssrn.com/abstract=1593853
Finke RA (1990) Creative imagery: discoveries and inventions in visualization. L. Erlbaum Associates, Hillsdale
Geiger D, Rosemann M, Fielt E, Schader M (2012) Crowdsourcing information systems – definition, typology, and design. In: ICIS 2012 proceedings. Retrieved from http://aisel.aisnet.org/icis2012/proceedings/ResearchInProgress/53
Glinz M, Wieringa RJ (2007) Guest editors’ introduction: stakeholders in requirements engineering. Software IEEE 24(2):18–20
Grier DA (2011) Foundational issues in human computing and crowdsourcing. In: CHI 2011
Heer J, Bostock M (2010). Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI conference on human factors in computing systems, March. ACM, pp 203–212
Hosseini M, Phalp K, Taylor J, Ali R (2013) Towards crowdsourcing for requirements engineering. In: Joint proceedings of REFSQ-2014 workshops, doctoral symposium, empirical track, and posters, co-located with the 20th international conference on requirements engineering: foundation for software quality (REFSQ 2014). Retrieved from http://ceur-ws.org/Vol-1138/et2.pdf
Howe J (2006) The rise of crowdsourcing. North 14:1–5, http://doi.org/10.1086/599595
Ipeirotis PG (2010) Analyzing the Amazon Mechanical Turk marketplace. XRDS Crossroads ACM Magaz Stud 17(2):16. http://doi.org/10.1145/1869086.1869094
Ko AJ, Chilana PK (2010) How power users help and hinder open bug reporting. ACM Press, New York, USA, p1665. http://doi.org/10.1145/1753326.1753576
Lewis JR (2014) Usability: lessons learned … yet to be learned. Int J Hum Comput Interact 30(9):663–684, http://doi.org/10.1080/10447318.2014.930311
Lim SL, Quercia D, Finkelstein A (2010a) StakeNet: using social networks to analyse the stakeholders of large-scale software projects. In: Proceedings of the 32Nd ACM/IEEE international conference on software engineering – Volume 1. ACM, New York, pp 295–304. http://doi.org/10.1145/1806799.1806844
Lim SL, Quercia D, Finkelstein A (2010b) StakeSource: harnessing the power of crowdsourcing and social networks in stakeholder analysis, vol 2. ACM Press, p 239. http://doi.org/10.1145/1810295.1810340
Liu D, Bias RG, Lease M, Kuipers R (2012) Crowdsourcing for usability testing. Proc Am Soc Inf Sci Technol 49(1):1–10, http://doi.org/10.1002/meet.14504901100
Li Z (2011) Research of crowdsourcing model based on case study. Manag Sci 1–5. http://doi.org/10.1109/ICSSSM.2011.5959456
Malone TW, Laubacher RJ (1999) The dawn of the E-Lance economy. In: Nüttgens M, Scheer A-W (eds) Electronic business engineering. Physica-Verlag HD, Heidelberg, pp 13–24. Retrieved from http://link.springer.com/10.1007/978-3-642-58663-7_2
Mason W, Watts DJ (2009) Financial incentives and the "performance of crowds". In: Bennett P, Chandrasekar R, Chickering M, Ipeirotis P, Law E, Mityagin A, Provost F, von Ahn L (eds) Proceedings of the ACM SIGKDD workshop on human computation (HCOMP '09). ACM, New York, pp 77–85, http://dx.doi.org/10.1145/1600150.1600175
Nelson ET, Stavrou A (2011) Advantages and disadvantages of remote asynchronous usability testing using Amazon Mechanical Turk. Proc Hum Fact Ergon Soc Annu Meet 55(1):1080–1084, http://doi.org/10.1177/1071181311551226
Poetz MK, Schreier M (2012) The value of crowdsourcing: can users really compete with professionals in generating new product ideas?: the value of crowdsourcing. J Prod Innov Manag 29(2):245–256, http://doi.org/10.1111/j.1540-5885.2011.00893.x
Quinn AJ, Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. ACM Press, New York, USA, p 1403. http://doi.org/10.1145/1978942.1979148
Rogers Y, Sharp H, Preece J (2011) Interaction design: beyond human computer interaction, 3rd edn. Wiley
Ross J, Irani L, Silberman MS, Zaldivar A, Tomlinson B (2010) Who are the crowdworkers?: shifting demographics in mechanical turk. ACM Press, p 2863. http://doi.org/10.1145/1753846.1753873
Schön DA (1983) The reflective practitioner: how professionals think in action. Basic Books, New York
Simula H, Ahola T (2014) A network perspective on idea and innovation crowdsourcing in industrial firms. Ind Mark Manag 43(3):400–408, http://doi.org/10.1016/j.indmarman.2013.12.008
Standish Group (1999) Chaos: a recipe for success. Standish Group International
Surowiecki J (2005) The wisdom of crowds. Anchor Books, New York
Tates K, Zwaanswijk M, Otten R, van Dulmen S, Hoogerbrugge PM, Kamps WA, Bensing JM (2009) Online focus groups as a tool to collect data in hard-to-include populations: examples from paediatric oncology. BMC Med Res Methodol 9(1):15, http://doi.org/10.1186/1471-2288-9-15
Taylor A (2000) IT projects: sink or swim. Comput Bull January, 24–26
Viskovic D, Varga M, Curko K (2008) Bad practices in complex IT projects. In: ITI 2008 – 30th international conference on information technology interfaces, p. 301
von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) reCAPTCHA: human-based character recognition via web security measures. Science 321(5895):1465–1468, http://doi.org/10.1126/science.1160379
Why Designers Hate Crowdsourcing (n.d.) Retrieved September 30, 2014, from http://www.forbes.com/2010/07/09/99designs-spec-graphic-technology-future-design-crowdsourcing.html
Yu L, Nickerson JV (2013) An internet-scale idea generation system. ACM Trans Interact Intell Syst 3(1):1–24, http://doi.org/10.1145/2448116.2448118
Acknowledgements
We would like to extend our gratitude to all trainees of the User-system interaction (USI) program of Eindhoven University of Technology (generation 2013) for their insights into using crowdsourcing for design and user research. Moreover, we would like to specially thank the Mindswarms team for their continuous support in helping us to use their service for our educational objectives and the other aforementioned services.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Khan, V.J., Dhillon, G., Piso, M., Schelle, K. (2016). Crowdsourcing User and Design Research. In: Markopoulos, P., Martens, JB., Malins, J., Coninx, K., Liapis, A. (eds) Collaboration in Creative Design. Springer, Cham. https://doi.org/10.1007/978-3-319-29155-0_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-29155-0_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-29153-6
Online ISBN: 978-3-319-29155-0
eBook Packages: Computer ScienceComputer Science (R0)