Skip to main content
SpringerLink
Log in

A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Samples drawn from commercial online panel data (OPD) are becoming more prevalent in applied psychology research, but they remain controversial due to concerns with data quality. In order to examine the validity of OPD, we conduct meta-analyses of online panel samples and compare internal reliability estimates for scales and effect size estimates for IV–DV relations commonly found in the field with those based on conventionally sourced data. Results based on 90 independent samples and 32,121 participants show OPD has similar psychometric properties and produces criterion validities that generally fall within the credibility intervals of existing meta-analytic results from conventionally sourced data. We suggest that, with appropriate caution, OPD are suitable for many exploratory research questions in the field of applied psychology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. These quotes are from an open-ended question (“Is there anything else you wish to say about online panel samples that haven’t been covered in this survey?”) from an anonymous survey sent to a randomized selection of 500 review board members from Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Organizational Behavior and Human Decision Processes, and Personnel Psychology in March 2014.

  2. Although the primary purpose of this research was to examine online panel data as a whole, there may be interest in examining differences between MTurk and other online panel sources (such as StudyResponse and Qualtrics). Therefore, we performed supplemental analysis for relationships where there were a minimum of three MTurk samples and three samples from other online panel sources. These results are not substantially different as 80% of the MTurk relationships and 88% of the Qualtrics/StudyResopnse/Zoomerang relationships were within the 80% credibility interval of the conventional meta-analyses. Results are presented in Table 5.

References

The articles marked with an asterisk are included in the meta-analysis.

  • *Alarcon, G. M. (2009). The development of the wright work engagement scale (Doctoral dissertation). Available from ProQuest Dissertations & Theses database. (UMI No. 3393395).

  • *Badger, J. M. (2014). The formative nature of perceived person-environment fit (Doctoral dissertation). Available from ProQuest Dissertations & Theses database. (UMI No. 3615149).

  • Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., et al. (2010). Research synthesis AAPOR report on online panels. Public Opinion Quarterly, 74(4), 711–781.

    Article  Google Scholar 

  • *Ballinger, G. A., Lehman, D. W., & Schoorman, F. D. (2010). Leader–member exchange and turnover before and after succession events. Organizational Behavior and Human Decision Processes, 113(1), 25–36.

  • *Baratta, P. (2014). The “Noonday Demon”, weariness, inattention, or all of the above? Refining the definition and measurement of state boredom (Unpublished doctoral dissertation). The University of Guelph.

  • *Basford, T. E., Offermann, L. R., & Behrend, T. S. (2014). Please accept my sincerest apologies: Examining follower reactions to leader apology. Journal of Business Ethics, 119(1), 99–117.

  • *Bauer, J. A. (2013). An investigation of OCB demands and workplace behaviors. (Doctoral dissertation). Graduate theses and dissertations. https://scholarcommons.usf.edu/etd/4634

  • Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800–813.

    Article  PubMed  Google Scholar 

  • Bergman, M. E., & Jean, V. A. (2016). Where have all the “workers” gone? A critical analysis of the unrepresentativeness of our samples relative to the labor market in the industrial–organizational psychology literature. Industrial and Organizational Psychology, 9(01), 84–113.

    Article  Google Scholar 

  • Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational deviance, and their common correlates: A review and meta-analysis. Journal of Applied Psychology, 92(2), 410–424.

    Article  PubMed  Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: John Wiley & Sons, Ltd.

    Book  Google Scholar 

  • Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal consistency coefficients. Psychological Methods, 15, 386–397.

    Article  PubMed  Google Scholar 

  • *Bowling, N. A., & Burns, G. N. (2010). A comparison of work-specific and general personality measures as predictors of work and non-work criteria. Personality and Individual Differences, 49(2), 95–101.

  • *Bowling, N. A., Burns, G. N., & Beehr, T. A. (2010). Productive and counterproductive attendance behavior: An examination of early and late arrival to and departure from work. Human Performance, 23(4), 305–322.

  • *Bowling, N. A., Burns, G. N., Stewart, S. M., & Gruys, M. L. (2011). Conscientiousness and agreeableness as moderators of the relationship between neuroticism and counterproductive work behaviors: A constructive replication. International Journal of Selection and Assessment, 19(3), 320–330.

  • *Bowling, N. A., & Eschleman, K. J. (2010). Employee personality as a moderator of the relationships between work stressors and counterproductive work behavior. Journal of Occupational Health Psychology, 15(1), 91–103.

  • *Bowling, N. A., & Michel, J. S. (2011). Why do you treat me badly? The role of attributions regarding the cause of abuse in subordinates’ responses to abusive supervision. Work & Stress, 25(4), 309–320.

  • Brüggen, E., Wetzels, M., de Ruyter, K., & Schillewaert, N. (2011). Individual differences in motivation to participate in online panels: The effect on response rate and response quality perceptions. International Journal of Market Research, 53(3), 369–390.

    Article  Google Scholar 

  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.

    Article  PubMed  Google Scholar 

  • *Bunk, J. A. (2006). The role of appraisals, emotions, and coping in understanding experiences of workplace incivility.

  • *Burton, J. P. (2014). The role of job embeddedness in the relationship between bullying and aggression. European Journal of Work and Organizational Psychology, 24(4), 518–529.

  • Callegaro, M., Villar, A., Yeager, D., & Krosnick, J. A. (2014). A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples. Online panel research: A data quality perspective, pp. 23–53.

  • *Carlsen, J. J. (2015). An investigation of work engagement as a moderator of the relationship between personality and work outcomes (Doctoral dissertation, San Diego State University).

  • *Carsten, M. K., & Uhl-Bien, M. (2012). Follower beliefs in the co-production of leadership. Zeitschrift für Psychologie, 220(4), 210–220.

  • *Castille, C. M. (2015). Bright or dark, or virtues and vices? A reexamination of the big five and job performance. Louisiana Tech University.

  • Chiaburu, D. S., Oh, I. S., Berry, C. M., Li, N., & Gardner, R. G. (2011). The five-factor model of personality traits and organizational citizenship behaviors: A meta-analysis. Journal of Applied Psychology, 96(6), 1140.

    Article  PubMed  Google Scholar 

  • Choi, D., Oh, I. S., & Colbert, A. (2015). Understanding organizational commitment: A meta-analytic examination of the roles of the five-factor model of personality and culture. Journal of Applied Psychology, 100(5), 1542–1567.

    Article  PubMed  Google Scholar 

  • *Chung-Yan, G. A. (2010). The nonlinear effects of job complexity and autonomy on job satisfaction, turnover, and psychological well-being. Journal of Occupational Health Psychology, 15(3), 237.

  • Cochran, M. N. (2014). Counterproductive work behaviors, justice, and affect: A meta-analysis (Unpublished doctoral dissertation). University of Central Florida, Orlando, FL.

  • *Cochrum-Nguyen, F. L. (2013). Predicting job performance and job satisfaction: An examination of the five-factor model of personality, polychronicity and role overload (Doctoral dissertation, San Diego State University).

  • *Cohen, T. R., Panter, A. T., & Turan, N. (2013). Predicting counterproductive work behavior from guilt proneness. Journal of Business Ethics, 114(1), 45–53.

  • *Cohen, T. R., Panter, A. T., Turan, N., Morse, L., & Kim, Y. (2013). Agreement and similarity in self-other perceptions of moral character. Journal of Research in Personality, 47(6), 816–830.

  • *Colbert, A. E., Bono, J. E., & Purvanova, R. (2008). Development of a relationship functions inventory: Assessing the functions of high-quality work relationships. Paper presented at the annual meeting of the Academy of Management, Anaheim, CA.

  • Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O., & Ng, K. Y. (2001). Justice at the millennium: A meta-analytic review of 25 years of organizational justice research. Journal of Applied Psychology, 86(3), 425.

    Article  PubMed  Google Scholar 

  • Cook, T. D., & Campbell, D. T. (1976). Four kinds of validity. Handbook of industrial and organizational psychology (pp. 224–246).

  • *Costa, J. B. (2015). Coping through counterproductive work behaviors: An examination of how employees deal with emotional labor (Master’s thesis, Roosevelt University).

  • *Credé, M., Harms, P., Niehorster, S., & Gaye-Valentine, A. (2012). An evaluation of the consequences of using short measures of the Big Five personality traits. Journal of Personality and Social Psychology, 102(4), 874.

  • *Dahling, J. J., & Thompson, M. N. (2013). Detrimental relations of maximization with academic and career attitudes. Journal of Career Assessment, 21(2), 278–294.

  • Dalal, R. S. (2005). A meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. Journal of Applied Psychology, 90(6), 1241–1255.

    Article  PubMed  Google Scholar 

  • *Decker, C., & Van Quaquebeke, N. (2015). Getting respect from a boss you respect: How different types of respect interact to explain subordinates’ job satisfaction as mediated by self-determination. Journal of Business Ethics, 131(3), 543–556.

  • DeGroot, T., Kiker, D. S., & Cross, T. C. (2000). A meta-analysis to review organizational outcomes related to charismatic leadership. Canadian Journal of Administrative Sciences/Revue Canadienne des Sciences de l’Administration, 17(4), 356–372.

    Article  Google Scholar 

  • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171–181.

    Article  Google Scholar 

  • Dillman, D, A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, NJ: John Wiley & Sons, Inc.

  • Duhachek, A., Coughlan, A. T., & Iacobucci, D. (2005). Results on the standard error of the coefficient alpha index of reliability. Marketing Science, 24, 294–301.

    Article  Google Scholar 

  • *Duniewicz, K. (2015). Don't get mad, get even: how employees abused by their supervisor retaliate against the organization and undermine their spouses. FIU Electronic Theses and Dissertations. Paper 1848. https://digitalcommons.fiu.edu/etd/1848.

  • *Eschleman, K. J., Bowling, N. A., & Judge, T. A. (2015). The dispositional basis of attitudes: A replication and extension of Hepler and Albarracín (2013). Journal of Personality and Social Psychology, 108(5), e1–e15.

  • Feitosa, J., Joseph, D. L., & Newman, D. A. (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52.

    Article  Google Scholar 

  • *Ferris, D. L., Johnson, R. E., Rosen, C. C., Djurdjevic, E., Chang, C. H. D., & Tan, J. A. (2013). When is success not satisfying? Integrating regulatory focus and approach/avoidance motivation theories to explain the relation between core self-evaluation and job satisfaction. Journal of Applied Psychology, 98(2), 342–353.

  • Fisher, R. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society. Series B (Methodological), 69–78.

  • Fisher, G. G., & Sandell, K. (2015). Sampling in industrial–organizational psychology research: Now what? Industrial and Organizational Psychology, 8(02), 232–237.

    Article  Google Scholar 

  • Fleischer, A., Mead, A. D., & Huang, J. (2015). Inattentive responding in MTurk and other online samples. Industrial and Organizational Psychology, 8(02), 196–202.

    Article  Google Scholar 

  • *Gabler, C. B., Nagy, K. R., & Hill, R. P. (2014). Causes and consequences of abusive supervision in sales management: A tale of two perspectives. Psychology & Marketing, 31(4), 278–293.

  • *Gangadharan, A. (2014). Can I smile with spirit? Towards a process model associating workplace spirituality and emotional labor (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3642187).

  • *Giacopelli, N. M., Simpson, K. M., Dalal, R. S., Randolph, K. L., & Holland, S. J. (2013). Maximizing as a predictor of job satisfaction and performance: A tale of three scales. Judgment and Decision making, 8(4), 448–469.

  • Gillespie, M. A., Gillespie, J. Z., Brodke, M. H., & Balzer, W. K. (2016). The importance of sample composition depends on the research question. Industrial and Organizational Psychology, 9(01), 207–211.

    Article  Google Scholar 

  • *Goo, W. (2015). Employee needs and job-related opportuniites: From the person-environment fit framework (Doctoral dissertation). Available from ProQuest Dissertations & Theses database. (UMI No. 3681240).

  • Goodman, J. K., & Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210.

    Article  Google Scholar 

  • Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93.

    Article  PubMed  Google Scholar 

  • Greco, L., O’Boyle, E., Cockburn, B., & Yuan, Z. (2015). Raising the .70 bar: A meta-analytic study of coefficient alpha. Working Paper.

  • Griffeth, R. W., Hom, P. W., & Gaertner, S. (2000). A meta-analysis of antecedents and correlates of employee turnover: Update, moderator tests, and research implications for the next millennium. Journal of Management, 26(3), 463–488.

    Article  Google Scholar 

  • *Hannah, S. T., Jennings, P. L., Bluhm, D., Peng, A. C., & Schaubroeck, J. M. (2014). Duty orientation: Theoretical development and preliminary construct testing. Organizational Behavior and Human Decision Processes, 123(2), 220–238.

  • Harms, P. D., & DeSimone, J. A. (2015). Caution! MTurk workers ahead—fines doubled. Industrial and Organizational Psychology, 8(02), 183–190.

    Article  Google Scholar 

  • *Hausknecht, J. P., Sturman, M. C., & Roberson, Q. M. (2011). Justice as a dynamic construct: Effects of individual trajectories on distal work outcomes. Journal of Applied Psychology, 96(4), 872–880.

  • Hershcovis, M. S., Turner, N., Barling, J., Arnold, K. A., Dupré, K. E., Inness, M., Mirielle LeBlanc, M., & Sivanathan, N. (2007). Predicting workplace aggression: A meta-analysis. Journal of Applied Psychology, 92(1), 228–238.

    Article  PubMed  Google Scholar 

  • Highhouse, S., & Gillespie, J. Z. (2009). Do samples really matter that much. Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 247–265).

  • Hillygus, D. S., Jackson, N., & Young, M. (2014). Professional respondents in non-probability online panels. Online panel research: A data quality perspective, 1. 219–237.

  • *Holtz, B. C., & Harold, C. M. (2013a). Effects of leadership consideration and structure on employee perceptions of justice and counterproductive work behavior. Journal of Organizational Behavior, 34(4), 492–519.

  • *Holtz, B. C., & Harold, C. M. (2013b). Interpersonal justice and deviance the moderating effects of interpersonal justice values and justice orientation. Journal of Management, 39(2), 339–365.

  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114.

    Article  Google Scholar 

  • Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845.

    Article  PubMed  Google Scholar 

  • Jackson, T. A., Meyer, J. P., & Wang, X. H. F. (2013). Leadership, commitment, and culture: A meta-analysis. Journal of Leadership & Organizational Studies, 20(1), 84–106.

    Article  Google Scholar 

  • *Jenkins, J. S., Heneghan, C. J., Bailey, S. F., & Barber, L. K. (2014). The work–family interface as a mediator between job demands and employee behaviour. Stress and Health.

  • *Jeon, G. (2011). Equity sensitivity versus equity preference: Validating a new viewpoint on equity sensitivity (Unpublished doctoral dissertation). University of Illionois at Urbana-Champaign.

  • *Johnson, V. A., Beehr, T. A., & O’Brien, K. E. (2015). Determining the relationship between employee psychopathy and strain: Does the type of psychopathy matter? International Journal of Stress Management, 22(2), 111–136.

  • *Johnston-Fisher, J. (2014). Testing a multi-level mediation model of workgroup incivility: The role of civility climate and group norms for civility (Unpublished master’s thesis). Western Kentucky University.

  • *Joseph, D. L. (2011). Emotional intelligence, leader-member exchange, and behavioral engagement: Considering mediation and reciprocity effects (Doctoral dissertation, University of Illinois at Urbana-Champaign).

  • Judge, T. A., Heller, D., & Mount, M. K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87(3), 530–541.

    Article  PubMed  Google Scholar 

  • *Kiffin-Petersen, S. A., Jordan, C. L., & Soutar, G. N. (2011). The Big Five, emotional exhaustion and citizenship behaviors in service settings: The mediating role of emotional labor. Personality and Individual Differences, 50(1), 43–48.

  • *Krischer, M. M., Penney, L. M., & Hunter, E. M. (2010). Can counterproductive work behaviors be productive? CWB as emotion-focused coping. Journal of Occupational Health Psychology, 15(2), 154.

  • *Lambert, L. S., Tepper, B. J., Carr, J. C., Holt, D. T., & Barelka, A. J. (2012). Forgotten but not gone: An examination of fit between leader consideration and initiating structure needed and received. Journal of Applied Psychology, 97(5), 913.

  • Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, mechanical turk, and other convenience samples. Industrial and Organizational Psychology, 1–23.

  • *Lee, J. (2012). The effects of leadership behavior on workplace harassment, employee outcomes, and organizational effectiveness in small businesses (Doctoral dissertation). Available from ABI/INFORM Global; ProQuest Dissertations & Theses database (UMI No. 3489453).

  • *Long, C. P., Bendersky, C., & Morrill, C. (2011). Fairness monitoring: Linking managerial controls and fairness judgments in organizations. Academy of Management Journal, 54(5), 1045–1068.

  • *Long, E. C., & Christian, M. S. (2015). Mindfulness buffers retaliatory responses to injustice: A regulatory approach. Journal of Applied Psychology, 100(5), 1409–1422.

  • *Lusin, J. M. (2014). Employee perceptions of authentic leadership and outcomes of planned organizational change (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3615782).

  • Mackey, J. D., Frieder, R. E., Brees, J. R., & Martinko, M. J. (2015). Abusive supervision: A meta-analysis and empirical review. Journal of Management, 0149206315573997.

  • Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23.

    Article  PubMed  Google Scholar 

  • *Mayer, D. M., Thau, S., Workman, K. M., Van Dijke, M., & De Cremer, D. (2012). Leader mistreatment, employee hostility, and deviant behaviors: Integrating self-uncertainty and thwarted needs perspectives on deviance. Organizational Behavior and Human Decision Processes, 117(1), 24–40.

  • McGonagle, A. K. (2015). Participant motivation: A critical consideration. Industrial and Organizational Psychology, 8(02), 208–214.

    Article  Google Scholar 

  • Meade, A. W. (2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95(4), 728.

    Article  PubMed  Google Scholar 

  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437.

    Article  PubMed  Google Scholar 

  • *Meyer, R. D., Dalal, R. S., José, I. J., Hermida, R., Chen, T. R., Vega, R. P., et al. (2014). Measuring job-related situational strength and assessing its interactive effects with personality on voluntary work behavior. Journal of Management, 40(4), 1010–1041.

  • *Michel, J. S., & Clark, M. A. (2009). Has it been affect all along? A test of work-to-family and family-to-work models of conflict, enrichment, and satisfaction. Personality and Individual Differences, 47(3), 163–168.

  • *Michel, J. S., Newness, K., & Duniewicz, K. (2016). How abusive supervision affects workplace deviance: A moderated-mediation examination of aggressiveness and work-related negative affect. Journal of Business and Psychology, 31(1), 1–22.

  • *Mullins, A. K. (2015). The dimensionality of destructive leadership: Toward an integration of the bright and dark sides. North Carolina State University.

  • *Murphy, S. L. (2015). Individual adaptability as a predictor of job performance. Louisiana Tech University.

  • *Nichols, A. L., & Cottrell, C. A. (2014). What do people desire in their leaders? The role of leadership level on trait desirability. The Leadership Quarterly, 25(4), 711–729.

  • Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.

    Google Scholar 

  • *O’Brien, K. E. (2008). A stressor-strain model of organizational citizenship behavior and counterproductive work behavior (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3347361).

  • *O’Boyle, E. H. (2010). A test of the general CWB-OCB emotion model (Doctoral dissertation). Available from ABI/INFORM Global; ProQuest Dissertations & Theses database (UMI No. 3411997).

  • Paolacci, G., & Chandler, J. (2014). Inside the turk understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188.

    Article  Google Scholar 

  • Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision making, 5(5), 411–419.

    Google Scholar 

  • *Penney, L. M., Hunter, E. M., & Perry, S. J. (2011). Personality and counterproductive work behaviour: Using conservation of resources theory to narrow the profile of deviant employees. Journal of Occupational and Organizational Psychology, 84(1), 58–77.

  • *Petersen, N. L. (2015). Retaliatory behavior as a response to executive compensation. Bowling Green State University.

  • *Porter, C., Woo, S. E., & Tak, J. (2015). Developing and validating short form protean and boundaryless career attitudes scales. Journal of Career Assessment, 1069072714565775.

  • Postoaca, A. (2006). Launching the bottle: The rhetoric of the online researcher. The anonymous elect: market research through online access panels (pp. 67–107).

  • *Powell, N. C. (2013). Responding to abusive supervision: Opposing arguments for the role of social class in predicting workplace deviance. (Masters thesis). Available from https://uwaterloo.ca.

  • *Ramirez, S. A. (2015). Impulsive and premeditated counterproductive work behaviors and the moderating effects of self-monitoring and core self-evaluation. North Carolina State University.

  • Ran, S., Liu, M., Marchiondo, L. A., & Huang, J. L. (2015). Difference in response effort across sample types: Perception or reality? Industrial and Organizational Psychology, 8(02), 202–208.

    Article  Google Scholar 

  • *Richards, D. A., & Schat, A. C. (2011). Attachment at (not to) work: Applying attachment theory to explain individual behavior in organizations. Journal of Applied Psychology, 96(1), 169–182.

  • *Rosen, C. C., Slater, D. J., & Johnson, R. E. (2013). Let’s make a deal development and validation of the ex post I-deals scale. Journal of Management, 39(3), 709–742.

  • Sackett, P. R., & Larson Jr, J. R. (1990). Research strategies and tactics in industrial and organizational psychology.

  • *Salvaggio, T. (2014). Towards a more complete understanding of the effects of the virtual environment on the relationship between leadership style and outcomes of LMX relationships (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3690538).

  • Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Schmidt, F. L., Shaffer, J. A., & Oh, I.-S. (2008). Increased accuracy for range restriction corrections: Implications for the role of personality and general mental ability in job and training performance. Personnel Psychology, 61, 827–868.

    Article  Google Scholar 

  • *Schultz, L. A. (2009). Exploring the relationship between the positive and negative sides of the work-family interface: The role of enrichment in buffering the effects of time-, strain-, and behavior-based conflict (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3378859).

  • *Scott, K. A., & Zweig, D. (2008, May). Dispositional predictors of organizational cynicism. In ASAC (Vol. 29, No. 5).

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton: Mifflin and Company.

    Google Scholar 

  • *Shao, P. (2010). Ethics-based leadership and employee ethical behavior: Examining the mediating role of ethical regulatory focus (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3430591).

  • *Shao, P., Resick, C. J., & Hargis, M. B. (2011). Helping and harming others in the workplace: The roles of personal values and abusive supervision. Human Relations, 64(8), 1051–1078.

  • *Sharif, M. M., & Scandura, T. A. (2014). Do perceptions of ethical conduct matter during organizational change? Ethical leadership and employee involvement. Journal of Business Ethics, 124(2), 185–196.

  • Sharpe Wessling, K., Huber, J., & Netzer, O. (2017). MTurk character misrepresentation: Assessment and solutions. Journal of Consumer Research, 44(1), 211–230.

    Article  Google Scholar 

  • Smith, R. & Hofma Brown, H. (2006). Panel and data quality: Comparing metrics and assessing claims. In Proceedings of the ESOMAR Panel Research Conference. Barcelona: ESOMAR.

  • Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S. (2015). A convenient solution: Using MTurk to sample from hard-to-reach populations. Industrial and Organizational Psychology, 8(02), 220–228.

    Article  Google Scholar 

  • Sparrow, N. (2007). Quality issues in online research. Journal of Advertising Research-New York, 47(2), 179.

    Article  Google Scholar 

  • Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155–167.

    Article  PubMed  Google Scholar 

  • *Sprung, J. M., & Jex, S. M. (2012). Work locus of control as a moderator of the relationship between work stressors and counterproductive work behavior. International Journal of Stress Management, 19(4), 272–291.

  • *Tepper, B. J., Carr, J. C., Breaux, D. M., Geider, S., Hu, C., & Hua, W. (2009). Abusive supervision, intentions to quit, and employees’ workplace deviance: A power/dependence analysis. Organizational Behavior and Human Decision Processes, 109(2), 156–167.

  • *Tepper, B. J., Mitchell, M. S., Haggard, D. L., Kwan, H. K., & Park, H. M. (2015). On the exchange of hostility with supervisors: An examination of self-enhancing and self-defeating perspectives. Personnel Psychology, 68(4), 723–758.

  • *Thau, S., Bennett, R. J., Mitchell, M. S., & Marrs, M. B. (2009). How management style moderates the relationship between abusive supervision and workplace deviance: An uncertainty management theory perspective. Organizational Behavior and Human Decision Processes, 108(1), 79–92.

  • *Thau, S., & Mitchell, M. S. (2010). Self-gain or self-regulation impairment? Tests of competing explanations of the supervisor abuse and employee deviance relationship through perceptions of distributive justice. Journal of Applied Psychology, 95(6), 1009.

  • *Thompson, C. N. (2008). Personal characteristics and the impact of transformational leadership behaviors on follower outcomes (Doctoral dissertation). Available from ABI/INFORM Global; ProQuest Dissertations & Theses database (UMI No. 3375475).

  • Thoresen, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003). The affective underpinnings of job perceptions and attitudes: A meta-analytic review and integration. Psychological Bulletin, 129(6), 914–945.

    Article  PubMed  Google Scholar 

  • *Toaddy, S. (2012). Validation of a measure of external organizational justice. North Carolina State University.

  • Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6–20.

    Article  Google Scholar 

  • Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70.

    Article  Google Scholar 

  • *van Prooijen, J. W., & de Vries, R. E. (2016). Organizational conspiracy beliefs: Implications for leadership styles and employee outcomes. Journal of Business and Psychology, 31(4), 479–491.

  • Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48.

    Article  Google Scholar 

  • *Vogel, R. M., & Mitchell, M. S. (2015). The motivational effects of diminished self-esteem for employees who experience abusive supervision. Journal of Management.

  • *Wall, A. (2014). Common method variance: An experimental manipulation (Doctoral dissertation). Available from ProQuest Dissertations & Theses database (UMI No. 3662469).

  • *Wilson, L. M. (2015). An examination of the moderating effects of work centrality on the relationships between person-group fit and work-related outcomes (Doctoral dissertation, San Diego State University).

  • *Wiltshire, J., Bourdage, J. S., & Lee, K. (2014). Honesty-humility and perceptions of organizational politics in predicting workplace outcomes. Journal of Business and Psychology, 29(2), 235–251.

  • Wood, J. A. (2008). Methodology for dealing with duplicate study effects in a meta-analysis. Organizational Research Methods, 11(1), 79–95.

    Article  Google Scholar 

  • *Wynne, K. T. (2012). Profiling leaders: Using a profiling approach to examine the effects of multifactor leadership on follower deviance (Master’s thesis). Available from ProQuest Dissertations & Theses database (UMI No. 1518818).

  • Zimmerman, R. D. (2008). Understanding the impact of personality traits on individuals' turnover decisions: A meta-analytic path model. Personnel Psychology, 61(2), 309–348.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Management and Entrepreneurship, Indiana University, 1309 E. 10th St., Bloomington, IN, 47405, USA

    Sheryl L. Walter & Ernest H. O’Boyle Jr

  2. Department of Management, University of Oregon, Eugene, OR, USA

    Scott E. Seibert

  3. Department of Management and Organizations, University of Iowa, Iowa City, IA, USA

    Daniel Goering

Authors
  1. Sheryl L. Walter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Scott E. Seibert
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Goering
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ernest H. O’Boyle Jr
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sheryl L. Walter.

Appendices

Appendix 1

Table 4 Results from existing meta-analyses using conventionally sourced data
Full size table
Table 5 Results for meta-analysis of online panel samples
Full size table

Appendix 2

Table 6 Main codes and input values for the primary OPD studies included the meta-analysis
Full size table

Appendix 3

Studies considered but excluded from the current meta-analyses (k = 23)

Excluded due to mixed samples (i.e., combined conventional and OPD samples) (k = 9):

  • Dennis, R., & Winston, B. E. (2003). A factor analysis of Page and Wong’s servant leadership instrument. Leadership & Organization Development Journal, 24(8), 455–459.

  • Irak, D. U. (2010). The role of affectivity in an expanded model of person-environment fit. (NR70552 Ph. D.), Carleton University (Canada). Ann Arbor. Retrieved from http://search. proquest. com/docview/851889665.

  • McAllister, C. P., Harris, J. N., Hochwarter, W. A., Perrewé, P. L., & Ferris, G. R. Got Resources? A multi-sample constructive replication of perceived resource availability’s role in work passion–job outcomes relationships. Journal of Business and Psychology, 1–18.

  • Raver, J. L., & Nishii, L. H. (2010). Once, twice, or three times as harmful? Ethnic harassment, gender harassment, and generalized workplace harassment. Journal of Applied Psychology, 95(2), 236.

  • Sandell, K. (2007). Transformational leadership, engagement, and performance: A new perspective (Doctoral dissertation, Colorado State University. Libraries).

  • Smith, C. L. (2007). The relational context of employee engagement: An intrinsic perspective (Doctoral dissertation, Colorado State University. Libraries).

  • Staples, D. S., & Webster, J. (2007). Exploring traditional and virtual team members’ “best practices” a social cognitive theory perspective. Small Group Research, 38(1), 60–97.

  • Thoroughgood, C. N., Tate, B. W., Sawyer, K. B., & Jacobs, R. (2012). Bad to the bone empirically defining and measuring destructive leader behavior. Journal of Leadership & Organizational Studies, 19(2), 230–255.

  • Tolentino, A. L. (2009). Are all good soldiers created equal? examining the “why” that underlies organizational citizenship behavior: The development of an OCB motives scale. (Doctoral dissertation, University of South Florida).

Excluded due to using online panel company’s survey webhosting but not panel data = (e.g., Survey Monkey) (k = 10):

  • Anderson, L. E. (2015). Relationship between leadership, organizational commitment, and intent to stay among junior executives (Doctoral dissertation, Walden University).

  • Ayers, J. P. (2010). Job satisfaction, job involvement, and perceived organizational support as predictors of organizational commitment (Doctoral dissertation, Walden University).

  • Barbuto Jr., J. E., & Millard, M. L. Developing wisdom and reducing emotional labor in the workplace: Testing the impact of servant leadership.

  • De Lacy, J. C. (2009). Employee engagement: the development of a three dimensional model of engagement; and an exploration of its relationship with affective leader behaviours.

  • Emu, K. E., & Umeh, O. J. (2014). How leadership practices impact job satisfaction of customer relationship officers’: An empirical study. Journal of Management, 2(3), 19–56.

  • Mutsvunguma, P. S. (2012). Ethical climate fit, leader-member exchange and employee job outcomes (Doctoral dissertation).

  • Rader, M. M. (2015). Effects of authentic leadership on job satisfaction and younger worker turnover intentions (Doctoral dissertation, The Chicago School of Professional Psychology).

  • Spector, P. E., & Che, X. X. (2014). Re-examining citizenship: How the control of measurement artifacts affects observed relationships of organizational citizenship behavior and organizational variables. Human Performance, 27(2), 165–182.

  • Yates, L. (2011). Exploring the relationship of ethical leadership with job satisfaction, organizational commitment, and organizational citizenship behavior.

  • Yukl, G., O’Donnell, M., & Taber, T. (2009). Influence of leader behaviors on the leader-member exchange relationship. Journal of Managerial Psychology, 24(4), 289–299.

Excluded due to niche or otherwise unique online panel (total k = 3):

Online panel of Dutch public sector employees (k = 1):

  • Ashikali, T., & Groeneveld, S. (2015). Diversity management in public organizations and its effect on employees’ affective commitment the role of transformational leadership and the inclusiveness of the organizational culture. Review of Public Personnel Administration, 35(2), 146–168.

Craigslist in Southeastern USA (k = 1):

  • Colquitt, J. A., Long, D. M., Rodell, J. B., & Halvorsen-Ganepola, M. D. (2015). Adding the “in” to justice: A qualitative and quantitative investigation of the differential effects of justice rule adherence and violation. Journal of Applied Psychology, 100(2), 278.

Social workers belonging to social work online community magazine (k = 1):

  • Sullivan, E. M. (2012). A correlational study of perceived transformational leadership styles and job satisfaction among social workers (Doctoral dissertation, University of Phoenix).

Excluded due to lack of reporting effect size for relationship of interest (k = 1):

  • Swee, H. Y. (2009). A cognitive perspective of self-other agreement: A look at outcomes and predictors of shared implicit performance theories (Doctoral dissertation, University of Akron).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Walter, S.L., Seibert, S.E., Goering, D. et al. A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?. J Bus Psychol 34, 425–452 (2019). https://doi.org/10.1007/s10869-018-9552-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-018-9552-y

Keywords

Search

Navigation