Abstract
We respond to the commentaries Waldman and Lilienfeld (Psychometrika, 2015) and Wigboldus and Dotch (Psychometrika, 2015) provided in response to Sijtsma’s (Sijtsma in Psychometrika, 2015) discussion article on questionable research practices. Specifically, we discuss the fear of an increased dichotomy between substantive and statistical aspects of research that may arise when the latter aspects are laid entirely in the hands of a statistician, remedies for false positives and replication failure, and the status of data exploration, and we provide a re-definition of the concept of questionable research practices.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678.
Bakker, M., & Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in t tests. The power of alternatives and recommendations. Psychological Methods, 19, 409–427.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.
Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425–440.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Cronbach, L. J. (1954). Report on a psychometric mission to Clinicia. Psychometrika, 19, 263–270.
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.
Fanelli, D. (2013). Redefine misconduct as distorted reporting. Nature, 494, 149.
Fisher, R. A. (1938). Presidential Address. Talk given at the 1st Indian Statistical Conference, Calcutta, India.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
Kuhn, T. S. (2012). The structure of scientific revolutions (4th ed.). Chicago, IL: The University of Chicago Press.
Martinson, B. C., Anderson, M. S., & De Vries, R. (2005). Scientists behaving badly. Nature, 435(7043), 737–738. doi:10.1038/435737a.
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.
Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127–134.
Sijtsma, K. (2015). Playing with data–Or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika. doi:10.1007/s11336-015-9446-0.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12, 53–74.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., Van Assen, M. A. L. M., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PloS ONE, 9, e114876.
Waldman, I. D., & Lilienfeld, S. O. (2015). Thinking about data, research methods, and statistical analyses: Commentary on Sijtsma’s (2014) “Playing with data". Psychometrika. doi:10.1007/s11336-015-9447-z.
Wigboldus, D. H. J., & Dotch, R. (2015). Encourage playing with data and discourage questionable reporting practices. Psychometrika. doi:10.1007/s11336-015-9445-1.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sijtsma, K., Veldkamp, C.L.S. & Wicherts, J.M. Improving the Conduct and Reporting of Statistical Analysis in Psychology. Psychometrika 81, 33–38 (2016). https://doi.org/10.1007/s11336-015-9444-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-015-9444-2