The Untrustworthy Evidence in Dishonesty Research

František Bartoš

doi:10.15626/MP.2023.3987

Authors

František Bartoš University of Amsterdam

DOI:

https://doi.org/10.15626/MP.2023.3987

Keywords:

z-curve, TIVA, test statistics, statistical power, false positive risk

Abstract

Replicable and reliable research is essential for cumulative science and its applications in practice. This article examines the quality of research on dishonesty using a sample of 286 hand-coded test statistics from 99 articles. Z-curve analysis indicates a low expected replication rate, a high proportion of missing studies, and an inflated false discovery risk. Test of insufficient variance (TIVA) finds that 11/61 articles with multiple test statistics contain results that are ``too-good-to-be-true''. Sensitivity analysis confirms the robustness of the findings. In conclusion, caution is advised when relying on or applying the existing literature on dishonesty.

Metrics

Metrics Loading ...

References

Abeler, J., Nosenzo, D., & Raymond, C. (2019). Preferences for truth-telling. Econometrica, 87(4), 1115–1153. https://doi.org/10.3982/ECTA14673 DOI: https://doi.org/10.3982/ECTA14673

Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a DOI: https://doi.org/10.1038/533452a

Bartoš, F. (2019). Faktory asociované s podvádˇením [Bachelor’s thesis]. Univerzita Karlova, Filozofická fakulta. https://dspace.cuni.cz/handle/20.500.11956/107893

Bartoš, F., Maier, M., Shanks, D. R., Stanley, T., Sladekova, M., & Wagenmakers, E.-J. (2023). Meta-analyses in psychology often overestimate evidence for and size of effects. Royal Society Open Science, 10(7), 1–12. https://doi.org/10.1098/rsos.230224 DOI: https://doi.org/10.1098/rsos.230224

Bartoš, F., Maier, M., Wagenmakers, E.-J., Doucouliagos, H., & Stanley, T. D. (2022). Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods. Research Synthesis Methods, 14(1), 99–116. https://doi.org/10.1002/jrsm.1594 DOI: https://doi.org/10.1002/jrsm.1594

Bartoš, F., Maier, M., Wagenmakers, E.-J., Nippold, F., Doucouliagos, H., Ioannidis, J. P. A., Otte, W. M., Sladekova, M., Deresssa, T. K., Bruns, S. B., Fanelli, D., & Stanley, T. D. (2022). Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics. https://doi.org/10.48550/arXiv.2208.12334

Bartoš, F., & Schimmack, U. (2020). zcurve: An R package for fitting z-curves [R package version 2.1.2]. https://CRAN.R-project.org/package=zcurve

Bartoš, F., & Schimmack, U. (2022). Z-curve. 2.0: Estimating replication rates and discovery rates. Meta-Psychology, 6, 1–14. https://doi.org/10.15626/MP.2021.2720 DOI: https://doi.org/10.15626/MP.2021.2720

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. https://doi.org/10.1038/s41562-017-0189-z DOI: https://doi.org/10.1038/s41562-017-0189-z

Brunner, J., & Schimmack, U. (2020). Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology, 4. https://doi.org/10.15626/MP.2018.874 DOI: https://doi.org/10.15626/MP.2018.874

Buckley, J., Hyland, T., & Seery, N. (2022). Estimating the replicability of technology education research. International Journal of Technology and Design Education, 1–22. https://doi.org/10.1007/s10798-022-09787-6 DOI: https://doi.org/10.31219/osf.io/93wxp

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475 DOI: https://doi.org/10.1038/nrn3475

Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918 DOI: https://doi.org/10.1126/science.aaf0918

Chambers, C. D. (2013). Registered reports: A new publishing initiative at cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016 DOI: https://doi.org/10.1016/j.cortex.2012.12.016

Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered reports: Realigning incentives in scientific publishing. Cortex, 66, A1–A2. https://doi.org/10.1016/j.cortex.2015.03.022 DOI: https://doi.org/10.1016/j.cortex.2015.03.022

Chen, L., Benjamin, R., Guo, Y., Lai, A., & Heine, S. J. (2023). Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature. https://doi.org/10.21203/rs.3.rs-1254756/v1 DOI: https://doi.org/10.21203/rs.3.rs-1254756/v1

Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341X.2000.00455.x DOI: https://doi.org/10.1111/j.0006-341X.2000.00455.x

Efendic, E., Bartoš, F., Vranka, M. A., & Bahník, Š. (2019). Unpacking the justifiability of dishonesty: Behavioral and process-tracing investigation [Preprint at https://psyarxiv.com/rn85h]. DOI: https://doi.org/10.31234/osf.io/rn85h

Fanelli, D. (2010). "Positive" results increase down the hierarchy of the sciences. PloS One, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068 DOI: https://doi.org/10.1371/journal.pone.0010068

Fanelli, D., Costas, R., & Ioannidis, J. P. (2017). Meta-assessment of bias in science. Proceedings of the National Academy of Sciences, 114(14), 3714–3719. https://doi.org/10.1073/pnas.1618569114 DOI: https://doi.org/10.1073/pnas.1618569114

Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PloS One, 9(10), 1–12. https://doi.org/10.1371/journal.pone.0109019 DOI: https://doi.org/10.1371/journal.pone.0109019

Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348. http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

Gerlach, P., Teodorescu, K., & Hertwig, R. (2019). The truth about lies: A meta-analysis on dishonest behavior. Psychological Bulletin, 145(1), 1–44. https://doi.org/10.1037/bul0000174 DOI: https://doi.org/10.1037/bul0000174

Gupta, A., & Bosco, F. (2023). Tempest in a teacup: An analysis of p-Hacking in organizational research. PloS One, 18(2), e0281938. https://doi.org/10.1371/journal.pone.0281938 DOI: https://doi.org/10.1371/journal.pone.0281938

Held, L., Micheloud, C., & Pawel, S. (2022). The assessment of replication success based on relative effect size. The Annals of Applied Statistics, 16(2), 706–720. https://doi.org/10.1214/21-AOAS1502 DOI: https://doi.org/10.1214/21-AOAS1502

Hoekstra, R., & Vazire, S. (2021). Aspiring to greater intellectual humility in science. Nature Human Behaviour, 5(12), 1602–1607. https://doi.org/10.1038/s41562-021-01203-8 DOI: https://doi.org/10.1038/s41562-021-01203-8

Hoogeveen, S., Berkhout, S. W., Gronau, Q. F., Wagenmakers, E.-J., & Haaf, J. M. (2023). Improving statistical analysis in team science: The case of a Bayesian multiverse of Many Labs 4. https://doi.org/10.31234/osf.io/cb9er DOI: https://doi.org/10.1177/25152459231182318

Ioannidis, J. P., Stanley, T. D., & Doucouliagos, H. (2017). The power of bias in economics research. The Economic Journal, 127(605), F236–F265. https://doi.org/10.1111/ecoj.12461 DOI: https://doi.org/10.1111/ecoj.12461

Kristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., & Ariely, D. (2020). Signing at the beginning versus at the end does not decrease dishonesty. Proceedings of the National Academy of Sciences, 117(13), 7103–7107. https://doi.org/10.1073/pnas.1911695117 DOI: https://doi.org/10.1073/pnas.1911695117

Kvarven, A., Strømland, E., & Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4(4), 423–434. https://doi.org/10.1038/s41562-019-0787-z DOI: https://doi.org/10.1038/s41562-019-0787-z

Ly, A., Etz, A., Marsman, M., & Wagenmakers, E.-J. (2019). Replication Bayes factors from evidence updating. Behavior Research Methods, 51(6), 2498–2508. https://doi.org/10.3758/s13428-018-1092-x DOI: https://doi.org/10.3758/s13428-018-1092-x

Maier, M., Bartoš, F., Stanley, T. D., Shanks, D., Harris, A. J., & Wagenmakers, E.-J. (2022). No evidence for nudging after adjusting for publication bias. Proceedings of the National Academy of Sciences, 119(31). https://doi.org/10.1073/pnas.2200300119 DOI: https://doi.org/10.1073/pnas.2200300119

Maier, M., Bartoš, F., & Wagenmakers, E.-J. (2023). Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Psychological Methods, 28(1), 107–122. https://doi.org/10.1037/met0000405 DOI: https://doi.org/10.1037/met0000405

McAuliffe, W. H., Edson, T. C., Louderback, E. R., LaRaja, A., & LaPlante, D. A. (2021). Responsible product design to mitigate excessive gambling: A scoping review and z-curve analysis of replicability. PLoS One, 16(4), e0249926. https://doi.org/10.1371/journal.pone.0249926 DOI: https://doi.org/10.1371/journal.pone.0249926

McKay, B., Bacelar, M. F., Parma, J. O., Miller, M. W., & Carter, M. J. (2023). The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: A meta-analysis. International Review of Sport and Exercise Psychology, 1–21. https://doi.org/10.1080/1750984X.2023.2207255 DOI: https://doi.org/10.1080/1750984X.2023.2207255

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science%20.aac4716 DOI: https://doi.org/10.1126/science.aac4716

Pawel, S., & Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3), 879–911. https://doi.org/10.1111/rssb.12491 DOI: https://doi.org/10.1111/rssb.12491

Proceedings of the National Academy of Sciences. (2021). Retraction for 'Shu et al., Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end'. Proceedings of the National Academy of Sciences, 118(38), 1–1. https://doi.org/10.1073/pnas.2115397118 DOI: https://doi.org/10.1073/pnas.2115397118

Prochazka, J., Fedoseeva, Y., & Houdek, P. (2021). A field experiment on dishonesty: A registered replication of Azar et al. (2013). Journal of Behavioral and Experimental Economics, 90, 101617. https://doi.org/10.1016/j.socec.2020.101617 DOI: https://doi.org/10.1016/j.socec.2020.101617

Psychological Science. (2023a). Retraction notice to 'Evil genius? How dishonesty can lead to greater creativity'. Psychological Science, 34(8), 947–947. https://doi.org/10.1177/09567976231187595 DOI: https://doi.org/10.1177/09567976231187595

Psychological Science. (2023b). Retraction notice to 'The moral virtue of authenticity: How inauthenticity produces feelings of immorality and impurity'. Psychological Science, 34(8), 948–948. https://doi.org/10.1177/09567976231187596 DOI: https://doi.org/10.1177/09567976231187596

R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. https://www.R-project.org/

Renkewitz, F., & Keiner, M. (2019). How to detect publication bias in psychological research. Zeitschrift für Psychologie, 227(4), 261–279. https://doi.org/10.1027/2151-2604/a000386 DOI: https://doi.org/10.1027/2151-2604/a000386

Riesthuis, P., Otgaar, H., Bogaard, G., & Mangiulli, I. (2023). Factors affecting the forced confabulation effect: A meta-analysis of laboratory studies. Memory, 31(5), 635–651. https://doi.org/10.1080/09658211.2023.2185931 DOI: https://doi.org/10.1080/09658211.2023.2185931

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638 DOI: https://doi.org/10.1037//0033-2909.86.3.638

Schimmack, U. (2014). The test of insufficient variance (TIVA): A new tool for the detection of questionable research practices [Blogpost at https://replicationindex.com/2014/12/30/tiva/]. https://replicationindex.com/2014/12/30/tiva/

Schimmack, U., & Bartoš, F. (n.d.). Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values. PLoS ONE, 18(8), 1–12. https://doi.org/10.1371/journal.pone.0290084 DOI: https://doi.org/10.1371/journal.pone.0290084

Schwab, S., Kreiliger, G., & Held, L. (2021). Assessing treatment effects and publication bias across different specialties in medicine: A meta-epidemiological study. BMJ Open, 11(9), e045942. https://doi.org/10.1136/bmjopen-2020-045942 DOI: https://doi.org/10.1136/bmjopen-2020-045942

Sorić, B. (1989). Statistical "discoveries" and effect-size estimation. Journal of the American Statistical Association, 84(406), 608–610. https://doi.org/10.2307/2289950 DOI: https://doi.org/10.1080/01621459.1989.10478811

Sotola, L. K., & Credé, M. (2022). On the predicted replicability of two decades of experimental research on system justification: A z-curve analysis. European Journal of Social Psychology, 52(5-6), 895–909. https://doi.org/10.1002/ejsp.2858 DOI: https://doi.org/10.1002/ejsp.2858

Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018). What meta-analyses reveal about the replicability of psychological research. Psychological Bulletin, 144(12), 1325–1346. https://doi.org/10.1037/bul0000169 DOI: https://doi.org/10.1037/bul0000169

Stanley, T. D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 8(1), 60–78. https://doi.org/10.1002/jrsm.1095 DOI: https://doi.org/10.1002/jrsm.1095

Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54(285), 30–34. https://doi.org/10.1080/01621459.1959.10501497 DOI: https://doi.org/10.1080/01621459.1959.10501497

Stern, J., Arslan, R. C., Gerlach, T. M., & Penke, L. (2019). No robust evidence for cycle shifts in preferences for men’s bodies in a multiverse analysis: A response to Gangestad, Dinh, Grebe, Del Giudice, and Emery Thompson (2019). Evolution and Human Behavior, 40(6), 517–525. https://doi.org/10.1016/j.evolhumbehav.2019.08.005 DOI: https://doi.org/10.1016/j.evolhumbehav.2019.08.005

Syrjänen, E., Fischer, H., Liuzza, M. T., Lindholm, T., & Olofsson, J. K. (2021). A review of the effects of valenced odors on face perception and evaluation. i-Perception, 12(2), 1–19. https://doi.org/10.1177/20416695211009552 DOI: https://doi.org/10.1177/20416695211009552

van der Cruyssen, I., D’hondt, J., Meijer, E., & Verschuere, B. (2020). Does honesty require time? Two preregistered direct replications of experiment 2 of Shalvi, Eldar, and Bereby-Meyer (2012). Psychological Science, 31(4), 460–467. https://doi.org/10.1177/0956797620903716 DOI: https://doi.org/10.1177/0956797620903716

van Aert, R. C., Wicherts, J. M., & Van Assen, M. A. (2019). Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis. PloS One, 14(4), e0215052. https://doi.org/10.1371/journal.pone.0215052 DOI: https://doi.org/10.1371/journal.pone.0215052

van Anen, A. (2022). How strong is our evidence? Evidential value and publication bias in research on social media use and self-esteem [Master’s thesis]. Tilburg University. http://arno.uvt.nl/show.cgi?fid=158963

Verschuere, B., Meijer, E. H., Jim, A., Hoogesteyn, K., Orthey, R., McCarthy, R. J., Skowronski, J. J., Acar, O. A., Aczel, B., Bakos, B. E., et al. (2018). Registered replication report on Mazar, Amir, and Ariely (2008). Advances in Methods and Practices in Psychological Science, 1(3), 299–317. https://doi.org/10.1177/2515245918781032 DOI: https://doi.org/10.1177/2515245918781032

Vevea, J. L., & Hedges, L. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60(3), 419–435. https://doi.org/10.1007/BF02294384 DOI: https://doi.org/10.1007/BF02294384

Wagenmakers, E.-J., Sarafoglou, A., & Aczel, B. (2022). One statistical analysis must not rule them all. Nature, 605(7910), 423–425. https://doi.org/10.1038/d41586-022-01332-8 DOI: https://doi.org/10.1038/d41586-022-01332-8

Wouda, J., Bijlstra, G., Frankenhuis, W. E., & Wigboldus, D. H. (2017). The collaborative roots of corruption? A replication of Weisel & Shalvi (2015). Collabra: Psychology, 3, 1–3. https://doi.org/10.1525/collabra.97 DOI: https://doi.org/10.1525/collabra.97