The Evolution of Data Sharing Practices in the Psychological Literature

Judith Neve; Guillaume Rousselet

doi:10.15626/MP.2021.2908

Downloads

PDF

Authors

Judith Neve University of Glasgow https://orcid.org/0000-0002-1168-0182
Guillaume A. Rousselet University of Glasgow https://orcid.org/0000-0003-0006-8729

DOI:

https://doi.org/10.15626/MP.2021.2908

Keywords:

journal policies, metascience, open science, reproducibility, scientific publishing

Abstract

Sharing data has many benefits. However, data sharing rates remain low, for the most part well below 50%. A variety of interventions encouraging data sharing have been proposed. We focus here on editorial policies. Kidwell et al. (2016) assessed the impact of the introduction of badges in Psychological Science; Hardwicke, Mathur, et al. (2018) assessed the impact of Cognition’s mandatory data sharing policy. Both studies found policies to improve data sharing practices, but only assessed the impact of the policy for up to 25 months after its implementation. We examined the effect of these policies over a longer term by reusing their data and collecting a follow-up sample including articles published up until December 31st, 2019. We fit generalized additive models as these allow for a flexible assessment of the effect of time, in particular to identify nonlinear changes in the trend. These models were compared to generalized linear models to examine whether the non-linearity is needed. Descriptive results and the outputs from generalized additive and linear models were coherent with previous findings: following the policies in Cognition and Psychological Science, data sharing statement rates increased immediately and continued to increase beyond the timeframes examined previously, until reaching close to 100%. In Clinical Psychological Science, data sharing statement rates started to increase only two years following the implementation of badges. Reusability rates jumped from close to 0% to around 50% but did not show changes within the pre-policy nor the post-policy timeframes. Journals that did not implement a policy showed no change in data sharing rates or reusability over time. There was variability across journals in the levels of increase, so we suggest future research should examine a larger number of policies to draw conclusions about their efficacy. We also encourage future research to investigate the barriers to data sharing specific to psychology subfields to identify the best interventions to tackle them.

Metrics

Metrics Loading ...

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.

Alkharusi, H. (2012). Categorical Variables in Regression Analysis: A Comparison of Dummy and Effect Coding. International Journal of Education, 4, 202–210.

Asendorpf, J. B., Conner, M., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., Perugini, M., Roberts, B. W., Schmitt, M., van Aken, M. A. G., Weber, H., & Wicherts, J. M. (2013). Recommendations for Increasing Replicability in Psychology. European Journal of Personality, 27(2), 108–119.

Audette, L. M., Hammond, M. S., & Rochester, N. K. (2020). Methodological Issues With Coding Participants in Anonymous Psychological Longitudinal Studies. Educational and Psychological Measurement, 80(1), 163–185.

Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7(6), 543–554.

Baldwin, S. A., & Del Re, A. C. (2016). Open Access Meta-Analysis for Psychotherapy Research. Journal of Counseling Psychology, 63(3), 249–260.

Björk, B.-C., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7(4), 914–923.

Chakrabarti, A., & Ghosh, J. K. (2011). AIC, BIC and Recent Advances in Model Selection. In P. S. Bandyopadhyay & M. R. Forster (Eds.), Philosophy of Statistics (pp. 583–605, Vol. 7). North-Holland.

Choudhury, S., Fishman, J. R., McGowan, M. L., & Juengst, E. T. (2014). Big data, open science and the brain: Lessons learned from genomics. Frontiers in Human Neuroscience, 8.

Deschaght, P., Vintém, A. P., Logghe, M., Conde, M., Felix, D., Mensink, R., Gonçalves, J., Audiens, J., Bruynooghe, Y., Figueiredo, R., Ramos, D., Tanghe, R., Teixeira, D., Van de Ven, L., Stortelers, C., & Dombrecht, B. (2017). Large Diversity

of Functional Nanobodies from a Camelid Immune Library Revealed by an Alternative Analysis of Next-Generation Sequencing Data. Frontiers in Immunology, 8.

Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016). Scientists’ Reputations Are Based on Getting It Right, Not Being Right. Plos Biology, 14(5), e1002460.

Eich, E. (2014). Business Not as Usual. Psychological Science, 25(1), 3–6.

Ethical principles of psychologists and code of conduct. (2017). https://www.apa.org. Retrieved December 5, 2020, from https://www.apa.org/ethics/code

Final NIH Policy for Data Management and Sharing. (2020). Retrieved December 9, 2020, from https://grants.nih.gov/grants/guide/noticefiles/NOT-OD-21-013.html

Forstmeier, W., Wagenmakers, E.-J., & Parker, T. H. (2017). Detecting and avoiding likely falsepositive findings – a practical guide. Biological Reviews, 92(4), 1941–1968.

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505.

Grolemond, G., &Wickman, H. (2011). Dates and Times Made Easy with lubridate. Journal of Statistical Software, 40(3), 1–25.

Hanson, B., Sugden, A., & Alberts, B. (2011). Making Data Maximally Available. Science, 331(6018), 649–649.

Hardwicke, T. E., Mathur, M., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., & Tessler, M. H. (2018a). Materials. Retrieved March 18, 2021, from https://osf.io/k2mdr/

Hardwicke, T. E., Mathur, M., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., & Tessler, M. H. (2018b). Data. Retrieved March 18, 2021, from https://osf.io/6s8b3/

Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Tessler, M. H., Lenne, R. L., Altman, S., Long, B., & Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science, 5(8).

Hastie, T., & Tibshirani, R. (1987). Generalized Additive Models: Some Applications. Journal of the American Statistical Association, 82(398), 371–386.

Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018). Data Sharing in Psychology: A Survey on Barriers and Preconditions. Advances in Methods and Practices in Psychological Science, 1(1), 70–85.

Ioannidis, J. P. A. (2014). How to Make More Published Research True. Plos Medicine, 11(10), e1001747.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524–532.

Jones, K., & Almond, S. (1992). Moving Out of the Linear Rut: The Possibilities of Generalized Additive Models. Transactions (Institute of British Geographers), 17, 434–447.

Kidwell, M. C., Lazarevic, L., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., & Hess-Holden, C. L. (2015a). Coding the Articles: Materials. Retrieved March 18, 2021, from https://osf.io/ 8kt4b/

Kidwell, M. C., Lazarevic, L., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., & Hess-Holden, C. L. (2015b). Data. Retrieved March 18, 2021, from https://osf.io/u6g7t/

Kidwell, M. C., Lazarevi´c, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., & Nosek, B. A. (2016). Badges to Acknowledge Open Practices:

A Simple, Low-Cost, Effective Method for Increasing Transparency. PLOS Biology, 14(5), e1002456.

Lilienfeld, S. O. (2017). Clinical Psychological Science: Then and Now. Clinical Psychological Science, 5(1), 3–13.

Lindsay, D. S. (2017). Sharing Data and Materials in Psychological Science: Psychological Science.

Longo, D. L., & Drazen, J. M. (2016). Data Sharing. New England Journal of Medicine, 374(3), 276–277.

Marks, D. F. (2020). Increasing the transparency, openness and replicability of psychological research: Mandatory data sharing for empirical studies in the Journal of Health Psychology. Journal of Health Psychology, 25(6), 729–732.

Martone, M. E., Garcia-Castro, A., & VandenBos, G. R. (2018). Data Sharing in Psychology. The American psychologist, 73(2), 111–125.

McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., Spies, J. R., Thaney, K., Updegrove, A., Woo, K. H., & Yarkoni, T. (2016). How open science helps researchers succeed. Elife, 5, e16800.

Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., Vanpaemel, W., Wagenmakers, E.-J., & Zwaan, R. A. (2016). The Peer Reviewers’ Openness Initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3(1), 150547.

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., du Sert, N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 0021.

Nielson, J. L., Guandique, C. F., Liu, A. W., Burke, D. A., Lash, A. T., Moseanko, R., Hawbecker, S., Strand, S. C., Zdunowski, S., Irvine, K.-A., Brock, J. H., Nout-Lomas, Y. S., Gensel, J. C., Anderson, K. D., Segal, M. R., Rosenzweig, E. S., Magnuson, D. S. K., Whittemore, S. R., McTigue, D. M., . . . Ferguson, A. R. (2014). Development of a Database for Translational Spinal Cord Injury Research. Journal of Neurotrauma, 31(21), 1789–1799.

Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology. Advances in Methods and Practices in Psychological Science, 2515245920918872.

O’Brien, R. M. (2012). Visualizing Rank Deficient Models: A Row Equation Geometry of Rank Deficient Matrices and Constrained-Regression. PLoS ONE, 7(6).

Perrino, T., Beardslee, W., Bernal, G., Brincks, A., Cruden, G., Howe, G., Murry, V., Pantin, H., Prado, G., Sandler, I., & Brown, C. H. (2015).

Toward Scientific Equity for the Prevention of Depression and Depressive Symptoms in Vulnerable Youth. Prevention Science, 16(5), 642–651. https://doi.org/10.1007/s11121- 014-0518-7

Piwowar, H. A., & Vision, T. J. (2013). Data reuse and the open data citation advantage. Peerj, 1, e175.

Rowhani-Farid, A., & Barnett, A. G. (2018). Badges for sharing data and code at Biostatistics: An observational study.

Schwarz, G. (1978). Estimating the Dimension of a Model. Annals of Statistics, 6(2), 461–464.

Science Journals: Editorial policies. (2018). Science | AAAS. Retrieved December 5, 2020, from https://www. sciencemag.org/authors/science-journals-editorial-policies

Simpson, G. L. (2018). Modelling Palaeoecological Time Series Using Generalised Additive Models. Frontiers in Ecology and Evolution, 6, 149.

Sloman, S. A. (2015). Opening editorial: The changing face of Cognition. Cognition, 135, 1–3.

Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American statistical association, 54(285), 30–34.

Stockemer, D., Koehler, S., & Lentz, T. (2018). Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature. PS: Political Science & Politics, 51(4), 799–803.

Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589.

Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017). It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science. Perspectives on Psychological Science, 12(5), 742–756.

R Core Team (2020). R: A Language and Environment for Statistical Computing. Vienna, Austria.

Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015). Are We Wasting a Good Crisis? The Availability of Psychological Research Data after the Storm. Collabra: Psychology, 1(1), Art. 3.

Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J.-S., Renaut, S., & Rennison, D. J. (2014). The Availability of Research Data Declines Rapidly with Article Age. Current Biology, 24(1), 94–97.

Wang, B., Ding, X., &Wang, F.-Y. (2017). Determination of Polynomial Degree in the Regression of Drug Combinations. Ieee-Caa Journal of Automatica Sinica, 4(1), 41–47.

Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V.,. . . Yutani, H. (2019). Welcome to the Tidyverse. Journal of Open Source Software, 4(43), 1686.

Wood, S. N. (2017). Generalized Additive Models: An Introduction with R, Second Edition. CRC Press. Zoltowski, D. M., & Pillow, J. W. (2018). Scaling the Poisson GLM to massive neural datasets through polynomial approximations. Advances in neural information processing systems, 31, 3517–3527.

Zotero. (2020). https://Zotero.org