ReproduceMe: Lessons from a pilot project on computational reproducibility

Authors

DOI:

https://doi.org/10.15626/MP.2023.4021

Keywords:

computational reproducibility, markdown, CI/CD, open science

Abstract

If a scientific paper is computationally reproducible, the analyses it reports can be repeated independently by others. At the present time most papers are not reproducible. However, the tools to enable computational reproducibility are now widely available, using free and open source software. We conducted a pilot study in which we offered ‘reproducibility as a service’ within a UK psychology department for a period of 6 months. Our rationale was that most researchers lack either the time or expertise to make their own work reproducible, but might be willing to allow this to be done by an independent team. Ten papers were converted into reproducible format using R markdown, such that all analyses were conducted by a single script that could download raw data from online platforms as required, generate figures, and produce a pdf of the final manuscript. For some studies this involved reproducing analyses originally conducted using commercial software. The project was an overall success, with strong support from the contributing authors who saw clear benefit from this work, including greater transparency and openness, and ease of use for the reader. Here we describe our framework for reproducibility, summarise the specific lessons learned during the project, and discuss the future of computational reproducibility. Our view is that computationally reproducible manuscripts embody many of the core principles of open science, and should become the default format for scientific communication.

Metrics

Metrics Loading ...

References

Abbasi, K. (2023). A commitment to act on data sharing. BMJ, p1609. DOI: https://doi.org/10.1136/bmj.p1609

Baker, D. H. (2021). Statistical analysis of periodic data in neuroscience. Neurons, Behavior, Data Analysis, and Theory, 5(3), 1–18. DOI: https://doi.org/10.51628/001c.27680

Baker, D. H., Vilidaite, G., & Wade, A. R. (2021). Steady-state measures of visual suppression. PLOS Computational Biology, 17(10), e1009507. DOI: https://doi.org/10.1371/journal.pcbi.1009507

Baxter, N., & Hobson, H. (2024). The role of emotional factors in face processing abilities in autism spectrum conditions. Research in Autism Spectrum Disorders, 115, 102400. DOI: https://doi.org/10.1016/j.rasd.2024.102400

Brennan, R. A., Enock, F. E., & Over, H. (2024). Attribution of undesirable character traits, rather than trait-based dehumanization, predicts punishment decisions. R Soc Open Sci, 11(7), 240087. DOI: https://doi.org/10.1098/rsos.240087

Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1). DOI: https://doi.org/10.18637/jss.v076.i01

Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L. (2023). What’s in a badge? A computational reproducibility investigation of the open data badge policy in one issue of Psychological Science. Psychological Science, 34(4), 512–522. DOI: https://doi.org/10.1177/09567976221140828

de Bruin, A., Kressel, H., & Hemmings, D. (2023). A comparison of language control while switching within versus between languages in younger and older adults. Sci Rep, 13(1), 16740. DOI: https://doi.org/10.1038/s41598-023-43886-1

Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., Oya, H., Ghosh, S. S., Wright, J., Durnez, J., Poldrack, R. A., & Gorgolewski, K. J. (2019). fMRIPrep: A robust preprocessing pipeline for functional MRI. Nature Methods, 16(1), 111–116. DOI: https://doi.org/10.1038/s41592-018-0235-4

Fiske, S. (2016). A call to change science’s culture of shaming. Observer, 29(9), 5–11. https://www.psychologicalscience.org/observer/a-call-to-change-sciences-culture-of-shaming

Fox, J., & Weisberg, S. (2019). An R companion to applied regression (3rd ed.). Sage. https://socialsciences.mcmaster.ca/jfox/Books/Companion/

Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., Flandin, G., Ghosh, S. S., Glatard, T., Halchenko, Y. O., Handwerker, D. A., Hanke, M., Keator, D., Li, X., Michael, Z., Maumet, C., Nichols, B. N., Nichols, T. E., Pellman, J., ... Poldrack, R. A. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3(1), 160044. DOI: https://doi.org/10.1038/sdata.2016.44

Grigoryan, L., Ponizovskiy, V., Weißflog, M. I., Osin, E., & Lickel, B. (2024). Guilt, shame, and antiwar action in an authoritarian country at war. Political Psychology. DOI: https://doi.org/10.1111/pops.12985

Hansford, K. J., Baker, D. H., McKenzie, K. J., & Preston, C. E. (2024). Multisensory processing and proprioceptive plasticity during resizing illusions. Experimental Brain Research, 242, 451–462. DOI: https://doi.org/10.1007/s00221-023-06759-7

Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021). Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study. Royal Society Open Science, 8(1), 201494. DOI: https://doi.org/10.1098/rsos.201494

Imbens, G. W., & Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475. DOI: https://doi.org/10.2307/2951620

Larkin, F., Oostenbroek, J., Lee, Y., Hayward, E., Fernandez, A., Wang, Y., Mitchell, A., Li, L. Y., & Meins, E. (2024). A smartphone app effectively facilitates mothers' mind-mindedness: A randomized controlled trial. Child Development, 95(3), 831–844. DOI: https://doi.org/10.1111/cdev.14039

Lee, Y. J., Meins, E., & Larkin, F. (2023). Parental mentalization across cultures: Mind-mindedness and parental reflective functioning in British and South Korean mothers. PsyArXiv. DOI: https://doi.org/10.31234/osf.io/qx9mh

Mak, M. H. C., O’Hagan, A., Horner, A. J., & Gaskell, M. G. (2023). A registered report testing the effect of sleep on Deese-Roediger-McDermott false memory: Greater lure and veridical recall but fewer intrusions after sleep. Royal Society Open Science, 10(12), 220595. DOI: https://doi.org/10.1098/rsos.220595

Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018). DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9), 1281–1289. DOI: https://doi.org/10.1038/s41593-018-0209-y

Meese, T. S., & Baker, D. H. (2023). Object image size is a fundamental coding dimension in human vision: New insights and model. Neuroscience, 514, 79–91. DOI: https://doi.org/10.1016/j.neuroscience.2023.01.025

Nuijten, M. B., & Wicherts, J. M. (2024). Implementing statcheck during peer review is related to a steep decline in statistical-reporting inconsistencies. Advances in Methods and Practices in Psychological Science, 7(2). DOI: https://doi.org/10.1177/25152459241258945

Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of open data and computational reproducibility in registered reports in psychology. Advances in Methods and Practices in Psychological Science, 3(2), 229–237. DOI: https://doi.org/10.1177/2515245920918872

Peikert, A., & Brandmaier, A. M. (2021). A reproducible data analysis workflow. Quantitative and Computational Methods in Behavioral Sciences, 1, e3763. DOI: https://doi.org/10.5964/qcmb.3763

Rouder, J. N., & Haaf, J. M. (2018). Power, dominance, and constraint: A note on the appeal of different design traditions. Advances in Methods and Practices in Psychological Science, 1(1), 19–26. DOI: https://doi.org/10.1177/2515245917745058

Segala, F. G., Bruno, A., Martin, J. T., Aung, M. T., Wade, A. R., & Baker, D. H. (2023). Different rules for binocular combination of luminance flicker in cortical and subcortical pathways. eLife, 12, RP87048. DOI: https://doi.org/10.7554/eLife.87048

Venables, W., & Ripley, B. (2002). Modern applied statistics with S (4th ed.). Springer. https://www.stats.ox.ac.uk/pub/MASS4/ DOI: https://doi.org/10.1007/978-0-387-21706-2

Wiebels, K., & Moreau, D. (2021). Leveraging containers for reproducible psychological research. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211017853. DOI: https://doi.org/10.1177/25152459211017853

Wolen, A., Hartgerink, C., Hafen, R., Richards, B., Soderberg, C., & York, T. (2020). Osfr: An R interface to the Open Science Framework. Journal of Open Source Software, 5(46), 2071. DOI: https://doi.org/10.21105/joss.02071

Downloads

Published

2024-09-06

Issue

Section

Original articles