ReproduceMe: Lessons from a pilot project on computational reproducibility
DOI:
https://doi.org/10.15626/MP.2023.4021Keywords:
computational reproducibility, markdown, CI/CD, open scienceAbstract
If a scientific paper is computationally reproducible, the analyses it reports can be repeated independently by others. At the present time most papers are not reproducible. However, the tools to enable computational reproducibility are now widely available, using free and open source software. We conducted a pilot study in which we offered ‘reproducibility as a service’ within a UK psychology department for a period of 6 months. Our rationale was that most researchers lack either the time or expertise to make their own work reproducible, but might be willing to allow this to be done by an independent team. Ten papers were converted into reproducible format using R markdown, such that all analyses were conducted by a single script that could download raw data from online platforms as required, generate figures, and produce a pdf of the final manuscript. For some studies this involved reproducing analyses originally conducted using commercial software. The project was an overall success, with strong support from the contributing authors who saw clear benefit from this work, including greater transparency and openness, and ease of use for the reader. Here we describe our framework for reproducibility, summarise the specific lessons learned during the project, and discuss the future of computational reproducibility. Our view is that computationally reproducible manuscripts embody many of the core principles of open science, and should become the default format for scientific communication.
Metrics
References
Abbasi, K. (2023). A commitment to act on data sharing. BMJ, p1609. DOI: https://doi.org/10.1136/bmj.p1609
Baker, D. H. (2021). Statistical analysis of periodic data in neuroscience. Neurons, Behavior, Data Analysis, and Theory, 5(3), 1–18. DOI: https://doi.org/10.51628/001c.27680
Baker, D. H., Vilidaite, G., & Wade, A. R. (2021). Steady-state measures of visual suppression. PLOS Computational Biology, 17(10), e1009507. DOI: https://doi.org/10.1371/journal.pcbi.1009507
Baxter, N., & Hobson, H. (2024). The role of emotional factors in face processing abilities in autism spectrum conditions. Research in Autism Spectrum Disorders, 115, 102400. DOI: https://doi.org/10.1016/j.rasd.2024.102400
Brennan, R. A., Enock, F. E., & Over, H. (2024). Attribution of undesirable character traits, rather than trait-based dehumanization, predicts punishment decisions. R Soc Open Sci, 11(7), 240087. DOI: https://doi.org/10.1098/rsos.240087
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1). DOI: https://doi.org/10.18637/jss.v076.i01
Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L. (2023). What’s in a badge? A computational reproducibility investigation of the open data badge policy in one issue of Psychological Science. Psychological Science, 34(4), 512–522. DOI: https://doi.org/10.1177/09567976221140828
de Bruin, A., Kressel, H., & Hemmings, D. (2023). A comparison of language control while switching within versus between languages in younger and older adults. Sci Rep, 13(1), 16740. DOI: https://doi.org/10.1038/s41598-023-43886-1
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., Oya, H., Ghosh, S. S., Wright, J., Durnez, J., Poldrack, R. A., & Gorgolewski, K. J. (2019). fMRIPrep: A robust preprocessing pipeline for functional MRI. Nature Methods, 16(1), 111–116. DOI: https://doi.org/10.1038/s41592-018-0235-4
Fiske, S. (2016). A call to change science’s culture of shaming. Observer, 29(9), 5–11. https://www.psychologicalscience.org/observer/a-call-to-change-sciences-culture-of-shaming
Fox, J., & Weisberg, S. (2019). An R companion to applied regression (3rd ed.). Sage. https://socialsciences.mcmaster.ca/jfox/Books/Companion/
Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., Flandin, G., Ghosh, S. S., Glatard, T., Halchenko, Y. O., Handwerker, D. A., Hanke, M., Keator, D., Li, X., Michael, Z., Maumet, C., Nichols, B. N., Nichols, T. E., Pellman, J., ... Poldrack, R. A. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3(1), 160044. DOI: https://doi.org/10.1038/sdata.2016.44
Grigoryan, L., Ponizovskiy, V., Weißflog, M. I., Osin, E., & Lickel, B. (2024). Guilt, shame, and antiwar action in an authoritarian country at war. Political Psychology. DOI: https://doi.org/10.1111/pops.12985
Hansford, K. J., Baker, D. H., McKenzie, K. J., & Preston, C. E. (2024). Multisensory processing and proprioceptive plasticity during resizing illusions. Experimental Brain Research, 242, 451–462. DOI: https://doi.org/10.1007/s00221-023-06759-7
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021). Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study. Royal Society Open Science, 8(1), 201494. DOI: https://doi.org/10.1098/rsos.201494
Imbens, G. W., & Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475. DOI: https://doi.org/10.2307/2951620
Larkin, F., Oostenbroek, J., Lee, Y., Hayward, E., Fernandez, A., Wang, Y., Mitchell, A., Li, L. Y., & Meins, E. (2024). A smartphone app effectively facilitates mothers' mind-mindedness: A randomized controlled trial. Child Development, 95(3), 831–844. DOI: https://doi.org/10.1111/cdev.14039
Lee, Y. J., Meins, E., & Larkin, F. (2023). Parental mentalization across cultures: Mind-mindedness and parental reflective functioning in British and South Korean mothers. PsyArXiv. DOI: https://doi.org/10.31234/osf.io/qx9mh
Mak, M. H. C., O’Hagan, A., Horner, A. J., & Gaskell, M. G. (2023). A registered report testing the effect of sleep on Deese-Roediger-McDermott false memory: Greater lure and veridical recall but fewer intrusions after sleep. Royal Society Open Science, 10(12), 220595. DOI: https://doi.org/10.1098/rsos.220595
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018). DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9), 1281–1289. DOI: https://doi.org/10.1038/s41593-018-0209-y
Meese, T. S., & Baker, D. H. (2023). Object image size is a fundamental coding dimension in human vision: New insights and model. Neuroscience, 514, 79–91. DOI: https://doi.org/10.1016/j.neuroscience.2023.01.025
Nuijten, M. B., & Wicherts, J. M. (2024). Implementing statcheck during peer review is related to a steep decline in statistical-reporting inconsistencies. Advances in Methods and Practices in Psychological Science, 7(2). DOI: https://doi.org/10.1177/25152459241258945
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of open data and computational reproducibility in registered reports in psychology. Advances in Methods and Practices in Psychological Science, 3(2), 229–237. DOI: https://doi.org/10.1177/2515245920918872
Peikert, A., & Brandmaier, A. M. (2021). A reproducible data analysis workflow. Quantitative and Computational Methods in Behavioral Sciences, 1, e3763. DOI: https://doi.org/10.5964/qcmb.3763
Rouder, J. N., & Haaf, J. M. (2018). Power, dominance, and constraint: A note on the appeal of different design traditions. Advances in Methods and Practices in Psychological Science, 1(1), 19–26. DOI: https://doi.org/10.1177/2515245917745058
Segala, F. G., Bruno, A., Martin, J. T., Aung, M. T., Wade, A. R., & Baker, D. H. (2023). Different rules for binocular combination of luminance flicker in cortical and subcortical pathways. eLife, 12, RP87048. DOI: https://doi.org/10.7554/eLife.87048
Venables, W., & Ripley, B. (2002). Modern applied statistics with S (4th ed.). Springer. https://www.stats.ox.ac.uk/pub/MASS4/ DOI: https://doi.org/10.1007/978-0-387-21706-2
Wiebels, K., & Moreau, D. (2021). Leveraging containers for reproducible psychological research. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211017853. DOI: https://doi.org/10.1177/25152459211017853
Wolen, A., Hartgerink, C., Hafen, R., Richards, B., Soderberg, C., & York, T. (2020). Osfr: An R interface to the Open Science Framework. Journal of Open Source Software, 5(46), 2071. DOI: https://doi.org/10.21105/joss.02071
Published
Issue
Section
License
Copyright (c) 2024 Daniel H. Baker, Mareike Berg, Kirralise J. Hansford, Bartholomew P.A. Quinn, Federico G. Segala, Erin L. Warden-English
This work is licensed under a Creative Commons Attribution 4.0 International License.
