The Open Empirical Cycle for Hypothesis Evaluation in Psychology
DOI:
https://doi.org/10.15626/MP.2023.3963Keywords:
empirical cycle, FAIR data, open access, Preregistration, replication researchAbstract
In the last decade it has become clear that replicability of empirical psychological research should be better. Open science practices aim to enhance the transparency and accessibility of research thereby both enabling others to reproduce the results presented in a paper and increasing the replicability of these results. Examples of these practices include preregistration, publication of data and analyses, open access publications, and replication research. Although open science practices are gaining traction, they have rarely been placed in a broader epistemological context. To address this shortcoming, this paper introduces the open empirical cycle. It draws upon De Groot's empirical cycle, a model of cumulative knowledge generation via scientific research and adds open science practices. Adhering to the open empirical cycle, if only partly, will structure the scientific workflow and create awareness of the adverse consequences of deviations. Following the open empirical cycle increases the transparency, quality, trustworthiness, and replicability of research.
References
Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18, e3000937. https://doi.org/10.1371/journal.pbio.3000937
Bosnjak, M., Fieback, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., & Sokol-Chang, R. I. (2021). A template for preregistration of quantitative research in psychology: Report of the joint psychological societies preregistration task force. American Psychologist, 77, 602–615. https://eprints.whiterose.ac.uk/174294/
Bourne, P. E., Polka, J. K., Vale, R. D., & Kiley, R. (2017). Ten simple rules to consider regarding preprint submission. PLoS Computational Biology, 13. https://doi.org/10.1371/journal.pcbi. 1005473
Burkholder, G. J., Cox, K. A., Crawford, L. M., & Hitchcock, J. (2019). Research Design and Methods: An Applied Guide for the Researcher Practitioner. SAGE.
Chambers, C. D. (2019). What’s next for Registered Reports? Nature, 573, 187–189. https://doi.org/10.1038/d41586-019-02674-6
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. https://doi.org/10.1037/0033-2909.112.1.155
de Groot, A. D. (1961). Methodologie. Grondslagen van onderzoek en denken in de gedragswetenschappen [Methodology. Foundations of research and thinking in the behavioral sciences]. https://www.dbnl.org/tekst/groo004meth01_01/
de Groot, A. D., Wagenmakers, E.-J., Borsboom, D., Verhagen, J., Kievit, R., Bakker, M., Cramer, A., Matzke, D., Mellenbergh, D., & van der Maas, H. L. J. (2014). The meaning of "significance" for different types of research. Acta Psychologica, 148, 188–194. https://doi.org/10.1016/j.actpsy.2014.02.001
Duken, S. B., Keessen, L., Hoijtink, H., Kindt, M., & van Ast, V. A. (2021). Episodic and affective memory distortions in dysphoria: Bayesian testing of diverging theories [Registered Report Stage 1 Protocol]. Nature Communications. https://doi.org/10.6084/m9.figshare.14605374.v1
Duken, S. B., Keessen, L., Hoijtink, H., Kindt, M., & van Ast, V. A. (2024). Episodic and affective memory distortions in dysphoria: Bayesian testing of diverging theories. Nature Communications, 15, 1320. https://doi.org/10.1038/s41467- 024-45203-4
Elsey, J. W. B., Filmer, A. I., & Stemerding, L. E. (2021). Don’t Forget the Psychology in Analyses of Psychological Data: The Case of Sequential Testing. Collabra: Psychology, 7. https://doi.org/10.1525.collabra.24953
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Reproducibility in cancer biology: Challenges for assessing replicability in preclinical cancer biology. eLife. https://doi.org/10.7554/eLife.67995
Errington, T. M., Mathur, M. M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Investigating the replicability of preclinical cancer biology. eLife. https://doi.org/10.7554/eLife.71601
Flake, J. K., & Fried, E. I. (2020). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science, 3, 456–465. https://doi.org/10.1177/2515245920952393
Frankenhuis, W. E., & Nettle, D. (2018). Open science is liberating and can foster creativity. Perspectives on Psychological Science, 13, 439–447. https://doi.org/10.1177/1745691618767878
Frankenhuis, W. E., Panchanathan, K., & Smaldino, P. E. (2023). Strategic ambiguity in the social sciences. Social Psychological Bulletin, 18, e9923. https://doi.org/10.23668/psycharchives.8381
Fried, E. I. (2020). Theories and Models: What They Are, What They Are for, and What They Are About. Psychological Inquiry, 31, 336–344. https://doi.org/10.1080/1047840X.2020.1854011
Fu, D. Y., & Hughey, J. J. (2020). Meta-Research: Releasing a preprint is associated with more attention and citations for the peer-reviewed article. eLife, 8, e52646. https://doi.org/10.7554/eLife.52646
Gopalakrishna, G., ter Riet, G., Cruyff, M., Vink, G., Stoop, I., Wicherts, J., & Bouter, L. (2021). Prevalence of questionable research practices, research misconduct and their potential explanatory factors: a survey among academic researchers in The Netherlands. https://doi.org/10.1371/journal.pone.0263023
Höfler, M., Scherbaum, S., Kanske, P., McDonald, B., & Miller, R. (2022). Means to valuable exploration I. The blending of confirmation and exploration and how to resolve it. Meta-Psychology, 6. https://doi.org/10.15626/MP.2021.2837
Hoijtink, H., Mulder, J., van Lissa, C., & Gu, X. (2019). A tutorial on testing hypotheses using the Bayes factor. Psychological Methods, 24, 539–556. https://doi.org/10.1037/met0000201
Ioannides, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. https://doi.org/10.1371/journal.pmed.0020124
Lakens, D. (2022). Sample Size Justification. Collabra: Psychology, 8, 33267. https://doi.org/10.1525/collabra.33267
Makel, M., Hodges, J., & Cook, B. (2021). Both questionable and open research practices are prevalent in education research. Educational Researcher, 50, 493–504. https://doi.org/10.3102/0013189X211001356
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., Yarkoni, T., et al. (2016). How open science helps researchers succeed. eLife, 5, 1–19. https://doi.org/10.7554/eLife.16800
Nikolopoulou, K. (2022). What is publication bias? Definition and examples. https : / / www. scribbr.com/research-bias/publication-bias/
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., Vazire, S., et al. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23, 815–818. https://doi.org/10.1016/j.tics.2019.07.009
Nosek, B. A., & Errington, T. (2020). What is replication? PLOS Biology, 18, e3000691. https://doi.org/10.1371/journal.pbio.3000691
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660. https://doi.org/10.1177/1745691612462588
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349. https://doi.org/10.1126/science.aac4716
Peikert, A., & Brandmaier, A. M. (2021). A reproducible data analysis workflow with R Markdown, Git, Make, and Docker. Quantitative and Computational Methods in Behavioral Sciences, e3763. https://doi.org/10.5964/qcmb.3763
Peikert, A., Ernst, S., Brandmaier, & M., A. (2023). Why does preregistration increase the persuasiveness of evidence? A Bayesian rationalization. https://osf.io/preprints/psyarxiv/cs8wb_v2
Peikert, A., van Lissa, C. J., & Brandmaier, A. M. (2021). Reproducible research in R: A tutorial on how to do the same thing more than once. Psych, 3, 836–867. https://doi.org/10.3390/psych3040053
Rathenau Instituut. (2021). Vertrouwde wetenschap – een kwalitatieve studie naar het publieke vertrouwen in wetenschap en opdrachtonder-zoek. https://www.rathenau.nl/sites/default/files / 2021 - 06 / Rathenau _ Instituut _ Rapport _Vertrouwde_wetenschap.pdf
Rohrer, J. M., Tierney, W., Uhlmann, E. L., DeBruine, L. M., Heyman, T., Jones, B., Schmukle, S. C., Silberzahn, R., Willén, R. M., Carlsson, R., Lucas, R. E., Strand, J., Vazire, S., Witt, J. K., Zentall, T. R., Chabris, C. F., & Yarkoni, T. (2021).
Putting the self in self-correction: Findings from the loss-of-confidence project. Perspectives on Psychological Science, 16, 1255–1269. https://doi.org/10.1177/1745691620964106
Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Rouder, J. N. (2014). Optional stopping: No problem for bayesians. Psychonomic Bulletin & Review, 21, 301–308. https://link.springer.com/article/10.3758/s13423-014-0595-4
van Buuren, S. (2018). Flexible imputation of missing data. Chapman; Hall/CRC.
van Lissa, C. J. (2022). Complementing preregistered confirmatory analyses with rigorous, reproducible exploration using machine learning. Religion, Brain & Behavior, 13, 347–351. https://doi.org/10.1080/2153599X.2022.2070254
van Lissa, C. J., Brandmaier, A. M., B., V., et al. (2021). Worcs: A workflow for open reproducible code in science. Data Science, 4, 29–49. https://doi.org/10.3233/DS-210031
van Lissa, C. J., Peikert, A., Brandmaier, A. M., Schonbrodt, F. D., & van Dongen, N. N. N. (2025). To be fair: Theory specification needs an update. https://osf.io/preprints/psyarxiv/t53np
van Lissa, C. (2022). Developmental data science: How machine learning can advance theory formation in developmental psychology. Infant and Child Development, 32, e2370. https://doi.org/10.1002/icd.2370
van den Broek-Honingh, N., & de Jonge, J. (2018). Trust in science in the Netherlands: Survey monitor 2018. https://www.rathenau.nl/en/knowledge-ecosystems/trust-science-netherlands
Wagenmakers, E.-J., Dutilh, G., & Sarafoglou, G. (2018). The creativity-verification cycle in psychological science: New methods to combat old idols. Perspectives on Psychological Science, 13, 418–427. https://doi.org/10.1177/1745691618771357
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. https://doi.org/10.1177/1745691612463078
Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019). Recommendations for increasing the transparency of analysis of preexisting data sets. Advances in Methods and Practices in Psychological Science, 2, 214–227. https://doi.org/10.1177/2515245919848684
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832
Wigboldus, D. H. J., & Dotsch, R. (2016). Encourage playing with data and discourage questionable reporting practices. Psychometrika, 81, 27–32. https://doi.org/10.1007/s11336-015-9445-1
Wilkinson, M. D., B., M., et al. (2016). The fair guiding principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18
Xie, Y., Wang, K., & Kong, Y. (2021). Prevalence of research misconduct and quesitionable research practices: A systematic review and meta-analysis. Science and Engineering Ethics, 27, 41. https://link.springer.com/article/10.1007/s11948-021-00314-9#citeas
Zondergeld, J., Scholten, R. H. H., Vreede, B. M. I., Hessels, R. S., Pijl, H. A. G., Buizer-Voskamp, J. E., Rasch, M., Lange, O. A., & Veldkamp, C. L. C. (2020). FAIR, safe and high-quality data: the data infrastructure and accessibility of the YOUth cohort study. Developmental Cognitive Neuroscience, 45, 100834. https://doi.org/10.1016/j.dcn.2020.100834
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Herbert Hoijtink, Jonathan de Bruin, Sascha Duken, Jacques Flores, Willem E. Frankenhuis, Caspar J. van Lissa
This work is licensed under a Creative Commons Attribution 4.0 International License.
