The is-ought problem in deciding what to replicate: Which motives guide current replication practices?
DOI:
https://doi.org/10.15626/MP.2024.4327Keywords:
replication, replication value, replication motive, conceptual replication, self-replication, publication biasAbstract
This commentary examines whether the motives that currently guide replication practices align with normative proposals for selecting replication targets. Drawing on the preliminary analysis of 1,075 replication studies in psychology, we find that researchers most often conduct replications to self-replicate, extend previous findings, or test their generalizability, whereas motives related to a study’s influence or uncertainty are mentioned less frequently. These findings suggest a discrepancy between what researchers do replicate and what meta-scientific frameworks recommend should be replicated, highlighting the continued influence of novelty-oriented incentives even within the replication landscape.
References
Anvari, F., & Lakens, D. (2018). The replicability crisis and public trust in psychological science. Comprehensive Results in Social Psychology, 3(3), 266–286. https://doi.org/10.1080/23743603.2019.1684822
Berinsky, A. J., Druckman, J. N., & Yamamoto, T. (2020). Publication biases in replication studies. Political Analysis, 29(3), 370–384. https://doi.org/10.1017/pan.2020.34
Field, S. M., Hoekstra, R., Bringmann, L., & van Ravenzwaaij, D. (2019). When and why to replicate: As easy as 1, 2, 3? Collabra: Psychology, 5(1). https://doi.org/10.1525/collabra.218
Hüffmeier, J., Mazei, J., & Schultze, T. (2016). Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology, 66, 81–92. https://doi.org/10.1016/j.jesp.2015.09.009
Isager, P. M. (2018). What to replicate? justifications of study choice from 85 replication studies (tech. rep.). Zenodo. https://doi.org/10.5281/zenodo.1470153
Isager, P. M., van Aert, R., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., & Lakens, D. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438
Isager, P. M., van ’t Veer, A. E., & Lakens, D. (2025). Replication value as a function of citation impact and sample size. Meta-Psychology. https://doi.org/10.15626/MP.2022.3300
Lynch, J. G., Bradlow, E. T., Huber, J. C., & Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333–342. https://doi.org/10.1016/j.ijresmar.2015.09.006
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688
Murphy, J., Mesquida, C., Caldwell, A. R., Earp, B. D., & Warne, J. P. (2022). Proposal of a selection protocol for replication of studies in sports and exercise science. Sports Medicine, 53(1), 281–291. https://doi.org/10.1007/s40279-022-01749-1
Pittelkow, M.-M., Field, S. M., Isager, P. M., van ’t Veer, A. E., Anderson, T., Cole, S. N., Dominik, T., Giner-Sorolla, R., Gok, S., Heyman, T., Jekel,
M., Luke, T. J., Mitchell, D. B., Peels, R., Pendrous, R., Sarrazin, S., Schauer, J. M., Specker, E., Tran, U. S., & van Ravenzwaaij, D. (2023). The process of replication target selection in psychology: What to consider? Royal Society Open Science, 10(2). https://doi.org/10.1098//rsos.210586
Rotello, C. M., Heit, E., & Dubé, C. (2015). When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin Review, 22, 944–954. https://doi.org/10.3758/s13423-014-0759-2
Wingen, T., Berkessel, J. B., & Englich, B. (2020). No replication, no trust? how low replicability influences trust in psychology. Social Psychological and Personality Science, 11(4), 454–463. https://doi.org/10.1177/1948550619877412
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Kevin Kamermans, Leonie Dudda, Tatsuya Daikoku, Steven Verheyen
This work is licensed under a Creative Commons Attribution 4.0 International License.
