Use and Misuse of a Fast Approximation: Not a Criticism, but a Caution
DOI:
https://doi.org/10.15626/MP.2024.4216Abstract
P. Isager et al. (2025) propose a fast approximation of replication value, RVCn, that relies on citation count and sample size. This approximation is simple, transparent, and easy to implement across many studies. It can potentially help metascientists evaluate large collections of studies. However, RVCn is not a precise statement of fact; it should not substitute for detailed substantive and methodological arguments. I make two counterclaims: (1) studies with few citations might be worth replicating and (2) studies with large samples might be worth replicating. While RVCn can helpfully supplement researchers’ judgments, it should not substitute for researchers’ judgments.
Metrics
References
Bakker, B. N., Bomm, L., & Peterson, D. J. (2025). Commentary on isager et al. (2025): Reflections on the replication value (rv) and a proposal for revision. Meta-Psychology, 9. https://doi.org/10.15626/MP.2024.4324
Bekkers, R. (2025). Replication value increases with transparency, test severity, and societal impact. Meta-Psychology, 9. https://doi.org/10.15626/MP.2024.4194
Blair, G., Coppock, A., & Moor, M. (2020). When to worry about sensitivity bias: A social reference theory and evidence from 30 years of list experiments. American Political Science Review, 114(4), 1297–1315. https://doi.org/10.1017/S0003055420000374
Box, G. E. P., & Draper, N. R. (2007). Response surfaces, mixtures, and ridge analyses (2nd). Wiley. https://doi.org/10.1002/0470072768
Chambers, C. (2019). The seven deadly sins of psychology (2nd). Princeton University Press.
Clifford, S., Sheagley, G., & Piston, S. (2021). Increasing precision without altering treatment effects: Repeated measures designs in survey experiments. American Political Science Review, 115(3), 1048–1065. https://doi.org/10.1017/S0003055421000241
Dion, M. L., Sumner, J. L., & Mitchell, S. M. (2018). Gendered citation patterns across political science and social science methodology fields. Political Analysis, 26(3), 312–327. https://doi.org/10.1017/pan.2018.12
Feldman, G. (2025). The value of replications goes beyond replicability and is associated with the value of the research it replicates: Commentary on Isager et al. (2025). Meta-Psychology, 9. https://doi.org/10.15626/MP.2024.4326
Isager, P., van ’t Veer, A., & Lakens, D. (2025). Replication value as a function of citation impact and sample size. Meta-Psychology, 9. https://doi.org/10.15626/MP.2022.3300
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., Krueger, J. I., et al. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438
Pittelkow, M.-M., Field, S. M., Isager, P. M., Van’t Veer, A. E., Anderson, T., Cole, S. N., Dominik, T., Giner-Sorolla, R., Gok, S., Heyman, T., Jekel, M., Luke, T. J., Mitchell, D. B., Peels, R., Pendrous, R., Sarrazin, S., Schauer, J. M., Specker, E., Tran, U. S., . . . van Ravenzwaaij, D. (2023).
The process of replication target selection in psychology: What to consider? Royal Society Open Science, 10(2). https://doi.org/10.1098/rsos.210586
Rainey, C. (2025). Power rules: Practical advice for computing power (and automating with pilot data) [Preprint]. https://doi.org/10.31219/osf.io/5am9q
Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41. https://doi.org/10.1017/S0140525X17001972
Published
Issue
Section
License
Copyright (c) 2025 Carlisle Rainey
This work is licensed under a Creative Commons Attribution 4.0 International License.
