Meta-Psychology

Replication Value Increases With Transparency, Test Severity, and Societal Impact

Rene Bekkers — 2025-10-29

This comment argues that replications should not be prioritized as a function of citation count and sample size, but instead as a function of societal impact, test severity, and the information value of a replication.

Replication value as a function of citation impact and sample size: response to commentaries

Peder Isager — 2025-10-29

The primary goal of our target article (Isager et al., 2025) is to give the research community an example of what a well-justified replication value metric could look like, and to encourage discussion of how replication value could be quantified in practice. Furthermore, in the target article we discuss practical hurdles to quantification and possible practical applications for RVCn and other metrics. As that article proposes a method for how to do research–in this case a method to select which claims in the literature need replication most–it is important to receive criticism, feedback, and viewpoints from a diverse range of authors interested in this topic. We are delighted to read the many thoughtful yet critical commentaries, several of which proposing adjustments or alternatives to the equations we have proposed in the target article. This is very encouraging to see, as our aim with initiating this call in Meta-Psychology was to create an open dialogue in the scientific record. RVCn is an efficient but limited metric. Its limitations should be laid bare, and we fully expect that improved metrics and selection procedures can be created in the future. We hope our target article and these commentaries together will inspire readers to continue the discussion of how to efficiently and transparently select studies for replication. In this rejoinder we will summarize what we see as the major themes touched on in the commentaries, and we will reply to some of the specific proposals and criticisms brought up by different commentary authors.

The value of replications goes beyond replicability and is associated with the value of the research it replicates: Commentary on Isager et al. (2025)

Gilad Feldman — 2025-10-29

Replications are essential for rigorous credible science yet are still grossly undervalued and very rare. The value of replications is directly tied to the value of the research they aim to replicate, and replications offer many benefits that go far beyond the mere testing of replicability, such as including verifications and error detection, promoting long-term reproducibility of all research outputs, clarifying theory, refining measurement, and testing generalizability. We need far more independent pre-registered well-powered direct replications to strengthen the credibility of scientific findings. Isager et al. (2025)’s aim to define a formula for the value of replications based on over-simplified metrics of citation count and sample size is misaligned, already misunderstood, and may backfire by hindering the pursuit and publication of replications.

Replication Value in the Courtroom

Edith Beerdsen — 2025-09-06

This commentary is a response to Isager, P. M., van ’t Veer, A. E. & Lakens, D. (2025): Replication value as a function of citation impact and sample size. It argues that, in assessing "the value of of being correct about the truth status of a claim," it is important to try to capture nonscientific impact. This commentary focuses in particular on the impact that original research can have in a legal context.

Replication Value Usage and its Performance for Large Sample Sizes - Commentary on Isager et al. (2025)

Linda Bomm — 2025-10-29

The Replication Value (RV_Cn) metric was introduced to help researchers prioritize studies for replication based on expected utility. While we welcome the introduction of this straightforward and systematic replication decision approach, we identify two limitations of the RV_Cn. First, when testing the “repeatability” of a study or systematically incorporating replication into a research workflow, the RV_Cn may not always be the most suitable metric to guide decisions. Use cases should consider the scope conditions of the metric. Second, the RV_Cn shows limited sensitivity in distinguishing between studies with large sample sizes. To address this, we propose a simple adjustment: a log transformation of the sample size component. This modification improves the metric’s discriminatory power for high-N studies and better aligns the (RV_Cn) with its intended purpose: guiding efficient and meaningful replication efforts.

Thinking Beyond RVCN: Addressing the Complexity of Replication Target Selection

Merle-Marie Pittelkow — 2025-10-29

Isager, van ’t Veer and Lakens (2025) proposed a quantitative operationalization of replication value (denoted RV_CN), using average yearly citation count and sample size as proxies for value and uncertainty, respectively. In this commentary, we suggest that the approach of Isager et al., while a good theoretical departure point, oversimplifies the complex decision-making process that underpins replication target selection in practice. We present what we view as some issues with RV_CN, notably the use of citation count and ambiguity as to whether RV_CNis prescriptive or descriptive. We also present preliminary empirical evidence that RV_CN diverges on its performance as a replication target selection method, compared with existing selection methods (such as those published by us in the past). We conclude with the recommendation that going forward, approaches should emphasize the multifaceted nature of replication target selection to maximize their practical utility.

The Replication Dilemma: Potential Challenges in Measuring Replication Value—A Commentary on Isager et al. (2025)

Adrien Fillon — 2025-10-29

The authors (Isager et al., 2025) start with the main assumption that researchers’ efforts toward replications are constrained by resources, and they propose a simple, practically scalable framework of research replication value that guides the researchers and the scientific community at large intending to achieve bigger bang for the buck. Specifically, the authors propose a framework that combines citation impact and sample size of the original articles as a metric for assessing replication value. This implies that original studies with higher scores on this metric can be prioritized for replication efforts. We thoroughly agree with the authors’ assumption and indeed support the view of working towards an optimal framework that helps the community achieve maximum research impact from the replication efforts. In this commentary, we propose to discuss three important limitations that have to be considered before using such metrics. We thoroughly agree with the authors' assumption, and indeed support the view of working towards an optimal framework that helps the community achieve maximum research impact from the replication efforts. In this commentary, we propose to discuss three important limitations that have to be considered before using such metrics.

Use and Misuse of a Fast Approximation: Not a Criticism, but a Caution

Carlisle Rainey — 2025-10-29

P. Isager et al. (2025) propose a fast approximation of replication value, RVCn, that relies on citation count and sample size. This approximation is simple, transparent, and easy to implement across many studies. It can potentially help metascientists evaluate large collections of studies. However, RVCn is not a precise statement of fact; it should not substitute for detailed substantive and methodological arguments. I make two counterclaims: (1) studies with few citations might be worth replicating and (2) studies with large samples might be worth replicating. While RVCn can helpfully supplement researchers’ judgments, it should not substitute for researchers’ judgments.

Valuing replication value

Kaito Takashima — 2025-10-29

This is a commentary piece on the proposal by Isager et al. (2025) for a new metric, RVCn, designed to evaluate the replication value of psychological studies. We discuss the hope of utilizing the RVCn metric in undergraduate education and possible improvements on using some elements other than original sample size to derive this metric.

Replication value as a function of citation impact and sample size

Peder Isager — 2025-10-29

Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication effort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2023). In this article, we propose a quantitative operationalization of replication value. We first discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RV_Cn. Third, we suggest how RV_Cn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RV_Cn estimates. The goal of this article is to explain how RV_Cn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.

Standalone Apps for Anxiety and Depression Show Promising Early Efficacy: A Commentary on Meta-Analytic Results

Nicholas Jacobson — 2025-10-29

The current commentary is focused on the methods and conclusions drawn in a recent meta-analysis which evaluated the impact of standalone interventions in treating anxiety and depressive symptoms (Weisel et al., 2019). The current commentary discusses the large impact of methodological choices made to exclude transdiagnostic treatments and group heterogeneous treatments on study conclusions. Additionally, the current commentary evaluates these conclusions considering opposing from two additional meta-analytic findings. The current review concludes that more research is needed effects before drawing any definitive conclusions, but the current evidence base suggests that apps show promising early efficacy in treating anxiety and depressive symptoms.

Reimagining Consumer Psychology through Open Science Principles

Salim Moussa — 2025-10-29

Consumer psychology is facing various challenges, including a lack of research integrity and unethical publishing practices. This commentary lists pivotal events and discusses related findings that point to the field's need for reform. Open Science principles are proposed as a transformative solution to promote transparency in data, methodology, access, and peer review. Consumer psychology can only be revitalized and regain credibility if it fully embraces these four pillars. Academic and professional associations with an impact on consumer psychology must set a good example by cultivating a culture of integrity and accountability in research and publishing. Consumer psychologists must educate future generations of researchers on research methodology and research ethics.

Data is not available upon request

Ian Hussey — 2025-09-25

Many journals now require data sharing and require articles to include a Data Availability Statement. However, several studies over the past two decades have shown that promissory notes about data sharing are rarely abided by and that data is generally not available upon request. This has negative consequences for many essential aspects of scientific knowledge production, including independent verification of results, efficient secondary use of data, and knowledge synthesis. I assessed the prevalence of data sharing upon request in articles employing the Implicit Relational Assessment Procedure published within the last 5 years. Of 52 articles, 42% contained a Data Availability Statement, most of which stated that data was available upon request. This rose from 0% in 2018 to 100% in 2022, indicating a change in journals’ policies. However, only 27% of articles’ authors actually shared data. Among articles stating that data was available upon request, only 17% shared data upon request. The presence of Data Availability Statements was not associated with higher rates of data sharing (p = .55), indicating a lack of adherence with journals’ policies. Results replicate those found elsewhere: data is generally not available upon request, and promissory Data Availability Statements are typically not adhered to. Issues, causes, and implications are considered.

Practicing Theory Building in a Many Modelers Hackathon

Noah van Dongen — 2025-04-15

Scientific theories reflect some of humanity's greatest epistemic achievements. The best theories motivate us to search for discoveries, guide us towards successful interventions, and help us to explain and organize knowledge. Such theories require a high degree of specificity, which in turn requires formal modeling. Yet, in psychological science, many theories are not precise and psychological scientists often lack the technical skills to formally specify existing theories. This problem raises the question: How can we promote formal theory development in psychology, where there are many content experts but few modelers? In this paper, we discuss one strategy for addressing this issue: a Many Modelers approach. Many Modelers consists of mixed teams of modelers and non-modelers that collaborate to create a formal theory of a phenomenon. Here, we report a proof of concept of this approach, which we piloted as a three-hour hackathon at the Society for the Improvement of Psychological Science conference in 2021. After surveying the participants, results suggest that (a) psychologists who have never developed a formal model can become (more) excited about formal modeling + and theorizing; (b) a division of labor in formal theorizing is possible where only one or a few team members possess the prerequisite modeling expertise; and (c) first working prototypes of a theoretical model can be created in a short period of time. These results show some promise for the many modelers approach as a team science tool for theory development.

Overview on the Null Hypothesis Significance Test

Noah Dongen — 2025-05-07

For decades, waxing and waning, there has been an ongoing debate on the values and problems of the ubiquitously used null hypothesis significance test (NHST). With the start of the replication crisis, this debate has flared-up once again, especially in the psychology and psychological methods literature. Arguing for or against the NHST method usually takes place in essays and opinion pieces that cover some, but not all the qualities and problems of the method. The NHST literature landscape is vast, a clear overview is lacking, and participants in the debate seem to be talking past one another. To contribute to a resolution, we conducted a systematic review on essay literature concerning NHST published in psychology and psychological methods journals between 2011 and 2018. We extracted all arguments in defense of (20) and against (70) NHST, and we extracted the solutions (33) that were proposed to remedy (some of) the perceived problems of NHST. Unfiltered, these 123 items form a landscape that is prohibitively difficult to keep in one’s sights. Our contribution to the resolution of the NHST debate is twofold. 1) We performed a thematic synthesis of the arguments and solutions, which carves the landscape in a framework of three zones: mild, moderate, and critical. This reduction summarizes groups of arguments and solutions, thus offering a manageable overview of NHST’s qualities, problems, and solutions. 2) We provide the data on the arguments and solutions as a resource for those who will carry-on the debate and/or study the use of NHST.

The Evolution of Data Sharing Practices in the Psychological Literature

Judith Neve — 2025-04-09

Sharing data has many benefits. However, data sharing rates remain low, for the most part well below 50%. A variety of interventions encouraging data sharing have been proposed. We focus here on editorial policies. Kidwell et al. (2016) assessed the impact of the introduction of badges in Psychological Science; Hardwicke, Mathur, et al. (2018) assessed the impact of Cognition’s mandatory data sharing policy. Both studies found policies to improve data sharing practices, but only assessed the impact of the policy for up to 25 months after its implementation. We examined the effect of these policies over a longer term by reusing their data and collecting a follow-up sample including articles published up until December 31st, 2019. We fit generalized additive models as these allow for a flexible assessment of the effect of time, in particular to identify nonlinear changes in the trend. These models were compared to generalized linear models to examine whether the non-linearity is needed. Descriptive results and the outputs from generalized additive and linear models were coherent with previous findings: following the policies in Cognition and Psychological Science, data sharing statement rates increased immediately and continued to increase beyond the timeframes examined previously, until reaching close to 100%. In Clinical Psychological Science, data sharing statement rates started to increase only two years following the implementation of badges. Reusability rates jumped from close to 0% to around 50% but did not show changes within the pre-policy nor the post-policy timeframes. Journals that did not implement a policy showed no change in data sharing rates or reusability over time. There was variability across journals in the levels of increase, so we suggest future research should examine a larger number of policies to draw conclusions about their efficacy. We also encourage future research to investigate the barriers to data sharing specific to psychology subfields to identify the best interventions to tackle them.

Folk Conceptions of Free Will: A Systematic Review and Narrative Synthesis of Psychological Research

Alison Lam — 2025-10-29

The existence of free will has been a subject of fierce academic debate for millennia, still the meaning of the term “free will” remains nebulous. In the past two decades, psychologists have made considerable progress in defining lay concepts of free will. We present the first systematic review of primary psychological evidence on how ordinary folk conceptualise free will, encompassing folk concepts, beliefs, intuitions, and attitudes about free will. A total of 1,384 records were identified following a pre-registered protocol. After abstract and full-text screening, 18 articles were eligible for inclusion, comprised of 36 studies and 10,176 participants from regions including the United States, Singapore, Hong Kong, India, Turkey, and Germany. A narrative synthesis of results showed that for ordinary folk, especially the more educated population from the United States, free will is a dynamic construct centred on the ability to choose following one’s goals and desires, whilst being uncoerced and reasonably free from constraints. Results suggesting metaphysical considerations regarding consciousness, dualism, and determinism were inconclusive. Our findings provided preliminary support for a psychological model of folk conception of free will, and elucidated potential pathways mediating the effects of consciousness and dualism on free will attributions. Further research is needed to explicate the distinction between having free will and having the ability to exercise free will, as well as the cross-cultural validity of findings on folk conceptions of free will.

A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles

Nataly Beribisky — 2025-10-29

Many bodies recommend that a sample planning procedure, such as traditional NHST a priori power analysis, is conducted during the planning stages of a study. Power analysis allows the researcher to estimate how many participants are required in order to detect a minimally meaningful effect size at a specific level of power and Type I error rate. However, there are several drawbacks to the procedure that render it “a mess.” Specifically, the identification of the minimally meaningful effect size is very challenging, the procedure is not precision oriented, and does not guide the researcher to collect as many participants as feasibly possible. In this study, we explore how these three theoretical issues are reflected in applied psychological research in order to better understand whether these issues are concerns in practice. To investigate how power analysis is currently used, this study reviewed the reporting of 443 power analyses in high impact Psychology journals in 2016 and 2017 using Google Scholar. It was found that researchers rarely use the minimally meaningful effect size as a rationale for the chosen effect in a power analysis. Further, precision-based approaches and collecting the maximum sample size feasible are almost never used in tandem with power analyses. In light of these findings, we offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as collecting the maximum sample size feasible.