Meta-Psychology
type: channel
description: Meta-Psychology publishes theoretical and empirical contributions that advance psychology as a science through critical discourse related to individual articles, research lines, research areas, or psychological science as a field.
publisher: Linnaeus University Press
language: en
publication name: Meta-Psychology
issn: 2003-2714
items: Replication Value Increases With Transparency, Test Severity, and Societal Impact
items: Replication value as a function of citation impact and sample size: response to commentaries
items: Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology
items: Responsible Research Assessment (Parts I and II): Responses to the Commentaries
items: Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology
items: The value of replications goes beyond replicability and is associated with the value of the research it replicates: Commentary on Isager et al. (2025)
items: Replication Value in the Courtroom
items: Replication Value Usage and its Performance for Large Sample Sizes - Commentary on Isager et al. (2025)
items: Thinking Beyond RVCN: Addressing the Complexity of Replication Target Selection
items: The Replication Dilemma: Potential Challenges in Measuring Replication Value—A Commentary on Isager et al. (2025)
items: Psychology’s Theory Crisis, and why formal modelling cannot solve it
items: Use and Misuse of a Fast Approximation: Not a Criticism, but a Caution
items: Valuing replication value
items: Standalone Apps for Anxiety and Depression Show Promising Early Efficacy: A Commentary on Meta-Analytic Results
items: Reimagining Consumer Psychology through Open Science Principles
items: Pre-replication: Anything Goes, Once
items: Falsifying the Insufficient Adjustment Model: No Evidence for Unidirectional Adjustment From Anchors
items: Data is not available upon request
items: Practicing Theory Building in a Many Modelers Hackathon
items: Replication value as a function of citation impact and sample size
items: Overview on the Null Hypothesis Significance Test
items: Comments on Cassese and Holman 2019 “Playing the Woman Card: Ambivalent Sexism in the 2016 U.S. Presidential Race”
items: The Evolution of Data Sharing Practices in the Psychological Literature
items: Folk Conceptions of Free Will: A Systematic Review and Narrative Synthesis of Psychological Research
items: A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles

Replication Value Increases With Transparency, Test Severity, and Societal Impact
type: item
description: This comment argues that replications should not be prioritized as a function of citation count and sample size, but instead as a function of societal impact, test severity, and the information value of a replication.
subject: Special Topic
  topic: section
subject: citation count
  topic: keywords
subject: replication
  topic: keywords
subject: replication value
  topic: keywords
subject: RVcn
  topic: keywords
subject: sample size
  topic: keywords
subject: study selection
  topic: keywords
creator: Rene Bekkers
rights: Copyright (c) 2025 Rene Bekkers https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4194

Replication value as a function of citation impact and sample size: response to commentaries
type: item
description: The primary goal of our target article (Isager et al., 2025) is to give the research community an example of what a well-justified replication value metric could look like, and to encourage discussion of how replication value could be quantified in practice. Furthermore, in the target article we discuss practical hurdles to quantification and possible practical applications for RVCn and other metrics. As that article proposes a method for how to do research–in this case a method to select which claims in the literature need replication most–it is important to receive criticism, feedback, and viewpoints from a diverse range of authors interested in this topic. We are delighted to read the many thoughtful yet critical commentaries, several of which proposing adjustments or alternatives to the equations we have proposed in the target article. This is very encouraging to see, as our aim with initiating this call in Meta-Psychology was to create an open dialogue in the scientific record. RVCn is an efficient but limited metric. Its limitations should be laid bare, and we fully expect that improved metrics and selection procedures can be created in the future. We hope our target article and these commentaries together will inspire readers to continue the discussion of how to efficiently and transparently select studies for replication. In this rejoinder we will summarize what we see as the major themes touched on in the commentaries, and we will reply to some of the specific proposals and criticisms brought up by different commentary authors.
subject: Special Topic
  topic: section
subject: RVcn
  topic: keywords
subject: replication value
  topic: keywords
subject: replication
  topic: keywords
creator: Peder Isager
creator: Anna E. van 't Veer
creator: Daniël Lakens
rights: Copyright (c) 2025 Peder Isager, Anna E. van 't Veer, Daniël Lakens https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2025.4738

Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology
type: item
description: Traditional metric indicators of scientific productivity (e.g., journal impact factor; h-index) have been heavily criticized for being invalid and fueling a culture that focuses on the quantity, rather than the quality, of a person's scientific output. There is now a wide-spread demand for viable alternatives to current academic evaluation practices. In a previous report, we laid out four basic principles of a more responsible research assessment in academic hiring and promotion processes (Schönbrodt et al., 2025). The present paper offers a specific proposal for how these principles may be implemented in practice: We argue in favor of broadening the range of relevant research contributions and propose a set of concrete quality criteria (including a ready-to-use online tool) for research articles. These criteria are supposed to be used primarily in the first phase of the assessment process. Their function is to help establish a minimum threshold of methodological (i.e., theoretical and empirical) rigor that candidates need to pass in order to be further considered for hiring or promotion. In contrast, the second phase of the assessment process focuses more on the actual content of candidates' research and necessarily uses more narrative means of assessment. The debate over ways of replacing current invalid evaluation criteria with ones that relate more closely to scientific quality continues. Its course and outcome will depend on the willingness of researchers to get involved and help shape it.
subject: Commentaries
  topic: section
subject: DORA
  topic: keywords
subject: CoARA
  topic: keywords
subject: research assessment
  topic: keywords
subject: research quality
  topic: keywords
subject: impact
  topic: keywords
creator: Anne Gärtner
creator: Daniel Leising
creator: Nele Freyer
creator: Philipp Musfeld
creator: Jens Lange
creator: Felix Schönbrodt
rights: Copyright (c) 2025 Anne Gärtner, Daniel Leising, Nele Freyer, Philipp Musfeld, Jens Lange, Felix D. Schönbrodt https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2024.4604

Responsible Research Assessment (Parts I and II): Responses to the Commentaries
type: item
description: We give a brief overview of our deliberations in responding to the commentaries on our two target papers presenting the RESQUE (Research Quality Evaluation) framework. While we were able to incorporate many suggestions for improvement directly, we acknowledge that other areas (e.g., quality in theorizing) will require further elaboration. In this paper, we specifically touch on the following issues: (a) eligible types of publication, (b) measurability, (c) quality criteria for software and datasets, (d) theoretical rigor, (e) quantity, (f) authorship, (g) potential bias (against certain methodologies, types of research contributions, or subdisciplines), (h) overall rigor score, (i) weighting of individual indicators, (j) types of data, (k) impact, (l) interdisciplinary value, (m) teaching, (n) expertise, (o) gaming the new metrics, and (p) representativeness. The RESQUE framework has met with largely positive reception so far, but continues to evolve and will thrive best when community involvement stays strong.
subject: Commentaries
  topic: section
subject: Metascience
  topic: keywords
subject: Incentive
  topic: keywords
subject: Evaluation
  topic: keywords
creator: Daniel Leising
creator: Anne Gärtner
creator: Felix Schönbrodt
rights: Copyright (c) 2025 Daniel Leising, Anne Gärtner, Felix Schönbrodt https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2024.4603

Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology
type: item
description: The use of journal impact factors and other metric indicators of research productivity, such as the h-index, has been heavily criticized for being invalid for the assessment of individual researchers and for fueling a detrimental "publish or perish" culture. Multiple initiatives call for developing alternatives to existing metrics that better reflect quality (instead of quantity) in research assessment. This report, written by a task force established by the German Psychological Society, proposes how responsible research assessment could be done in the field of psychology. We present four principles of responsible research assessment in hiring and promotion and suggest a two-phase assessment procedure that combines the objectivity and efficiency of indicators with a qualitative, discursive assessment of shortlisted candidates. The main aspects of our proposal are (a) to broaden the range of relevant research contributions to include published data sets and research software, along with research papers, and (b) to place greater emphasis on quality and methodological rigor in research evaluation.
subject: Commentaries
  topic: section
subject: DORA
  topic: keywords
subject: CoARA
  topic: keywords
subject: research assessment
  topic: keywords
subject: research quality
  topic: keywords
subject: impact
  topic: keywords
creator: Felix D. Schönbrodt
creator: Anne Gärtner
creator: Maximilian Frank
creator: Mario Gollwitzer
creator: Malika Ihle
creator: Dorothee Mischkowski
creator: Le Vy Phan
creator: Manfred Schmitt
creator: Anne M. Scheel
creator: Anna-Lena Schubert
creator: Ulf Steinberg
creator: Daniel Leising
rights: Copyright (c) 2025 Felix D. Schönbrodt, Anne Gärtner, Maximilian Frank, Mario Gollwitzer, Malika Ihle, Dorothee Mischkowski, Le Vy Phan, Manfred Schmitt, Anne M. Scheel, Anna-Lena Schubert, Ulf Steinberg, Daniel Leising https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2024.4601

The value of replications goes beyond replicability and is associated with the value of the research it replicates: Commentary on Isager et al. (2025)
type: item
description: Replications are essential for rigorous credible science yet are still grossly undervalued and very rare. The value of replications is directly tied to the value of the research they aim to replicate, and replications offer many benefits that go far beyond the mere testing of replicability, such as including verifications and error detection, promoting long-term reproducibility of all research outputs, clarifying theory, refining measurement, and testing generalizability. We need far more independent pre-registered well-powered direct replications to strengthen the credibility of scientific findings. Isager et al. (2025)’s aim to define a formula for the value of replications based on over-simplified metrics of citation count and sample size is misaligned, already misunderstood, and may backfire by hindering the pursuit and publication of replications.
subject: Special Topic
  topic: section
subject: replications
  topic: keywords
subject: value
  topic: keywords
subject: commentary
  topic: keywords
subject: replicability
  topic: keywords
creator: Gilad Feldman
rights: Copyright (c) 2025 Gilad Feldman https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4326

Replication Value in the Courtroom
type: item
description: This commentary is a response to Isager, P. M., van ’t Veer, A. E. & Lakens, D. (2025): Replication value as a function of citation impact and sample size. It argues that, in assessing "the value of of being correct about the truth status of a claim," it is important to try to capture nonscientific impact. This commentary focuses in particular on the impact that original research can have in a legal context.
subject: Special Topic
  topic: section
subject: replication
  topic: keywords
subject: impact
  topic: keywords
subject: law
  topic: keywords
creator: Edith Beerdsen
rights: Copyright (c) 2025 Edith Beerdsen https://creativecommons.org/licenses/by/4.0/
date: 2025-09-06
publication date: 2025-09-06
doi: 10.15626/MP.2024.4325

Replication Value Usage and its Performance for Large Sample Sizes - Commentary on Isager et al. (2025)
type: item
description: The Replication Value (RVCn) metric was introduced to help researchers prioritize studies for replication based on expected utility. While we welcome the introduction of this straightforward and systematic replication decision approach, we identify two limitations of the RVCn. First, when testing the “repeatability” of a study or systematically incorporating replication into a research workflow, the RVCn may not always be the most suitable metric to guide decisions. Use cases should consider the scope conditions of the metric. Second, the RVCn shows limited sensitivity in distinguishing between studies with large sample sizes. To address this, we propose a simple adjustment: a log transformation of the sample size component. This modification improves the metric’s discriminatory power for high-N studies and better aligns the (RVCn) with its intended purpose: guiding efficient and meaningful replication efforts.
subject: Special Topic
  topic: section
subject: Replication Value
  topic: keywords
subject: RVCn
  topic: keywords
subject: replication
  topic: keywords
subject: study selection
  topic: keywords
subject: study comparison
  topic: keywords
creator: Linda Bomm
creator: Delaney Peterson
creator: Bert Bakker
rights: Copyright (c) 2025 Linda Bomm, Delaney J. Peterson, Bert N. Bakker https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4324

Thinking Beyond RVCN: Addressing the Complexity of Replication Target Selection
type: item
description: Isager, van ’t Veer and Lakens (2025) proposed a quantitative operationalization of replication value (denoted RVCN), using average yearly citation count and sample size as proxies for value and uncertainty, respectively. In this commentary, we suggest that the approach of Isager et al., while a good theoretical departure point, oversimplifies the complex decision-making process that underpins replication target selection in practice. We present what we view as some issues with RVCN, notably the use of citation count and ambiguity as to whether RVCN is prescriptive or descriptive. We also present preliminary empirical evidence that RVCN diverges on its performance as a replication target selection method, compared with existing selection methods (such as those published by us in the past). We conclude with the recommendation that going forward, approaches should emphasize the multifaceted nature of replication target selection to maximize their practical utility.
subject: Special Topic
  topic: section
subject: replication target selection
  topic: keywords
subject: qualitative frameworks
  topic: keywords
creator: Merle-Marie Pittelkow
creator: Sarahanne M. Field
creator: Don van Ravenzwaaij
rights: Copyright (c) 2025 Merle-Marie Pittelkow, Sarahanne M. Field, Don van Ravenzwaaij https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4323

The Replication Dilemma: Potential Challenges in Measuring Replication Value—A Commentary on Isager et al. (2025)
type: item
description: The authors (Isager et al., 2025) start with the main assumption that researchers’ efforts toward replications are constrained by resources, and they propose a simple, practically scalable framework of research replication value that guides the researchers and the scientific community at large intending to achieve bigger bang for the buck. Specifically, the authors propose a framework that combines citation impact and sample size of the original articles as a metric for assessing replication value. This implies that original studies with higher scores on this metric can be prioritized for replication efforts. We thoroughly agree with the authors’ assumption and indeed support the view of working towards an optimal framework that helps the community achieve maximum research impact from the replication efforts. In this commentary, we propose to discuss three important limitations that have to be considered before using such metrics. We thoroughly agree with the authors' assumption, and indeed support the view of working towards an optimal framework that helps the community achieve maximum research impact from the replication efforts. In this commentary, we propose to discuss three important limitations that have to be considered before using such metrics.
subject: Special Topic
  topic: section
subject: replication
  topic: keywords
subject: value
  topic: keywords
subject: open science
  topic: keywords
subject: metascience
  topic: keywords
creator: Adrien Fillon
creator: Subramanya Prasad Chandrashekar
rights: Copyright (c) 2025 Adrien Fillon, Subramanya Prasad Chandrashekar https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4312

Psychology’s Theory Crisis, and why formal modelling cannot solve it
type: item
description: In light of psychology’s ‘theory crisis’, multiple authors have argued that adopting formalization and/or formal modelling would constitute a useful or even necessary step towards stronger psychological theory. In this article, I instead argue that formal modelling cannot solve the core problem the psychological ‘theory crisis’ refers to, which are the currently high degrees of contrastive and holistic underdetermination of our theories by our data. I do so by first introducing underdetermination as an explanatory framework for determining the evidential import of research findings for theories, and showing how both broader theoretical considerations and informal assumptions are key to this process. Then, I derive the aforementioned core problem from the current ‘theory crisis’ literature and tentatively explore its possible solutions. Lastly, I show that formal modelling is neither a necessary nor sufficient solution for either contrastive or holistic underdetermination, and that its uncritical adoption might instead worsen the crisis.
subject: Commentaries
  topic: section
subject: theory
  topic: keywords
subject: theory crisis
  topic: keywords
subject: formal modelling
  topic: keywords
subject: underdetermination
  topic: keywords
subject: evidence
  topic: keywords
creator: Freek Oude Maatman
rights: Copyright (c) 2025 Freek Oude Maatman https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2024.4224

Use and Misuse of a Fast Approximation: Not a Criticism, but a Caution
type: item
description: P. Isager et al. (2025) propose a fast approximation of replication value, RVCn, that relies on citation count and sample size. This approximation is simple, transparent, and easy to implement across many studies. It can potentially help metascientists evaluate large collections of studies. However, RVCn is not a precise statement of fact; it should not substitute for detailed substantive and methodological arguments. I make two counterclaims: (1) studies with few citations might be worth replicating and (2) studies with large samples might be worth replicating. While RVCn can helpfully supplement researchers’ judgments, it should not substitute for researchers’ judgments.
subject: Special Topic
  topic: section
creator: Carlisle Rainey
rights: Copyright (c) 2025 Carlisle Rainey https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4216

Valuing replication value
type: item
description: This is a commentary piece on the proposal by Isager et al. (2025) for a new metric, RVCn, designed to evaluate the replication value of psychological studies. We discuss the hope of utilizing the RVCn metric in undergraduate education and possible improvements on using some elements other than original sample size to derive this metric.
subject: Special Topic
  topic: section
creator: Kaito Takashima
creator: Yuki Yamada
rights: Copyright (c) 2025 Kaito Takashima, Yuki Yamada https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4210

Standalone Apps for Anxiety and Depression Show Promising Early Efficacy: A Commentary on Meta-Analytic Results
type: item
description: The current commentary is focused on the methods and conclusions drawn in a recent meta-analysis which evaluated the impact of standalone interventions in treating anxiety and depressive symptoms (Weisel et al., 2019). The current commentary discusses the large impact of methodological choices made to exclude transdiagnostic treatments and group heterogeneous treatments on study conclusions. Additionally, the current commentary evaluates these conclusions considering opposing from two additional meta-analytic findings. The current review concludes that more research is needed effects before drawing any definitive conclusions, but the current evidence base suggests that apps show promising early efficacy in treating anxiety and depressive symptoms.
subject: Commentaries
  topic: section
subject: meta-analysis
  topic: keywords
subject: apps
  topic: keywords
subject: anxiety
  topic: keywords
subject: depression
  topic: keywords
creator: Nicholas Jacobson
creator: William Chen
creator: Raphael Huang
rights: Copyright (c) 2025 Nicholas Jacobson, William Chen, Raphael Huang https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2020.2458

Reimagining Consumer Psychology through Open Science Principles
type: item
description: Consumer psychology is facing various challenges, including a lack of research integrity and unethical publishing practices. This commentary lists pivotal events and discusses related findings that point to the field's need for reform. Open Science principles are proposed as a transformative solution to promote transparency in data, methodology, access, and peer review. Consumer psychology can only be revitalized and regain credibility if it fully embraces these four pillars. Academic and professional associations with an impact on consumer psychology must set a good example by cultivating a culture of integrity and accountability in research and publishing. Consumer psychologists must educate future generations of researchers on research methodology and research ethics.
subject: Commentaries
  topic: section
subject: Consumer Psychology
  topic: keywords
subject: research integrity
  topic: keywords
subject: publication ethics
  topic: keywords
subject: Open Science principles
  topic: keywords
creator: Salim Moussa
rights: Copyright (c) 2025 Salim Moussa https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2024.4164

Pre-replication: Anything Goes, Once
type: item
description: In this commentary, we propose pre-replication as a necessary epistemic, analytic, and axiologic exercise to be carried out before each replication.
subject: Commentaries
  topic: section
subject: axiology
  topic: keywords
subject: epistemology
  topic: keywords
subject: methodology
  topic: keywords
creator: Veli-Matti Karhulahti
creator: Marcel Martončik
creator: Matúš Adamkovič
rights: Copyright (c) 2025 Veli-Matti Karhulahti, Marcel Martončik, Matúš Adamkovič https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2024.4155

Falsifying the Insufficient Adjustment Model: No Evidence for Unidirectional Adjustment From Anchors
type: item
description: After considering a more or less random number (i.e., an anchor), people’s subsequent estimates are biased toward that number. Such anchoring phenomena have been explained via an adjustment process that ends too early. We present a formalized version of the insufficient adjustment model, which captures the idea that decreasing the time that people have to adjust from anchors draws their estimates closer to the anchors. In four independent studies (N = 898), we could not confirm this effect of time on anchoring. Moreover, anchoring effects vanished in the two studies that deviated from classical paradigms by using a visual scale or a two-alternative forced-choice paradigm to allow faster responses. Although we propose that the current version of the insufficient adjustment model should be discarded, we believe that adjustment models hold the most potential for the future of anchoring research, and we make suggestions for what these might look like.
subject: File-Drawer Reports
  topic: section
subject: anchoring
  topic: keywords
subject: TOTE
  topic: keywords
subject: time pressure
  topic: keywords
subject: two-alternative forced-choice
  topic: keywords
subject: insufficient adjustment
  topic: keywords
creator: Lukas Röseler
creator: Lisa Incerti
creator: Tobias R. Rebholz
creator: Christian Seida
creator: Frank Papenmeier
rights: Copyright (c) 2025 Lukas Röseler, Lisa Incerti, Tobias R. Rebholz, Christian Seida, Frank Papenmeier https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2024.4137

Data is not available upon request
type: item
description: Many journals now require data sharing and require articles to include a Data Availability Statement. However, several studies over the past two decades have shown that promissory notes about data sharing are rarely abided by and that data is generally not available upon request. This has negative consequences for many essential aspects of scientific knowledge production, including independent verification of results, efficient secondary use of data, and knowledge synthesis. I assessed the prevalence of data sharing upon request in articles employing the Implicit Relational Assessment Procedure published within the last 5 years. Of 52 articles, 42% contained a Data Availability Statement, most of which stated that data was available upon request. This rose from 0% in 2018 to 100% in 2022, indicating a change in journals’ policies. However, only 27% of articles’ authors actually shared data. Among articles stating that data was available upon request, only 17% shared data upon request. The presence of Data Availability Statements was not associated with higher rates of data sharing (p = .55), indicating a lack of adherence with journals’ policies. Results replicate those found elsewhere: data is generally not available upon request, and promissory Data Availability Statements are typically not adhered to. Issues, causes, and implications are considered.
subject: Original articles
  topic: section
subject: data sharing
  topic: keywords
subject: data availability statement
  topic: keywords
subject: implicit relational assessment procedure
  topic: keywords
subject: implicit social cognition
  topic: keywords
creator: Ian Hussey
rights: Copyright (c) 2025 Ian Hussey https://creativecommons.org/licenses/by/4.0/
date: 2025-09-25
publication date: 2025-09-25
doi: 10.15626/MP.2023.4008

Practicing Theory Building in a Many Modelers Hackathon
type: item
description: Scientific theories reflect some of humanity's greatest epistemic achievements. The best theories motivate us to search for discoveries, guide us towards successful interventions, and help us to explain and organize knowledge. Such theories require a high degree of specificity, which in turn requires formal modeling. Yet, in psychological science, many theories are not precise and psychological scientists often lack the technical skills to formally specify existing theories. This problem raises the question: How can we promote formal theory development in psychology, where there are many content experts but few modelers? In this paper, we discuss one strategy for addressing this issue: a Many Modelers approach. Many Modelers consists of mixed teams of modelers and non-modelers that collaborate to create a formal theory of a phenomenon. Here, we report a proof of concept of this approach, which we piloted as a three-hour hackathon at the Society for the Improvement of Psychological Science conference in 2021. After surveying the participants, results suggest that (a) psychologists who have never developed a formal model can become (more) excited about formal modeling + and theorizing; (b) a division of labor in formal theorizing is possible where only one or a few team members possess the prerequisite modeling expertise; and (c) first working prototypes of a theoretical model can be created in a short period of time. These results show some promise for the many modelers approach as a team science tool for theory development.
subject: Original articles
  topic: section
subject: Modeling
  topic: keywords
subject: formal theory
  topic: keywords
subject: theory development
  topic: keywords
subject: hackathon
  topic: keywords
subject: team science
  topic: keywords
subject: education
  topic: keywords
creator: Noah van Dongen
creator: Adam Finnemann
creator: Jill de Ron
creator: Leonid Tiokhin
creator: Shirley Wang
creator: Johannes Algermissen
creator: Elena C. Altmann
creator: Štěpán Bahník
creator: Li-Ching Chuang
creator: Andrei Dumbravă
creator: Jens H. Fünderich
creator: Sandra J. Geiger
creator: Daria Gerasimova
creator: Aidai Golan
creator: Judith Herbers
creator: Marc Jekel
creator: Anton Kunnari
creator: Yih-Shiuan Lin
creator: David Moreau
creator: Yvonne Oberholzer
creator: Hannah K. Peetz
creator: Julia Rohrer
creator: Adrian Rothers
creator: Felix Schönbrodt
creator: Yashvin Seetahul
creator: Anna Szabelska
creator: Natasha Tonge
creator: Nicole Walasek
creator: Marlene Werner
creator: Denny Borsboom
rights: Copyright (c) 2025 Noah van Dongen, Adam Finnemann, Jill de Ron, Leonid Tiokhin, Shirley B. Wang, Johannes Algermissen, Elena C. Altmann, Štěpán Bahník, Li-Ching Chuang, Andrei Dumbravă, Jens H. Fünderich, Sandra J. Geiger, Daria Gerasimova, Aidai Golan, Judith Herbers, Marc Jekel, Anton Kunnari, Yih-Shiuan Lin, David Moreau, Yvonne Oberholzer, Hannah K. Peetz, Julia Rohrer, Adrian Rothers, Felix Schönbrodt, Yashvin Seetahul, Anna Szabelska, Natasha Tonge, Nicole Walasek, Marlene Werner, Denny Borsboom https://creativecommons.org/licenses/by/4.0/
date: 2025-04-15
publication date: 2025-04-15
doi: 10.15626/MP.2023.3688

Replication value as a function of citation impact and sample size
type: item
description: Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication effort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2023). In this article, we propose a quantitative operationalization of replication value. We first discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.
subject: Special Topic
  topic: section
subject: replication value
  topic: keywords
subject: replication
  topic: keywords
subject: study selection
  topic: keywords
subject: expected utility
  topic: keywords
subject: RVcn
  topic: keywords
creator: Peder Isager
creator: Anna van 't Veer
creator: Daniël Lakens
rights: Copyright (c) 2025 Peder Isager, Anna van 't Veer, Daniël Lakens https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2022.3300

Overview on the Null Hypothesis Significance Test
type: item
description: For decades, waxing and waning, there has been an ongoing debate on the values and problems of the ubiquitously used null hypothesis significance test (NHST). With the start of the replication crisis, this debate has flared-up once again, especially in the psychology and psychological methods literature. Arguing for or against the NHST method usually takes place in essays and opinion pieces that cover some, but not all the qualities and problems of the method. The NHST literature landscape is vast, a clear overview is lacking, and participants in the debate seem to be talking past one another. To contribute to a resolution, we conducted a systematic review on essay literature concerning NHST published in psychology and psychological methods journals between 2011 and 2018. We extracted all arguments in defense of (20) and against (70) NHST, and we extracted the solutions (33) that were proposed to remedy (some of) the perceived problems of NHST. Unfiltered, these 123 items form a landscape that is prohibitively difficult to keep in one’s sights. Our contribution to the resolution of the NHST debate is twofold. 1) We performed a thematic synthesis of the arguments and solutions, which carves the landscape in a framework of three zones: mild, moderate, and critical. This reduction summarizes groups of arguments and solutions, thus offering a manageable overview of NHST’s qualities, problems, and solutions. 2) We provide the data on the arguments and solutions as a resource for those who will carry-on the debate and/or study the use of NHST.
subject: Original articles
  topic: section
subject: null hypothesis significance test
  topic: keywords
subject: systematic review
  topic: keywords
subject: essay literature
  topic: keywords
subject: opinion literature
  topic: keywords
subject: thematic synthesis
  topic: keywords
creator: Noah Dongen
creator: Leonie van Grootel
rights: Copyright (c) 2025 Noah Dongen, Leonie van Grootel https://creativecommons.org/licenses/by/4.0/
date: 2025-05-07
publication date: 2025-05-07
doi: 10.15626/MP.2021.2927

Comments on Cassese and Holman 2019 “Playing the Woman Card: Ambivalent Sexism in the 2016 U.S. Presidential Race”
type: item
description: During his campaign for the Republican Party nomination and for U.S. president, Donald Trump suggested that Hillary Clinton benefited from playing a “woman card”. The effect of exposure to Trump’s woman-card attack was investigated in the Cassese and Holman (2019) Political Psychology article “Playing the woman card: Ambivalent sexism in the 2016 U.S. presidential race”. However, neither Cassese and Holman (2019) nor a reanalysis of data analyzed in the article provided sufficient evidence for key claims in the article. Moreover, Cassese and Holman (2019) is unclear whether its Study 2 experimental data could be used to test claims made based on its Study 1 non-experimental data, providing an example of how journal policy requiring access to survey questionnaires could help peer reviewers and readers better assess reported research.
subject: Commentaries
  topic: section
subject: hostile sexism
  topic: keywords
subject: ambivalent sexism
  topic: keywords
subject: sexism
  topic: keywords
subject: woman card
  topic: keywords
subject: gender
  topic: keywords
subject: Donald Trump
  topic: keywords
subject: Hillary Clinton
  topic: keywords
creator: Lawrence James Zigerell
rights: Copyright (c) 2025 Lawrence James Zigerell https://creativecommons.org/licenses/by/4.0/
date: 2025-12-30
publication date: 2025-12-30
doi: 10.15626/MP.2021.2916

The Evolution of Data Sharing Practices in the Psychological Literature
type: item
description: Sharing data has many benefits. However, data sharing rates remain low, for the most part well below 50%. A variety of interventions encouraging data sharing have been proposed. We focus here on editorial policies. Kidwell et al. (2016) assessed the impact of the introduction of badges in Psychological Science; Hardwicke, Mathur, et al. (2018) assessed the impact of Cognition’s mandatory data sharing policy. Both studies found policies to improve data sharing practices, but only assessed the impact of the policy for up to 25 months after its implementation. We examined the effect of these policies over a longer term by reusing their data and collecting a follow-up sample including articles published up until December 31st, 2019. We fit generalized additive models as these allow for a flexible assessment of the effect of time, in particular to identify nonlinear changes in the trend. These models were compared to generalized linear models to examine whether the non-linearity is needed. Descriptive results and the outputs from generalized additive and linear models were coherent with previous findings: following the policies in Cognition and Psychological Science, data sharing statement rates increased immediately and continued to increase beyond the timeframes examined previously, until reaching close to 100%. In Clinical Psychological Science, data sharing statement rates started to increase only two years following the implementation of badges. Reusability rates jumped from close to 0% to around 50% but did not show changes within the pre-policy nor the post-policy timeframes. Journals that did not implement a policy showed no change in data sharing rates or reusability over time. There was variability across journals in the levels of increase, so we suggest future research should examine a larger number of policies to draw conclusions about their efficacy. We also encourage future research to investigate the barriers to data sharing specific to psychology subfield…
subject: Original articles
  topic: section
subject: journal policies
  topic: keywords
subject: metascience
  topic: keywords
subject: open science
  topic: keywords
subject: reproducibility
  topic: keywords
subject: scientific publishing
  topic: keywords
creator: Judith Neve
creator: Guillaume Rousselet
rights: Copyright (c) 2025 Judith Neve, Guillaume A. Rousselet https://creativecommons.org/licenses/by/4.0/
date: 2025-04-09
publication date: 2025-04-09
doi: 10.15626/MP.2021.2908

Folk Conceptions of Free Will: A Systematic Review and Narrative Synthesis of Psychological Research
type: item
description: The existence of free will has been a subject of fierce academic debate for millennia, still the meaning of the term “free will” remains nebulous. In the past two decades, psychologists have made considerable progress in defining lay concepts of free will. We present the first systematic review of primary psychological evidence on how ordinary folk conceptualise free will, encompassing folk concepts, beliefs, intuitions, and attitudes about free will. A total of 1,384 records were identified following a pre-registered protocol. After abstract and full-text screening, 18 articles were eligible for inclusion, comprised of 36 studies and 10,176 participants from regions including the United States, Singapore, Hong Kong, India, Turkey, and Germany. A narrative synthesis of results showed that for ordinary folk, especially the more educated population from the United States, free will is a dynamic construct centred on the ability to choose following one’s goals and desires, whilst being uncoerced and reasonably free from constraints. Results suggesting metaphysical considerations regarding consciousness, dualism, and determinism were inconclusive. Our findings provided preliminary support for a psychological model of folk conception of free will, and elucidated potential pathways mediating the effects of consciousness and dualism on free will attributions. Further research is needed to explicate the distinction between having free will and having the ability to exercise free will, as well as the cross-cultural validity of findings on folk conceptions of free will.
subject: Original articles
  topic: section
subject: free will
  topic: keywords
subject: choice
  topic: keywords
subject: folk concepts
  topic: keywords
subject: lay beliefs
  topic: keywords
subject: systematic review
  topic: keywords
subject: open science
  topic: keywords
creator: Alison Lam
creator: Siu Kit Yeung
rights: Copyright (c) 2025 Alison Lam, Siu Kit Yeung https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2021.2829

A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles
type: item
description: Many bodies recommend that a sample planning procedure, such as traditional NHST a priori power analysis, is conducted during the planning stages of a study. Power analysis allows the researcher to estimate how many participants are required in order to detect a minimally meaningful effect size at a specific level of power and Type I error rate. However, there are several drawbacks to the procedure that render it “a mess.” Specifically, the identification of the minimally meaningful effect size is very challenging, the procedure is not precision oriented, and does not guide the researcher to collect as many participants as feasibly possible. In this study, we explore how these three theoretical issues are reflected in applied psychological research in order to better understand whether these issues are concerns in practice. To investigate how power analysis is currently used, this study reviewed the reporting of 443 power analyses in high impact Psychology journals in 2016 and 2017 using Google Scholar. It was found that researchers rarely use the minimally meaningful effect size as a rationale for the chosen effect in a power analysis. Further, precision-based approaches and collecting the maximum sample size feasible are almost never used in tandem with power analyses. In light of these findings, we offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as collecting the maximum sample size feasible.
subject: Original articles
  topic: section
subject: power analysis
  topic: keywords
subject: statistical power
  topic: keywords
subject: Type II error rate
  topic: keywords
subject: precision-based sample planning
  topic: keywords
subject: minimally meaningful effect size
  topic: keywords
creator: Nataly Beribisky
creator: Udi Alter
creator: Robert Cribbie
rights: Copyright (c) 2025 Nataly Beribisky, Udi Alter, Robert Cribbie https://creativecommons.org/licenses/by/4.0/
date: 2025-10-29
publication date: 2025-10-29
doi: 10.15626/MP.2020.2643