A Brief Guide to Evaluate Replications

  • Etienne Philippe LeBel University of Western Ontario
  • Wolf Vanpaemel
  • Irene Cheung
  • Lorne Campbell
Keywords: transparency, reproducibility, direct replication, replicability, evaluating replications;


The importance of replication is becoming increasingly appreciated, however, considerably less consensus exists about how to evaluate the design and results of replications. We make concrete recommendations on how to evaluate replications with more nuance than what is typically done currently in the literature. We highlight six study characteristics that are crucial for evaluating replications: replication method similarity, replication differences, investigator independence, method/data transparency, analytic result reproducibility, and auxiliary hypotheses’ plausibility evidence. We also recommend a more nuanced approach to statistically interpret replication results at the individual-study and meta-analytic levels, and propose clearer language to communicate replication results.

Original articles