# How Can I Study from Below, that which Is Above?

## Comparing Replicability Estimated by Z-Curve to Real Large-Scale Replication Attempts

## DOI:

https://doi.org/10.15626/MP.2022.3299## Keywords:

z-curve, p-curve, replicability, replication studies## Abstract

Z-curve is an analytic technique with which one can estimate the percent of a set of studies of interest that would replicate if one were to run actual replication studies. I compared the estimates *z*-curve yields to the outcome of real large-scale replication studies, such as the Open Science Collaboration (2015) work or the various ManyLabs projects (e.g., Klein et al., 2014). I collected *p*-values from the original studies examined in six different large-scale replication efforts to the extent possible, ran *z*-curves on all the original studies, and compared the *z*-curve results to the results of the actual replication studies. My results show that across 163 replication studies taken from the six replication efforts, 85 (52.15%) showed statistically significant resultsin the expected direction as indicated by the authors of the replication studies. The outcome of the *z*-curve of all these studies was accurate, with the midpoint between the expected replication rate and the expected discovery rate, 50.55%, being almost exactly the same as the true replication rate. Its replicability estimate was also more accurate than that of *p*-curve analysis. Comparison of *z*-curve analysis of studies that did successfully replicate to studies that did not does suggest heterogeneity in the accuracy of its estimates, however. The pros and cons of z-curve analysis are discussed.

### Metrics

## Downloads

## Published

## Issue

## Section

## License

Copyright (c) 2023 Lukas Sotola

This work is licensed under a Creative Commons Attribution 4.0 International License.