How Can I Study from Below, that which Is Above?

Comparing Replicability Estimated by Z-Curve to Real Large-Scale Replication Attempts

Abstract

Z-curve is an analytic technique with which one can estimate the percent of a set of studies of interest that would replicate if one were to run actual replication studies. I compared the estimates z-curve yields to the outcome of real large-scale replication studies, such as the Open Science Collaboration (2015) work or the various ManyLabs projects (e.g., Klein et al., 2014). I collected  p-values from the original studies examined in six different large-scale replication efforts to the extent possible, ran z-curves on all the original studies, and compared the z-curve results to the results of the actual replication studies. My results show that across 163 replication studies taken from the six replication efforts, 85 (52.15%) showed statistically significant resultsin the expected direction as indicated by the authors of the replication studies. The outcome of the z-curve of all these studies was accurate, with the midpoint between the expected replication rate and the expected discovery rate, 50.55%, being almost exactly the same as the true replication rate. Its replicability estimate was also more accurate than that of p-curve analysis. Comparison of z-curve analysis of studies that did successfully replicate to studies that did not does suggest heterogeneity in the accuracy of its estimates, however. The pros and cons of z-curve analysis are discussed.