Means to valuable exploration: I. The blending of confirmation and exploration and how to resolve it

Abstract

Data exploration has enormous potential to modify and create hypotheses, models, and theories. Harnessing the potential of transparent exploration replaces the common, flawed purpose of intransparent exploration: to produce results that appear to confirm a claim by hiding steps of an analysis. For transparent exploration to succeed, however, methodological guidance, elaboration and implementation in the publication system is required. We present some basic conceptions to stimulate further development. In this first of two parts, we describe the current blending of confirmatory and exploratory research and propose how to separate the two via severe testing. A claim is confirmed if it passes a test that probably would have failed if the claim was false. Such a severe test makes a risky prediction. It adheres to an evidential norm with a threshold, usually p < α = .05, but other norms are possible, for example, with Bayesian approaches. To this end, adherence requires control against questionable research practices like p-hacking and HARKing. At present, preregistration seems to be the most feasible mode of control. Analyses that do not adhere to a norm or where this cannot be controlled should be considered as exploratory. We propose that exploration serves to modify or create new claims that are likely to pass severe testing with new data. Confirmation and exploration, if sound and transparent, benefit from one another. The second part will provide suggestions for planning and conducting exploration and for implementing more transparent exploratory research.