It is too early to know whether the recent period of methodological introspection in psychological science, and the sciences in general, will lead to positive changes in practices. As with any revolution, there is the potential of moving backward. One problem that has been consistently acknowledged is the problem of incentives: what gets one attention and advancement is not necessarily good for science. These incentives are just as real for reformers as they are for everyone else, and so we must be careful. Under these circumstances, even a well-thought-out reform might backfire if it is oversold.

I have seen this with Bayesian statistics. As discomfort with existing methods spreads, some authors have included Bayesian analyses as a way to differentiate their manuscript from others. Inclusion of ill-considered Bayesian analyses when they don’t add to the manuscript actually makes a manuscript less transparent and adds more potential for statistical errors. That good statistical practices are not incentivised is part of the problem; any reform that does not address the core problem will become part of the problem rather than part of the solution.

I have fears about another reform: preregistration. I believe preregistration is an important tool for scientists to increase transparency, clarify thinking, and make science more systematic. These are important goals, and when used in the right context, preregistration—and its big brother Registered Report—has the potential to help improve science. The current incentive structure, however, might cause problems rather than solve them. I will focus here on some implications for preregistration of statistical analyses, which have been offered as a solution for p-hacking and other opportunistic practices. My post therefore connects well to yesterday’s contribution by Klaus Oberauer.

Opportunistic practices are real and they have multiple causes, including incentives to publish and human cognitive biases. Let’s consider what results preregistered analyses might produce in such an environment. First, we imagine we have a system where preregistered analyses are trusted more, and so people begin to preregister their analyses to help them publish. They may have a weak understanding of statistics, but they typically perform the same (parametric) statistical test for every data set. They might preregister this test, along with any assumption checks SPSS returns by default.

Preregistration prevents them from, for instance, transforming the data in multiple ways to obtain a significant result—at least, they cannot do this without suspicion (as long as someone reads the preregistration document). That is, it prevents them doing what bad scientists do. This can be a good thing. But what would a good scientist do? A bad scientist opportunistically looks for reasons to trust their own hypothesis. A good scientist does the opposite: she opportunistically looks for reasons to doubt themselves, including their analysis and hypothesis. They check for robustness, both in ways that they might have thought of before (and potentially preregistered) and ways they might not have.

A bad scientist illicitly mines “truth” from their data; a good scientist mines doubt.

An advocate of preregistration will note that preregistration does not prevent checking for robustness. So-called “exploratory” analyses are allowed, and so a good scientist is not prevented from due diligence. But the reason why preregistration is being suggested now is to prevent bad practices. If good statistical practices were properly incentivised, we would not have a methodological crisis in the first place. The scientist who was willing to hide the fact that they performed multiple analyses and picked the best, or who opportunistically removed “outliers”, will also be willing to ignore clues that their data is not as high-quality as they thought, or that their analysis is mis-specified. Why would they tug on that thread, and watch their entire paper unravel?

Alternative approaches to preregistration exist. One that I think emulates what good scientists do better than preregistration is multiverse analysis (Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2015). Multiverse analysis formalizes what a good data analyst would do: approaching a data analysis problem in many ways and getting a sense of the robustness of the result. The multiverse analysis is one way of looking for reasons to doubt your conclusions, in spite of the fact that one test might seem to confirm your hypothesis. (Yesterday’s post also touched on this.)

But even this is not enough. We need to incentivise people to tug the thread on their own analyses. If preregistration might help people ignore potential problems, what might we do to counteract the problem? One possibility is the expectation of data sharing and more secondary analysis of data. If you knew that someone else might tug the thread later, you’d have an incentive to make sure your analysis is robust.

People also need to be asked to move away from simple default analyses like t-tests with bar plots. Alternative ways of approaching an analysis, such as nonparametric statistics, are not just something you do in special cases (such as when a default assumption check fails). Recently, in response to my suggestion that nonparametric methods be taught to undergraduate students alongside parametric ones, I was told that this was unnecessary because one could always just look up the nonparametric test in a book when you needed it. This attitude clearly shows the tyranny of the default: the “standard” methods are the ones you need, except in special cases. Looking at a statistical problem from multiple perspectives as a part of one’s scientific process is a foreign idea to many people. This attitude needs to change.

In a world where researchers only use a single test type and a single graphical method—and where their statistical software allows little else—the problem is not that preregistration is a straightjacket, but rather that it actually lulls one into thinking that methodological vice is virtue. The tyranny of the default is real, and any practice that encourages such thinking should be carefully considered.

There is much to lose if preregistered analyses become a trusted gold standard instead of useful tool in some contexts.