We all want to be free. Freedom of choice is perhaps one of the most foundational principles of western societies, and it can empower lives. As academics, we cherish academic freedom, and academic freedom has been called essential to democracy for very good reasons.

However, no matter how enthusiastically we may endorse academic freedom, we probably would be a little more cautious to endorse the freedom to fool ourselves (or indeed others). There has been much recent discussion, including on this blog, about researchers’ degrees of freedom and how they might fool us into unwarranted beliefs.

As our digital team member Joachim Vandekerckhove noted:

“It is not always recognized that researchers who analyze a new data set, and implicitly traverse a potentially tall decision tree of choices on the way, possess a number of degrees of freedom also.  For example, a researcher analyzing reaction time data might choose to discard fast responses below 200 ms, or below 250 or 300 ms, or they might not discard fast responses at all.  Independently, they might discard slow responses above 2000 ms, 3000 ms, or any number that seems reasonable.  Then, they might apply a variance stabilizing transformation to the reaction times—perhaps a logarithmic transform, or the inverse.”

Should we be free to make those choices?

Yes—but only at certain times and not others.

Because if we collect the data first and then decide how to analyze them, then the decisions about what cutoff to use on the reaction times can be based on their beneficial effect on, umm, the desired outcome.

And that is when we risk turning into the Texas sharpshooter.

Legend has it that in particularly rural regions of Texas (there are many),  sharpshooters market their skills by first firing a shotgun at a barn door and then painting a bull’s eye around their preferred hole. That’s freedom for you, but it’s a freedom that fools people.

So yes, I should be free to choose a lower cutoff of 200 ms, an upper cutoff of 3,000 ms and a logarithmic transform, but I should exercise that freedom not because it makes the effect significant. And the easiest way to guarantee my freedom of choice while preventing me from fooling myself or others is to make the choices before the data are known.

Which is precisely the point of preregistration.

Preregistration means that the bull’s eye is drawn onto the barn door before the shotgun is fired. By putting on record the motivation, design, and a sampling and analysis plan of your experiment before you conduct it, you cannot inadvertently find a target by painting it on the barn afterwards.

The steps to preregistration are simple: several online public repositories exist that specialize in putting any collection of documents into a data base. For members of the Psychonomic Society, the repository that is most likely to be relevant is the Open Science Framework www.osf.io. (A great introduction is here).

But the picture is a little more nuanced, and opinions on preregistration diverge.

On the one hand, scientists have claimed that preregistrations are becoming the norm in psychological science. On the other hand, commentators have likened their utility to Basil Fawlty’s suitcase full of hippopotamus sauce.

In the aggregate, experts are favourably inclined towards preregistration and believe that it increases the likelihood of a study replicating. This conclusion is based on an expert survey that Klaus Oberauer and I conducted at the international meeting of the Psychonomic Society in Amsterdam last year. We have reported the results on this blog already, but highlight the responses relating to preregistration again.

More than 100 respondents volunteered to take the survey. Each item involved a quasi-continuous scale with marked end points, and respondents indicated their response by placing a tick mark along the scale. We asked our experts two questions relating to preregistration: First, whether a study whose method was preregistered would be more likely to replicate than one without preregistration, and second, whether a preregistration of method and analysis plan would herald greater replicability than a preregistration of the method alone.

The results are shown in the histograms below, which show the distribution of responses:

It is quite clear that a strong majority of experts favored preregistration (over doing nothing) and an even stronger majority placed more faith in a preregistered analysis plan than in preregistration of the method alone.

This is encouraging, but it does not follow that the issue is resolved.

Quite on the contrary, as preregistration becomes more common and turns into a norm, we must ensure that this norm is fully understood by members of the community. The digital event #PSprereg that is commencing tomorrow here provides a diverse set of perspectives on preregistration:

• Monday               Steve Lindsay (University of Victoria)
• Tuesday               Klaus Oberauer (University of Zurich)
• Wednesday          Richard Morey (Cardiff University)
• Thursday             Danielle Navarro (University of New South Wales)
• Friday                  Iris van Rooij (Radboud University Nijmegen)

This initial line-up will be followed by at least one more post by me, and if there is appetite for a continuation—given the importance of the issue—please get in touch with suggestions for contributions (or contributors) and we may extend the event a little further.

We have run an extended event before, when we discussed another important issue, namely diversity and inclusion.

Just in case you are still wondering about the hippopotamus sauce, it is explained here.