Richard Shiffrin’s 2018 PNAS article on the nature of scientific progress is beautifully written, erudite, and insightful. I learned a lot from it and agree with most of his arguments. But with the greatest respect I would like to counter Shiffrin’s expressions of doubt regarding the value of preregistering research plans. Those doubts were lightly touched on in the PNAS article but were, reportedly, more strongly emphasized in Shiffrin’s Psychonomics talk on 16 November 2018 (I missed this session, but the slides are here).
Preregistration Versus Registered Reports
It is important to distinguish between Registered Reports (RRs) and preregistrations. Sometimes people confuse the two. Susan Goldin-Meadow published an Observer piece in 2016 that was entitled “Why Preregistration Makes me Nervous” but to my eye the concerns pertain to RRs, not preregistration. Rich Shiffrin’s slides suggest that he too had RRs in mind when arguing against preregistration. (I was not involved in the development of this terminology.)
With an RR, a researcher submits a proposal to conduct a study to a journal and the editor sends it to expert reviewers; if the proposal is accepted then the journal commits to publishing a report of the study provided that it is completed to specifications, regardless of how the key hypothesis tests come out. More information about that is here.
RRs have many fine qualities and I think it is terrific that more and more journals are inviting RR submissions (thanks in no small part to the tireless efforts of Chris Chambers; see, .e.g., this blog post). But, in my opinion, RRs are not well suited for all types of research. For one thing, they do not seem apt (albeit possible) for purely exploratory research. For another, RRs necessitate a long delay between initially developing a research idea and beginning data collection. This is because it is likely to take quite a few weeks for the proposed protocol to be reviewed, revised, and negotiated. Doubtless the RR review process usually enhances the quality of the plan. But delay has costs. It might, for example, not be feasible for an honours or masters student to develop and conduct a thesis as an RR. Also, researchers who use within-subject designs with readily available subjects tested on hundreds of trials per condition could conduct several high-powered direct replications in the time it would take to get an initial action letter on an RR submission. Sometimes a time-sensitive idea for research is inspired by an environmental event (e.g., a fire or flood). Finally, although I appreciate the value of publishing informative null results, in my experience even the best laid plans often go awry. Again, RRs have many virtues. I salute those involved in promoting and conducting them. As Editor of Psychological Science I added RRs in the context of direct replications of findings originally reported in Psychological Science (see my editorial on this submission category). But I do not believe that RRs are universally appropriate for psychological science (and maybe that’s what Rich Shiffrin meant too).
Be that as it may, the key point for present purposes is that RRs are qualitatively different from preregistration. Preregistering a research plan merely involves creating a time-stamped, immutable record of one’s plans for that study (and making that plan available to reviewers if the work is submitted for publication and to readers if the work is published). The “pre” part has to do with making the plan before the data are examined. The “registration” part has to do with using a website such as the Open Science Framework or AsPredicted.org to create a time-stamped, immutable record of the plan.
Questions and Responses Regarding Preregistration
Does a preregistration have to specify every single detail of a study in advance? No. A preregistered research plan should provide as much detail as is appropriate for the project. For an initial foray into a new area of research, a preregistration might do little more than describe the domain in broad terms and make clear that the researcher has left most facets of the work open to be resolved on the fly. Such a document might be called an “exploratory preregistration.” Such documents may help researchers later avoid falling prey to hindsight bias and mistaking post hoc decisions for a priori ones and/or serendipitous findings for hypothesized ones.
For a hypothesis-testing experiment in a well-established research paradigm, a preregistration can be exquisitely detailed, including fine points such as the sampling procedure, data-collection stopping rule, exclusion rules, and specification of what measures are to be analyzed in what statistical models and with what hypotheses and criteria. Such a preregistration helps establish (for the researcher, reviewers, and readers) that those aspects of the study were indeed planned prior to data collection. For computer-run studies, the preregistration can also include the program that runs the study as well as scripts for the planned data analyses.
For some projects, the preregistration might be at an intermediate level of detail, making clear to all parties which aspects of the study were planned in advance and which were not. It may often make sense to begin with exploratory preregistrations and gradually home in a fully detailed plan across a series of studies. To warrant a preregistration badge, the plan must be sufficiently detailed that it substantially constrains “researcher degrees of freedom.” Further information can be found here.
Why is it important to know which aspects of a study were planned in advanced versus decided upon during the course of conducting and analyzing the study? Because the meaning of the p-values yielded by inferential statistical tests such as t-tests or ANOVAs becomes opaque if the data influenced decisions as to which measures were submitted to which analyses. It is key to the logic of null hypothesis significance testing that decisions as to which measures are analysed in which ways are not based on the findings that happened to be obtained. (See also yesterday’s post that introduced this “Texas sharpshooter” problem.) This is the crux of the matter. For further information, see Simmons, Nelson, and Simonsohn’s (2011) article on “False Positive Psychology” and the more recent Annual Review chapter by Nelson, Simmons, and Simonsohn (2018). These authors coined the term “p-hacking,” referring to practices such as (a) selectively excluding subjects, observations, measures, or conditions; (b) adding covariates or interaction terms post hoc; (c) tinkering with data transforms, etc. These and related points have been raised and elaborated by many other researchers. To cite but one example, Kerr (1998) presciently detailed the perils of what he termed HARKing, for “hypothesizing after the results are known.” Letting the results influence decisions regarding which measures are submitted to which analyses inflates the Type I error rate and exaggerates the estimated sizes of real effects.
Isn’t preregistration essentially the same as developing research plans, as commonly done in grant proposals, theses, lab meetings, etc.? Yes, in many ways preregistration is not that different from standard planning of a study, as labs have done forever, except that (a) it is typically a bit more formalized, (b) fewer decisions are deferred, (c) part of the exercise is to make a permanent record of the plan, prepared in such a way that other competent researchers can assess it, and (d) when a report of the study is submitted for publication (or published) the preregistration is shared with reviewers (or readers) to help them evaluate the findings.
Does preregistering prevent researchers from changing their plans in response to exigencies that arise in the course of collecting or analyzing data? No, researchers can depart from a preregistered research plan whenever they judge it appropriate to do so. But if they later submit a manuscript reporting that study, then they must describe therein any such departures and explain the rationales for them. Readers can then judge those arguments on their merits.
Does preregistration discourage exploratory analyses? No. Having preregistered a plan for research does not prevent researchers from conducting exploratory analyses of the data post hoc. It is a good and proper thing to forage around in your data, looking at it every which way in an exploratory spirit. And it is also good to know while doing so that this is what you are doing, and to remember later which patterns observed during such explorations were not predicted a priori. Serendipitous observations can be reported as such and inspire hypotheses to be tested in future preregistered research.
Does preregistering put researchers at risk of being scooped? No, or not necessarily. Preregistrations do not have to be made public. The Open Science Framework (OSF) requires that embargoes on registrations be lifted within 5 4 years of registration (on the ground that there is value in making preregistrations searchable by other researchers), but researchers can if they wish remove an OSF preregistration, leaving only basic meta-data. Other sites that provide preregistration (e.g., AsPredicted.org) enable researchers to control access to their preregistrations indefinitely. Thus concern about controlling access is not a strong argument against preregistration.
Doesn’t preregistering slow the pace of research? Yes, compared to just leaping in and making decisions on the fly, creating a detailed preregistration will delay commencement of data collection. That is especially true for the first preregistered study in a line of work. But it also takes time and energy to make decisions on the fly. Also, thinking carefully about plans (e.g., planning sample size, setting data exclusion rules, specifying statistical analyses) before data collection can save a lot of grief later. Sites such as OSF and AsPredicted.org offer templates that can make it quicker and easier to complete a preregistration. And after the first preregistered study in a line of work, subsequent preregistrations are generally trivially easy. So although taking time to prepare a detailed preregistered research plan does delay commencement of data collection, it probably does not slow the pace of research.
Do preregistrations prevent fraud? No, researchers can game preregistrations in various ways. A researcher could, for example, conduct a study using questionable research practices and then write a “preregistration” that describes post hoc analyses as if they had been planned. Or they could deviate from the preregistered plan in various ways but not report those deviations. Or craft a preregistration and make up the data from whole cloth. Preregistration is not about preventing outright fraud. It is about helping well-meaning researchers to remember and communicate to others which aspects of their studies were planned in advance, independent of the results, versus which were adopted post hoc in view of the results.
Do preregistrations make sense for analyses of archival data? Yes, provided the researcher is unaware of the data patterns. If so, then studies of large archival data sets are ideally well-suited for preregistration, because such data sets often have numerous measures and observations and therefore lend themselves to HARKing. An archival project might begin with an exploratory preregistration to study a random subset of cases and then use those results to develop a more detailed preregistration to test hypotheses in another random subset.
Is preregistration always appropriate? I can imagine circumstances in which it might be appropriate to leap into data collection and perhaps even data analysis without formulating and recording a plan (e.g., if a time-limited opportunity suddenly arose or if one was embarking on completely exploratory work in uncharted territory). But even in such situations a brief preregistration that says that there is no plan seems to me to have some potential value and very little cost.
Won’t preregistering make it harder for researchers to find statistically significant results? Yes, to the extent they had previously relied on HARKing and p-hacking. That is a feature, not a bug.
Doesn’t the call for preregistration impugn researcher’s integrity? Some have complained that preregistration is insulting because it implies that researchers cannot be trusted to report which aspects of their studies were versus were not specified in advance. As Sanjay Srivastivas noted in a recent blog post, this is analogous to being offended by the recommendation that testers be kept blind to condition to avoid experimenter effects. It is widely accepted that well-intentioned researchers sometimes unintentionally influence research participants to perform in hypothesis-confirming ways unless blinded to condition (for recent demonstrations, see Gilder & Heerey, 2018). Likewise, absent a detailed preregistration, honest scientists’ decisions about analyses might unwittingly be influenced by outcomes, and they might suffer hindsight bias and believe that serendipitous findings had been hypothesized. Preregistering hypothesis-testing research is just good scientific practice.
Will preregistration solve all of the problems of psychological science? No, it is perfectly possible to preregister bad studies. Even well-designed preregistered studies can yield ambiguous results. Preregistration is more important in some areas of scientific psychology than others (e.g., for various reasons psychophysics would benefit less from preregistration than, say, social cognition; see Brian Scholl’s 2017 APS Observer piece). As Trish Van Zandt argued in her recent Psychonomics presentation (which I also missed but heard about; her slides are here), some sorts of analyses are difficult to preregister in a detailed and specific way. Many of the preregistrations that I have seen as Editor of Psychological Science were of scant value because they were so vague that they failed to make clear which aspects of the study were versus were not specified in advance. More fundamentally, preregistration by itself is not going to solve the great challenges of developing valid and relevant measures and theories that will make psychology a genuinely useful science. Preregistration is simply one of many tools that can help improve research quality.
Conclusion
I am convinced that before conducting a hypothesis-testing experiment it is a good idea to make a record of your plans for that study that clearly specifies which aspects of the study were specified a priori. Registering a copy of that plan adds value at little cost. For further information about preregistration see Lindsay, Simons, and Lilienfeld (2016), van t’Veer & Giner Sorolla (2016), and Wagenmakers (2016)
Acknowledgements
Thanks to Gordon Pennycook for helpful input on an earlier draft and for suggesting the Psychonomic Society’s blog as a potential outlet. Thanks also to Dan Simons and Simine Vazire for insightful suggestions regarding a near-final version of this post.