The #BayesInPsych Digital Event kicked off yesterday and as the leading Guest Editor of the special issue of Psychonomic Bulletin & Review, I take this opportunity to provide more context for this week’s posts.
The simple act of deciding which among competing theories is most likely—or which is most supported by the data—is the most basic goal of empirical science, but the fact that it has a canonical solution in probability theory is seemingly poorly appreciated. It is likely that this lack of appreciation is not for want of interest in the scientific community; rather, I suspect that many scientists hold misconceptions about statistical methods.
Indeed, many psychologists attach false probabilistic interpretations to the outcomes of classical statistical procedures (p-values, rejected null hypotheses, confidence intervals, and the like; we know this from papers such as those by Gigerenzer, 1998; Hoekstra, Morey, Rouder, & Wagenmakers, 2014; Oakes, 1986). You can also use yesterday’s post to catch a quick glimpse at the reasons for those misconceptions, and the adverse consequences they may entail.
Because the false belief that classical methods provide the probabilistic quantities that scientists need is so widespread, researchers may be poorly motivated to abandon these practices.
The American Statistical Association recently published an unusual warning against inference based on p-values (Wasserstein & Lazar, 2016). Unfortunately, their cautionary message did not conclude with a consensus recommendation regarding best-practice alternatives, leaving something of a recommendation gap for applied researchers.
In psychological science, however, a replacement had already been suggested in the form of the “New Statistics” (Cumming, 2014)—a set of methods that focus on effect size estimation, precision, and meta-analysis, and that would forgo the practice of ritualistic null hypothesis testing and the use of the maligned p-value. However, because the New Statistics’ recommendations regarding inference are based on the same flawed logic as the thoughtless application of p-values, they are subject to the same misconceptions and are lacking in the same department. It is not clear how to interpret effect size estimates without also knowing the uncertainty of the estimate.
And despite common misconceptions, confidence intervals do not measure uncertainty; Morey, Hoekstra, Rouder, Lee, & Wagenmakers, 2016). You may recall that we ran a Digital Event that was dedicated to those common misconceptions about confidence intervals. The “New Statistics” also does not tell us how to decide which among competing theories is most supported by data.
In the #BayesInPsych special issue of Psychonomic Bulletin & Review, we review a different set of methods and principles, now based on the theory of probability and its deterministic sibling, formal logic (Jaynes, 2003; Jeffreys, 1939). The aim of the special issue is to provide and recommend this collection of statistical tools that derives from probability theory: Bayesian statistics.
Overview of the special issue on Bayesian inference
The special issue is divided into four sections. The first section is a coordinated five-part introduction that starts from the most basic concepts and works up to the general structure of complex problems and to contemporary issues. The second section is a selection of advanced topics covered in-depth by some of the world’s leading experts on statistical inference in psychology. The third section is an extensive collection of teaching resources, reading lists, and strong arguments for the use of Bayesian methods at the expense of classical methods. The final section contains a number of applications of advanced Bayesian analyses that provides an idea of the wide reach of Bayesian methods for psychological science.
Section I: Bayesian Inference for Psychology
The special issue opens with Introduction to Bayesian inference for psychology, in which Etz and Vandekerckhove describe the foundations of Bayesian inference. They illustrate how all aspects of Bayesian statistics can be brought back to the most basic rules of probability, and show that Bayesian statistics is nothing more nor less than the systematic application of probability theory to problems that involve uncertainty.
Wagenmakers, Marsman, et al. continue in Part I: Theoretical advantages and practical ramifications by illustrating the added value of Bayesian methods, with a focus on its desirable theoretical and practical aspects. Then, in Part II: Example applications with JASP, Wagenmakers, Love, et al. showcase JASP: free software that can perform the statistical analyses that are most common in psychology, and that can execute them in both a classical (frequentist) and Bayesian way.
However, the full power of Bayesian statistics comes to light in its ability to work seamlessly with far more complex statistical models. In Part III: Parameter estimation in nonstandard models, Matzke, Boehm, and Vandekerckhove discuss the nature of formal models and how to implement models of high complexity in modern statistical software packages.
Rounding out the section, Rouder, Haaf, and Vandekerckhove in Part IV: Parameter estimation and Bayes factors discuss the fraught issue of estimation-versus-testing. The paper illustrates that the two tasks are one and the same in Bayesian statistics, and that the distinction in practice is not a distinction of method but of how hypotheses are translated from verbal to formal statements.
Section II: Advanced Topics
The Advanced Topics section covers three important issues that go beyond the off-the-shelf use of statistical analysis. In Determining informative priors for cognitive models, Lee and Vanpaemel highlight the sizable advantages that prior information can bring to the data analyst become cognitive modeler.
Because Bayesian analyses are not in general invalidated by “peeking” at data, the use for sample size planning and power analysis is somewhat diminished. Nonetheless, it is sometimes useful for logistical reasons to calculate ahead of time how many participants a study is likely to need. In Bayes factor design analysis: Planning for compelling evidence, Schoenbrodt and Wagenmakers provide exactly that.
Finally, there arise occasions where even the most sophisticated general-purpose software will not meet the needs of the expert cognitive modeler. In A simple introduction to Markov chain Monte-Carlo sampling, van Ravenzwaaij, Cassey, and Brown describe the basics of sampling-based algorithms and illustrate how to construct custom algorithms for Bayesian computation.
Section III: Learning and Teaching
Four articles make up the Learning and Teaching section. The goal of this section is to collect the most accessible, self-paced learning resources for an engaged novice.
While it is of course their mathematical underpinnings that support the use of Bayesian methods, their intuitive nature provides a great advantage to novice learners. In Bayesian data analysis for newcomers, Kruschke and Liddell cover the basic foundations of Bayesian methods using examples that emphasize this intuitive nature of probabilistic inference.
With The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and planning from a Bayesian perspective, Kruschke and Liddell lay out a broad and comprehensive case for Bayesian statistics as a better fit for the goals of the aforementioned New Statistics (Cumming, 2014).
How to become a Bayesian in eight easy steps is notable in part because it is an entirely student-contributed paper. Etz, Gronau, Dablander, Edelsbrunner, and Baribault thoroughly review a selection of eight basic works (four theoretical, four practical) that together cover the bases of Bayesian methods.
The fourth and final paper in the section for teaching resources is Four reasons to prefer Bayesian analyses over significance testing by Dienes and McLatchie. It is likely that widespread misconceptions about classical methods have made it seem to researchers that their staple methods have the desirable properties of Bayesian statistics that are, in fact, missing (as we noted yesterday). Dienes and McLatchie present a selection of realistic scenarios that illustrate how classical and Bayesian methods may agree or disagree, proving that the attractive properties of Bayesian inference are often missing in classical analyses.
Section IV: Bayesian Methods in Action
The concluding section contains a selection of fully-worked examples of Bayesian analyses. Three powerful examples were chosen to showcase the broad applicability of the unifying Bayesian framework.
The first paper, Fitting growth curve models in the Bayesian framework by Oravecz and Muth, provides an example of a longitudinal analysis using growth models. This framework is likely to gain prominence as more psychologists focus on the interplay of cognitive, behavioral, affective, and physiological processes that unfold in real time and whose joint dynamics are of theoretical interest.
In a similar vein, methods for dimension reduction have become increasingly useful in the era of Big Data. In Bayesian latent variable models for the analysis of experimental psychology data, Merkle and Wang give an example of an experimental data set whose various measures are jointly analyzed in a Bayesian latent variable model.
The final section of the special issue is rounded out by Sensitivity to the prototype in children with high-functioning autism spectrum disorder: An example of Bayesian cognitive psychometrics by Voorspoels, Rutten, Bartlema, Tuerlinckx, and Vanpaemel. The practice of cognitive psychometrics involves the construction of often complex nonlinear random-effects models, which are typically intractable in a classical context but pose no unique challenges in the Bayesian framework.
Additional coverage
As part of our efforts to make our introductions to Bayesian methods as widely accessible as possible, we have set up a social media help desk where questions regarding Bayesian methods and Bayesian inference, especially as they are relevant for psychological scientists, are welcomed. This digital resource is likely to expand in the future to cover new developments in the dissemination and implementation of Bayesian inference for psychology.
Finally, we have worked to make many of the contributions to the special issue freely available online. The full text of many articles is freely available via the Open Science Framework. Here, too, development of these materials is ongoing, for example with the gradual addition of exercises and learning goals for self-teaching or classroom use.
References
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.
Gigerenzer, G. (1998). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199–200.
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.- J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21, 1157–1164.
Jaynes, E. T. (2003). Probability theory: The logic of science. Cambridge: Cambridge University Press.
Jeffreys, H. (1939). Theory of probability (1st ed.). Oxford, UK: Oxford University Press.
Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103–123.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p–values: Context, process, and purpose. The American Statistician.
This post is an abridged and edited version of the Editorial to the special issue on #BayesInPsych that was co-authored with Jeffrey N. Rouder and John K. Kruschke.