Quests

A quest is a journey or mission to some goal, usually fraught with obstacles, twists and turns, and hopefully, some epiphanies along the way.

The Quest for Statistical “Truth”

Although most psychologists and statisticians “know” statistical “truth” to be an oxymoron, as scientists, we strive to determine the “best” or “most probable” explanation based on the statistical probabilities produced from a research study, its measurements, its sample, and the magic of statistical analyses. And while the goal of each study is to minimize error and maximize the probability of detecting a meaningful effect or relationship (yet another quest fraught with many temptations and pitfalls), if it is present, over the course of psychological science, the ability to replicate previously published research findings has produced questionable results, a.k.a. the replication crises in psychological science.

Multiple solutions have been offered to address the replication crisis, including framing it as a base rate fallacy, incorporating Bayesian statistics to address both statistical and scientific inferences, or conducting meta-analyses, a statistical tool in which the effect sizes of previously published and, ideally, unpublished research studies, are aggregated into a single study and statistic that provides a more accurate estimate of the effect size across samples and methodologies. Although some scientists do not agree that meta-analyses are the panacea, there does appear to be some “trust” in the conclusions when meta-analyses are done thoroughly and well.

Avoiding Lotus Flowers, Cyclops, and Sirens

In an article published by the Psychonomic Society’s Behavior Research Methods, Belén Fernández-Castilla, Sameh Said-Metwaly, Rodrigo Kreitchmann, and Wim Van Den Noortgate provided a set of guidelines and a checklist, the SEMI, for authors of primary research studies to follow when publishing the findings of their studies. Guidelines on reporting quantitative research already exist:

• American Psychological Association (APA) guidelines for reporting quantitative research (Applebaum et al., 2018)
• Preferred Reporting Items for Systematic reviews and Meta-Analyses PRISMA review guidelines (Page et al., 2020)

However, a comprehensive list of information needed for a quality meta-analysis has not been compiled to date. Fernández-Castilla and coauthors met this challenge and created a thorough set of steps to be followed by researchers interested in conducting a meta-analysis.

The Hitchhiker’s Guide to the Galaxy and Statistical “Truth”

Step one. Conduct a systematic literature review where the topic of interest will be searched, and resulting papers will be screened for pre-determined inclusion criteria. Recommended search strategies include

1. identifying critical keywords that can be streamlined by restricting the keyword search to the title or abstract
2. conducting a backward search by utilizing primary study references
3. examining the full text of a relevant paper

APA guidelines recommend that authors incorporate critical keywords within their titles or abstracts and clearly present the research objectives and outcomes within the relevant sections of their papers (i.e., introduction or discussion).

Step two. Code the literature selected for inclusion for specific information that will aid in conducting the meta-analysis. Fernández-Castilla and coauthors recommended several different frameworks that can help identify critical, relevant information from a primary source: PICO, SPICE, SPIDER. The main takeaway from these frameworks is that they provide keys to maintaining power for the study (i.e., keep as many primary studies from the search as possible) by understanding the sample (all relevant characteristics), the study conditions, the variables of interest (explanatory/factors and outcomes), and evaluation tools that depend on study design.

Step three. Calculate the index or effect size measurement. This step is where the details really matter. It is not enough to just report the effect size. Authors should report the specific formula used to obtain the effect size, along with information that can help with the calculation, including sampling variance or sample size. Fernández-Castilla and coauthors provided a useful table summarizing commonly used formulas for effect size and their relationships to specific study designs.

Seeing the Future

Fernández-Castilla and coauthors provided their top recommendations of how primary study researchers can help meta-analysts. Beginning with the importance of open science, pre-registered studies, and publicly available datasets and code used to conduct analyses, the authors illustrated how these actions can remove barriers to achieving more accurate findings. (Too bad Odysseus didn’t have these opportunities!) Two additional but related recommendations were also provided:

1. primary authors should report statistics on the whole sample both with and without excluded data,
2. descriptive statistics regarding the relationship between all variables within the study, regardless of the type of variables (categorical or quantitative), should also be provided.

The final recommendation is to be aware of the possibility of unpublished negative findings. Whether that is to put a request out on relevant listservs to locate those studies or to encourage journals to publish quality studies with “null” results more frequently, the authors offered some excellent suggestions (i.e., registered reports) while also reminding us that it is our ethical obligation to utilize the information provided by our participants.

Finding the Grail

The SEMI checklist below lists the 28 pieces of information from primary sources that would aid future researchers interested in conducting meta-analyses on a related topic. As shown in the image below, the checklist is divided into five sections (Title and abstract, Background, Methods, Results, Open sciences and practices) and can be used concurrently with other checklists, such as PRISMA.