Imagine a world in which Einstein crafted his general theory of relativity – discovering and codifying the laws that govern light, space, and time – then went about his life, never publishing, and never telling another person.

It may seem obvious, but scientific knowledge, in order to influence the generation of future scientific knowledge or impact society, must be communicated.

Hence, scientific publications – journal articles, book chapters, and conference proceedings – have historically been the primary unit by which scientific careers are measured. Beyond measuring how many articles one has published, evaluating scientific careers typically also takes into account how often those published articles are cited. Each citation reveals a connection to the publication being cited, as if to say “This work is important, and I am building off it.” By a first approximation, citation rates reveal the impact of a publication (and by proxy, the impact of the scientist who wrote it).

But do citation rates purely reflect scientific impact and importance? If that were true, arbitrary factors should not relate to whether a work is cited or not. Of course, deciding whether to cite an article (and discovering articles that you may later cite) is a process that must be a done by a human. A human who, as we have discussed repeatedly on this blog, may have biases and may not even be aware of those biases.

Research on citation rates has revealed, naturally, that human bias creeps into decisions about who to cite. Articles appearing earlier in the table of contents or in list-servs are cited more frequently. Those orderings are likely random, though, so their effects across authors should wash out. (Journal editors typically decide which article to place first in each issue, although with predominantly electronic access this can no longer have much of an effect.)

More worryingly, articles with first author surnames appearing earlier in the alphabet are cited more frequently. But unless we randomly assign authors last names (which may create some confusion), we need to know why this alphabetic bias occurs.

Jeffery Stevens and Juan Duque tackle this question in a recent article appearing in Psychonomic Bulletin & Review. (As an aside, amazingly, it is becoming dull to point out that this article features open data and code, as well as a pre-registered replication of the main findings – yet the authors deserve major kudos for these efforts.) The authors considered two possible sources of bias in citations, both attributable to the primacy effect, which is a cognitive bias for items appearing earlier in lists to be remembered more easily.

The authors reasoned that citation rate would, in part, relate not just to the primacy effect but to the “citation environment” – that is, the way that a journal chooses to format citations.

On the one hand, readers of scientific publications could scan the list of references at the end of the article. If this is the case, journals that order their reference lists based alphabetically by first author surname should show a bias in citation rate, whereas journals that order their reference lists based on order of appearance in the article should not.

On the other hand, readers of scientific publications could use the in-text citations. In-text citations can appear alphabetical (Einstein et al., 1935; Galilei, 1610), or chronologically by date published (Galilei, 1610; Einstein et al., 1935). If readers use in-text citations to guide citation decisions, an alphabetic bias should appear for journals using the alphabetical ordering, but not journals using the chronological ordering.

To test these possibilities, Stevens and Duque scraped data on citation rates from journals in the fields of psychology, biology, ecology, evolutionary biology, and the geosciences, then coded which type of citation format each journal used.

As can be seen in the figure above, alphabetical in-text citations relate most strongly to an alphabetical bias. These data suggest that the primacy effect is likely the result of readers of scientific publications tending to focus on the first names listed for in-text citations, rather than scanning lists of references. If citations are ordered chronologically or by ordinal number, then the bias does not emerge (or to a lesser extent). The bottom panel reveals a stronger alphabetical bias in psychology, which as a field  tends to use alphabetical in-text citations, but not for biology, which tends to use chronological in-text citations or numerical citations.

Stevens and Duque recommend a simple solution to journals to fix this arbitrary source of bias: switch to chronological in-text citations. This solution is easy to implement, and is likely to solve a nuisance factor affecting citation rates. (Full disclosure: I am late-alphabet surnamed author in psychology. With a “W” in my last name, I sat in the back of classrooms, was among the last to walk onto the stage during graduation, and now it appears I am less likely to get cited!)

Research on the influence of arbitrary factors in affecting citation rates (which are central to obtaining grants, and getting jobs and promotions) reveals, for me, the limits of the scientific publication. In a world that is increasingly reliant on technology and the internet, publications are static, fixed in time, and driven by narrative. They afford little-to-no interaction with data, and are not particularly well-suited to searching or integration across different pieces of work.

One step in this direction has just been announced at the forward-thinking online journal eLife, modeled after Jupyter notebooks, which are designed to report data, visualization, and explanation in an interactive side-by-side frame. ELife’s computationally-reproducible articles feature editable code, which readers can then use to perform their own analyses in addition to the analyses reported by the authors.

While, at least in the mind of yours truly, the future of the manuscript is very much in flux, the immediate future is clear: to paraphrase Gil Scott-Heron – the revolution will not be alphabetized.

Reference for the article discussed in this post:
Stevens, J. R., & Duque, J. F. (2018). Order matters: Alphabetizing in-text citations biases citation rates. Psychonomic Bulletin & Review. DOI:10.3758/s13423-018-1532-8.