The big data special issue from the Psychonomic Society’s Behavior Research Methods is particularly timely. Big data is becoming increasingly prevalent in behavioural sciences and it is arguably transforming many areas of research. However, this change is not one that was planned or designed by scientists. Advances in technology and digital records mean that governments, employers, retailers, navigation services, and countless other organisations now often collect detailed data on the behaviour of potentially millions of people. Almost all of these datasets have been collected and stored for a particular purpose or aim that is independent of scientific investigation. This could be tracking the tax requirements of citizens, the performance of employees, online purchases of customers, or the location of users when providing navigation information.
The proliferation of these digital services, means that as modern citizens move throughout the world, their actions leave a detailed trail of digital breadcrumbs. These can reveal where those citizens have been, what they have said, and the decisions they have made. None of this was collected by behavioural scientists, or for the purposes of scientific experiments. No one was experimented on, or manipulated. None of these individuals set foot in a laboratory.
However, the structure of our digital world means that there are now petabytes, if not exabytes, of data giving clues to individuals’ actions: Digital breadcrumbs that behavioural scientists can follow to learn more about human behaviour. And, whilst the data of government or personal services is quite rightly private and secured, many sources are publically available. For example, reviews left by customers, the literary works that authors publish, or comments made on public forums.
This explosion in available data has occurred at a time when behavioural science’s traditional laboratory based approaches have faced considerable criticism. There have been several prominent examples where large, well-funded projects have failed to replicate famous behavioural results (see the Open Science Collaboration). In many cases, the most likely explanation is that the apparent patterns found in early experiments were the result of random chance: Perhaps a higher than average number of introverted people signed up to take part in a study. This problem is greatest in smaller studies, with these random variations effectively washing out in larger studies with many participants.
Other critics cite the lack of realistic context in many laboratory studies. Recent work has investigated some of the most robust and widely replicated lab results showing how one can manipulate choices using simple changes in context (Frederick, S., Lee, L., & Baskin, E., 2014). Results show that famous findings in the lab completely disappear when the choices are presented as realistic pictures and descriptions as opposed to more easily analysed schematic numerical information.
A number of papers in this special issue demonstrate how big data can address these issues. The presence of the word “big” in big data is of course the clue that these studies are much less susceptible to the random variations of small samples. Typically, the largest laboratory studies may recruit up to a few hundred people. By contrast, the article by Thorstad and Wolff follows the contributions and linguistic patterns of more than 120,000 individuals on public forums on reddit.com. More specifically, these are individuals reporting a form of mental illness: a group of people often difficult to engage in traditional research, even in small numbers.
Not only can big data projects include data from more individuals, but they can acquire data over a much longer period of time. Laboratory studies rarely last more than a few hours, or a few sessions. In Thorstad and Wolff article, individuals’ contributions were tracked for up to 5 years. For this study, this also has the additional benefit of providing measures of individuals’ behaviour before a particular mental-health issue is identified. This is something that would require enormous resources to approximate with traditional research approaches.
Of course, these are not the only advantages to using real world databases. Another aspect is that the complexities and distractions of everyday life are included by default. For example, when subjects participate in a laboratory study, the experimental task is typically all they have to fill their thoughts. Memories of previous stimuli are fresh, and experiments typically ask subjects to think of unusual topics that have little bearing upon the individual’s wider life. When rating or judging one item, a participant’s foremost thought will be the previous item that they rated only seconds ago. To counter this, complex distractor tasks have been developed to approximate the cognitive loads experienced in the real world. But by taking advantage of big data, no such approximation is required.
By taking advantage of employee training records, the article by Kim and colleagues replicated lab-based research into the design of training schedules. The upshot is that longer intervals between study/training sessions lead to better performance when the information is required/tested less frequently, whereas shorter intervals lead to better performance when the information is required/tested more frequently.
Finally, the article by Vinson, Dale and Jones demonstrates a common lab phenomena in product ratings. The researchers used public reviews on Yelp and Amazon to show the sequential effect of contrast. This is the pattern by which reviewers contrast their rating of the current product/restaurant/movie, to that of the one they rated most recently. Put simply, if the last restaurant reviewed was terrible, then a mediocre one will contrast positively, and receive a higher than average rating. If the last restaurant was especially good, then a mediocre one will contrast poorly and receive a lower than average rating.
Both of these findings—by Kim and colleagues and by Vinson and colleagues—are remarkable in that they take findings from abstract laboratory experiments that were typically completed across short timeframes, and then find that the effect is present in the real world. This generalization arise despite the fact that “real life” is a cognitive environment cluttered with all of life’s ongoing stresses, interruptions and complexities. And because there is no experiment, no one is trying to second-guess an ever-watching experimenter.
This work suggests a bright future for big data approaches to validating the results of laboratory experiments.
What is also noteworthy, is that is suggests great promise from a reverse approach: in addition to providing validation for laboratory work, big data can begin to question it. An important and seemingly inevitable next step is for work to show not just positive real-world replications, but to use these same data sources to show where lab results may not replicate. The analysis techniques, and impressive methods developed in this special issue will be invaluable in supporting this work.
Psychonomics articles highlighted in this post:
Kim, A. S. N., Wong-Kee-You, A. M. B., Wiseheart, M., & Rosenbaum, R. S. (2019). The spacing effect stands up to big data. Behavior Research Methods, DOI: 10.3758/s13428-018-1184-7.
Thorstad, R., & Wolff, P. (2019). Predicting future mental illness from social media: A big-data approach. Behavior Research Methods, DOI: 10.3758/s13428-019-01235-z.
Vinson, D. W., Dale, R., & Jones, M. N. (2019). Decision contamination in the wild: Sequential dependencies in online review ratings. Behavior Research Methods, DOI: 10.3758/s13428-018-1175-8.