You push the elevator button and you start waiting on the 4th Floor, in a rush to grab a sandwich before your lunch time is over. You wait. And you wait some more. At what point do you stop waiting and take the stairs?
We make countless such decisions every day, where we need to predict an outcome on the basis of partial experience. Whenever you are put on hold when you call the bank, you constantly build up additional partial experience—namely the time spent waiting—and you predict how much longer you will need to listen to the obligatory “your call is important to us message” before you hang up.
How do people make such judgments?
A few years ago, Tom Griffiths and Josh Tenenbaum published a provocative paper in which they showed that people are quite adept at making such judgments, even for rather esoteric quantities such as the duration of the reign of Pharaohs. When given a question such as “If a Pharaoh has ruled 20 years, what will be the total duration of his reign?”, people’s responses were found to capture the actual historical distribution of the reigns of Pharaohs surprisingly well. In a follow-up paper, on which I happened to be the lead author, I showed together with Tom Griffith and Mike Kalish that the knowledge people use in those prediction tasks is quite sophisticated and is used seemingly “optimally.” (Specifically, people seemed to sample from a Bayesian posterior.)But what is the exact nature of that knowledge? To what extent does it rely on specific expertise? Are all possible quantities equally predictable?
A recent paper published in the Psychonomic Society’s journal Memory & Cognition sheds more light on this issue. Researcher Rachel Stephens and collaborators employed the prediction task in conjunction with a so-called “iterated learning” procedure. In an iterated-learning procedure a participant’s response on one trial is used to generate a stimulus on a later trial in the same sequence. That is, unbeknownst to the participant, the stimuli they encounter late in the sequence are a consequence of their own earlier responses.
This procedure is known to reveal people’s prior knowledge in the long run—that is, if your own responses are used to generate further stimuli, then under some reasonable assumptions, your responses will ultimately converge on revealing your prior biases or prior knowledge.
In order to reveal more about the nature of that prior knowledge, Stephens and colleagues compared the performance of two groups of subjects, drawn from samples in Australia and China. All participants were confronted with tasks that were either unfamiliar to both groups (reign of Pharaohs); familiar to the Australians only (an analogous question involving cake baking times); familiar to the Chinese participants only (the length of Beijing bus routes).
How would familiarity with the subject matter affect people’s responding? In case you are wondering, baking cakes apparently is uncommon in China, and for your edification the distribution of the route lengths of the 822 bus routes in Beijing is shown in the figure below:
The data for the Australians are shown in the figure below, which plots the actual quantiles of the distribution of cake baking times and the number of bus stops (on the Y axis) against the corresponding quantiles provided by participants. To understand the data you need not worry about the details of quantiles: What is important to know is that if participants were accurate, then all data points would fall along the diagonal.
It is clear that the Australians were pretty good with cakes but didn’t quite capture the distribution of bus routes in Beijing (they over-estimated their lengths considerably).
Now consider the Chinese participants:
As expected, the Chinese participants clearly did better with the buses than the cakes, quite the reverse of the Australians, but intriguingly, their predictions for the buses were also not well calibrated—participants tended to under-estimate the route length of a bus system they were familiar with. Both groups of participants were, however, well calibrated on the Pharaoh task, just as had been observed in previous research.
In two follow-up experiments, Stephens and colleagues showed that this under-estimation of bus routes and bus journeys was also observed with Australian participants when they had to make judgments of public transport in their own city. That is, familiarity with a public transport system was systematically associated with under-estimation, whereas lack of familiarity led to over-estimation (viz. when the Aussies made predictions about buses in Beijing).
Taken together, at first glance the data throw up a paradox: Why are people—irrespective of their culture—well calibrated for esoteric quantities such as the reign of Pharaohs, whereas they under-estimate quantities that they are highly familiar with?
The authors suggest that the difference results from the direct retrievability of numerical answers, which is possible in some domains but not others. That is, even seemingly esoteric quantities such as the reigns of Pharaohs permit direct retrieval, based on knowledge of factors such as human life expectancies and estimates of the reign of contemporary monarchs. For familiar but less tractable quantities such as the number of bus stops on a local bus route, people find it difficult to retrieve numerical answers—they must instead resort to more complex processes such as “ordinal conversion” (i.e., establish a plausible response range first, and then attempt to find a suitable response to the question from within that range). One implication of engaging those more complex processes is that they are subject to all the known biases in human decision making, such as anchoring and adjustment.
The bottom line is that people are good at estimating quantities based on partial information if those quantities have been experienced as retrievable numerical quantities—we all know what it is like to wait 5 minutes for an elevator. But we typically don’t count the number of bus stops or freeway exits on our way home, and hence we do not predict those quantities with equal aplomb.