Here’s a disconcerting thought: what if the baggage screener at the airport – not that many of us are spending much time in airports these days – got paid a bonus for confidently flagging bags for additional inspection? In an ideal world, they would just assess bags based on the perceptual information they have from the X-ray machine, or, as Locke, Gaffin-Cahn, Hosseinizaveh, Mamassian and Landy (pictured below) put it, in their recent paper in Attention, Perception & Psychophysics, their prior. What might happen in our scenario? Would the screener’s confidence in their judgments change? We would like to think that they wouldn’t, that how confident the screener was about any particular bag needing additional attention was just based on what they saw, rather than their bonus, but is that really the case? Might the reward – the payoff – influence how confident they are, when it has no reason to?

Testing the relationship between priors and payoffs

To test this question, Locke and colleagues performed an experiment where observers viewed a tilted Gabor (that is, a sinusoidal grating in a Gaussian aperture; see image below) and had to report whether the Gabor was tilted to the right and how confident they were in their perception.

Critically, this experiment manipulated both the prior probability, that is, how often the Gabor was, in fact, tilted rightward, and the payoff, whether observers would be rewarded more relative to the state of the prior. Prior to running in the main experiment, each observer went through a thresholding session to find their orientation threshold – that is, to ensure that the Gabor orientation task in the main experiment was sufficiently difficult that changing the prior and the payoff might change their percept and how confident they were in it.

What manipulates confidence judgments?

In an ideal world, we would expect only the prior, that is, the stimulus itself, to have an impact on how confident someone is in their percept of it, since that’s, fundamentally, what a confidence judgment is asking. One possibility here is that, yes, observers would only be influenced by what they saw, rather than the reward. However, that is not the only option, and Locke and colleagues suggest two alternatives that might exist if an observer’s confidence judgments are impacted by the rewards they receive.

If the reward impacts how confident an observer is in their judgments, their confidence ratings might shift based on the expected gain – to go back to our theoretical baggage screener, they might be more confident in their judgment when they’re paid a bonus for confidently flagging bags. If, on the other hand, confidence is basically fixed around the decision boundary – in the experiment, judging whether the Gabor was tilted rightward – neither the prior nor the payoff would have much effect. Confidence ratings from such a baggage screener, for example, might not mean much, because the confidence level would be fundamentally sticky, and somewhat decoupled from either the bags they were seeing or any bonus they might be paid.

Observers aren’t optimal

Surprisingly, only one observer in the experiment behaved optimally, that is, showed confidence judgments that were only influenced by what they had seen, rather than the reward. A plurality of observers’ confidence judgments was influenced by the potential gains, with the remaining observers keeping their confidence judgments relatively fixed around their decision boundary. So, some observers were essentially oblivious to the prior/payoff context – the ones who stayed relatively neutral when reporting how confident they were – but others were influenced by reward.

As the authors put it, these gains-shift observers assume that “this highly rewarding perceptual alternative that I have selected is certainly the state of the world.” If observers in the experimental task make this error when determining how confident they are when judging the tilt of Gabors, what might this mean outside the lab? There are some pretty disturbing implications of this when we look at the world – our baggage screener is just one of them! Given how this effect might intersect with varying probabilities of events in the world, like looking for abnormalities in radiological images, or determining whether the chicken at the side of the road will run in front of your car, it’s very much worth being aware of these errors in judgment, and how they might distort our confidence in perceiving the world.

Why should we worry?

Let’s go back to our very bored baggage screener who we left at the beginning of this blogpost (who hasn’t been seeing too many bags lately, owing to the pandemic) – what impacts their confidence in recommending bags for extra attention? Based on these findings, it’s very unlikely that they are only relying on what they’ve seen. In all likelihood, they will be using one of two strategies. Either their confidence ratings might be kind of sticky around their decision boundary, or they might be changing their confidence in their judgments based on the reward they receive. If they are changing their behavior based on how they’re rewarded, which this work suggests they might, we might want to be cautious about interpreting confidence ratings under these conditions. So, in the future, if you are rushing to catch your flight, better hope that your baggage screener is not being rewarded for flagging bags for screening, because they might be more confident in their recommendations than they would be otherwise, and you might be stuck at security rather than making it to your plane.

Psychonomic Society’s article focused on in this post:

Locke, S. M., Gaffin-Cahn, E., Hosseinizaveh, N., Mamassian, P., & Landy, M. (2020). Priors and payoffs in confidence judgments. Attention, Perception & Psychophysics, 82, 3158–3175. https://doi.org/10.3758/s13414-020-02018-x