In this episode of All Things Cognition, I interviewed Anna Kosovicheva and Ben Wolfe (pictured below) about their research described in an article recently published in the Psychonomic Society journal, Psychonomic Bulletin & Review. They co-authored the paper with the first Editor in Chief of another of the Society’s journals, Cognitive Research: Principles and Implications, Jeremy Wolfe. Wolfe will be the keynote speaker at this year’s Annual Meeting.

Transcription

Intro

Rivers: You’re listening to All Things Cognition, a Psychonomic Society podcast. Now here is your host, Laura Mickes.

Intro to the interview

In the upcoming All Things Cognition interview, I speak with Anna Kosovicheva and Ben Wolfe about their paper published with Jeremy Wolfe, in the Psychonomic Society journal, Psychonomic Bulletin & Review.

Interview

Mickes: Hi Anna and Ben. I know you both because you are part of the Psychonomic Society’s digital team and it’s nice to be talking to you in a different capacity. So thank you for talking to me about your recent publication in Psychonomic Bulletin & Review. So you have a paper that you recently published and you’re gonna be talking about that with me. Um, thank you so much.

Kosovicheva: We’re happy to be here and excited to talk about it.

Mickes: Thanks, Anna.

Wolfe: Looking forward to it.

Mickes: Thanks, Ben. I love the paper that gets straight to the matter at hand and your title does it. So the title of the paper is “Taking Prevalence Effects on the Road Rare Hazards are Often Missed.” Is it a spoiler alert? I would say partially, but I’ll tell the listeners to keep listening cause there’s more to it than that. You also have another co-author on the paper. Who’s that?

Wolfe: Our other co-author is Jeremy Wolfe [pictured below], both because he’s done a whole lot of work in low prevalence effects and search and I may have gone into the family business. Um, he’s my father, so we thought it would be a little silly to work in an area that he’s an acknowledged expert in and not bring him into the project.

Mickes: Ah, so it was your idea and you said, “Hey Jeremy, do you wanna be involved in this?” And he said, “Heck yeah.”

Wolfe: Yeah, that and we have this list of projects, you know, that we want to do and you know, we’ll get to in our copious free time. We had just moved to the Toronto area at the beginning of 2021. Everything’s in Covid lockdown. We’re like, what can we do that’s actually science when we’re sitting at home with the cat [pictured below] in a new country where we can’t go anywhere? And we realized that, oh this, this would actually work. We could do this online. And so emailed Jeremy and you know, say, Hey can we pick your head? And that’s how this all got started.

Kosovicheva: In some ways I think it’s kind of like a perfect intersection of our interests, being in more applied topics and perception as well as Jeremy’s interests because he’s done a lot of research looking at how it is that people tend to miss items that are rare.

Mickes: Jeremy has a reputation come out with the creative titles for his papers. I was gonna ask you who wrote this one?

Wolfe: I don’t even remember which one of us came up with this title.

Kosovicheva: It might have been me I think.

Wolfe: Yeah, this, this has the feeling of an Anna title. Um, yeah, yeah, if, if I’d come up with it, it would involve moose.

Kosovicheva: I gave a talk on this at Vision Sciences Society and that did have moose in the title.

Mickes: Moose?

Kosovicheva: Yeah, “The moose came out of nowhere: Low prevalence effects in road hazard detection.” So the backstory for this is that when we started this project, we’ve been doing a series of driving-related studies in the lab. And the way that we can research say something like hazard detection safely in the lab is to go online and find people’s dashcam videos. People post these all the time: “Look at this jerk. They cut me off. How rude.” Or some other thing that might happen. “Oh my gosh, this cyclist came out of nowhere. How can they be so unsafe?” And so what we do actually in the lab is we kind of gather these videos and so one of the earliest videos we found involved a moose running onto the road. [Example of stimuli video of a car and moose accident.]

[videopress us9CTTy9]

Kosovicheva: So running theme across a number of our studies is that they somehow involve moose. So the, there’s like the, the kind of the “original moose”, but now we’ve kind of branched out into “prevalence moose”. We’re also looking at the effects of blur on hazard detection. So that’s “blurry moose”. So a lot of them kind of have the word moose in them in the lab lingo for how we refer to these studies.

Mickes: You tested how prevalence rates – or how often something happens, prevalence rates of road hazards impact our responses to them. Why did you do that?

Kosovicheva: Basically previous work has looked at how it is that people tend to miss rare items. So this is something that’s been commonly looked at in tasks like baggage screening. So if you’re working at the airport and you’re searching through luggage for guns, bombs and knives, it’s something that is fairly difficult to do. But it’s also really severely impacted by the prevalence of the items that you’re looking for. So if you’re looking for something that’s fairly rare, like some kind of explosive in luggage, people will tend to miss those rare items.

The short tagline for this is “If you don’t find it often, you often don’t find it.” This is also something that is seen a lot in radiology too. So if you’re a radiologist, you’re looking at a mammogram for example, tumors might be fairly rare and so if you don’t see them very often, you might frequently miss the rare tumors that are there.

What we wanted to see in this study was whether something similar can happen in more dynamic situations that are more time sensitive and that are kind of representative of real world situations or moments of driving.

Mickes: What did the participants do?

Wolfe: This paper is a series of five experiments. We ran all of these online. We were doing this all during pandemic lockdowns here in Ontario. And the basic paradigm went like this.

If you’re a participant in this study, you’re being shown very brief clips of dashcam video. So dashcam video is video looking out from inside a car and on each trial participants are being shown a 333 millisecond clip. And that sounds like a really, really brief video duration. And that’s actually about one and a half times the minimum duration threshold you need to do hazard detection. We’ve looked at that before if we’re talking about young drivers. So people between about 20 and their mid thirties, they need about 200 milliseconds of footage to be able to say hazard or no hazard. And so setting our durations to a third of a second meant that we were well above that you should be able to reliably say, yeah, there was something here. [Diagram of a trial is below (a) and examples of hazards shown.]

Mickes: So it’s not the time that that would cause them to miss that it was too short so that they missed a hazard. Okay.

Wolfe: Yeah. So we, we know that they’re seeing enough video to be able to do this task. What we just asked them to do is report “is there a hazard in this video?” So to do this in a low prevalence case, they’re seeing a very small number of hazards and they’re doing hundreds of trials in a session. These trials do move very quickly with the videos only being a third of a second long.

Generally, all of these experiments were two sessions. They do about four to 500 trials per session, which goes fairly fast. Although it’s probably a little, a little scary if you’re, you know, seeing variously prevalent near-collisions. We would vary the prevalence level. So we would generally use 50% as a baseline just because that’s what we’ve used in previous studies. Doesn’t look anything like the world, but it’s a good baseline level. And we would vary the prevalence level. We had some experiments where prevalence was at 10%, at 4%. We even went down as low as what, 1%?

Kosovicheva: [Nodded]

Wolfe: We varied this between experiments, but the basic paradigm was always you saw a third of a second of video and had to respond. Was there a hazard or not? A very simple perceptual task.

Mickes: So it’s just a yes or a no, there was a hazard? And are you recording the response times?

Kosovicheva: The reaction time data are decently reliable. We did look at reaction time data; it didn’t actually make it into the paper. Basically, we see there’s something that’s fairly predictable in that we see some speed-accuracy trade-offs. In the low prevalence condition, when hazards are rare, people tend to be pretty quick at saying no hazard, right? They kind of fall into this pattern of there’s no hazard, no hazard, no hazard, no hazard.

Mickes: Oh.

Kosovicheva: And so those RTs [reaction times] are fairly low. So for situations where people tend to miss the low prevalence hazards, then reaction times are actually fairly short. And that’s something that you see in visual search data as well.

Mickes: Right.

Wolfe: They kind of tend to fall into this pattern. It doesn’t seem to depend on people’s ability to correct the responses. It’s not just kind of a finger error type of thing. So we did an additional experiment where if we give people a chance to correct their responses, they still miss the rare events.

Mickes: So, and that was one of the experiments that you did. Okay.

Wolfe: That’s experiment 4.

Mickes: Okay. And in all five experiments, you found the same thing. I shouldn’t say what what you found. So will you tell me what you found across the five experiments?

Kosovicheva: Generally what we find is that people tend to miss the rare road hazards that they see. So compared to the condition, the 50% baseline condition where they see a mix of videos. So 440 videos, half of them have a hazard, half of them don’t. People tend to miss, I think it’s something like 18% of hazards, which is pretty high, but it’s a fairly demanding task and some of them can be kind of subtle, but basically, the proportion of hazards that they miss basically doubles in the low prevalence condition. So people miss the hazards they see almost twice as frequently when they are rare. So people tend to miss the rare road hazards.

Mickes: And this happened in all five experiments? [Results for all five experiments are shown below.]

Kosovicheva: What’s interesting is that it actually, there’s one case in which the, we don’t see this effect and that is if we remove any feedback. What we did was, in a follow-up experiment, so normally what we do is we give people feedback, we tell them correct or incorrect after each response. When we get rid of that feedback, that effect tends to go away.

When we get rid of the feedback that we provide to participants, what happens is that they miss the high-prevalence events and the low-prevalence events at a similar rate. Basically what it suggests is that people are somehow using this feedback to get a sense of how often these events occur. For example, you might decide, you know, if you have no feedback you might say, “Oh well I keep saying no hazard all the time. Maybe I should say hazard more often. Maybe this, this constitutes a hazard” or something like that. So it does seem to be feedback-dependent, which is pretty interesting. Eliminating the feedback that participants receive seems to eliminate this effect.

Mickes: Entirely?

Kosovicheva: In some ways.

Mickes: Wow.

Kosovicheva: Largely, so in the no-feedback condition, that’s the only case where we don’t observe the low prevalence effect. We’re currently doing some follow-ups to look into this a little bit more.

Mickes: Yeah.

Kosovicheva: But what’s interesting is that if you consider real-world driving, what you have is feedback all the time, right? You’re continually getting a continuous stream of feedback if you’re driving right, If you hit the brake unnecessarily, you kind of get a sense of like, oh, I didn’t need to hit the brake. Or if you collide with something that’s obvious, very dramatic feedback. But we get feedback in the real world all the time. So the fact that this effect kind of tends to go away if we eliminate feedback, doesn’t really tell us how we can potentially eliminate this effect in the real world.

Mickes: Right? Because there’s no way we can do away with that feedback. Oh, that’s really interesting. Is it true that the accuracy is good? d’ is the same, whether it’s low or high or 50-50 base rate? It’s the same but it’s just they shift around their criterion depending on whether it’s a, a low prevalence, they become very concerned.

Wolfe: That’s exactly what happens.

Mickes: Yeah, you can move them around. Would you want more false alarms in a road situation? Some people are more alerted or something they think it’s a road hazard but it’s not, but they’re on higher alert. Is that what you would prefer to miss and then crash?

Wolfe: Yeah, so that’s one of the interesting things about this is that we found this effect. How do you fix it?

We have one attempt on that in experiment five, which is hey, let’s just tell people that this is a bug in cognition. Let’s straight up tell you, if you don’t see it often, you often don’t see it.

Turns out to do nothing. It has absolutely no effect on the low prevalence effect, which is both interesting, because it suggests that it’s cognitively impenetrable, and disconcerting because it suggests that we can’t just record a bunch of public service announcements and, you know, plaster them on the billboards and be like, you should be careful. That’s not gonna help us. One of the things we’re doing in the lab now is trying to think about how else can you break the low prevalence effect in this context. And so one thing we’re thinking about doing is can we build training regimes where we actually train people to better recognize hazards under certain circumstances?

Wolfe: And there’s been some work in hazard perception in driving that suggests maybe this would work. No one’s looked at it in a low prevalence context. So we are thinking about this. But it turns out to be a, a very difficult and sticky problem in actually in exactly the same ways that the low prevalence effect in visual search is.

Wolfe: Which is interesting because visual search tasks, whether it’s bag screening or radiological screening or you’re looking for Ts among Ls. All of those tend to be look at the stimulus until you choose to respond. So unlimited duration, no, no functional time pressure, whereas we’re showing people a third of a second of video and saying respond as quickly as you can, trying to approximate some of the actual time pressure that we would experience in the world.

Mickes: Right.

Wolfe: Because if you’re on the road, you don’t get 2, 3, 4 seconds to ponder deeply and think, is that something I need to do something about? No. If you do that, the moose is sitting in your lap and you’ve had a bad day.

Mickes: Laughs. Right. How then can you possibly, if you say it’s cognitively impenetrable, how can you possibly break through that? Any pupilometry measurements or any eye-tracking that you’re doing? Or is that a ridiculous suggestion?

Wolfe: I will, I will say that the eye-tracking side of things is something we’d love to do. Um, it’s, we actually the stimulus set that we’ve built to do these experiments, we’ve actually annotated at this point pretty much all of it for where the hazards are in frame. If this sounds like the sort of truly mind numbing task you give undergraduates who are new in the lab, you might be right. But one of the things we’re planning on doing is actually asking, okay, if you are sitting in the lab looking at these road videos, where do you look when you are trying to detect hazards? It turns out that there’s a lot of, again, much more applied literature where lots of weird assumptions get made about this. You know, maybe the, the more common assumption is, you know, you have to fixate the hazard in order to see it and well, where’s that plaque? [Found it!]

Wolfe: “If vision worked like that, we’d all be dead.”

Mickes: Laughs.

Wolfe: But seriously, it’s one of the things that’s on our list of projects to do now that we’re, we’re back to some in-person research here.

The pupillometry question is, is actually a good one. It’s, it’s a thought that a lot of people in engineering and human factors have because they love the idea of a covert signal that you can just record. I am, myself, very skeptical of whether it would ever work, particularly because good pupillometry in the lab when you control say all of the lighting is hard enough. Yeah. And then imagine doing this in the car when you have to compete with the terrible day star in the sky.

Wolfe: I think, I think you would have a very hard time getting reliable pupillometry data.

Wolfe: So that’s just impractical right now. Yeah. The larger question actually of we’ve got better and better technology, what can we shove it in the car and try to make people safer? The question of how do we figure out what the driver knows about their environment is a really big question right now. There’s a lot of attempts to do it, you know, with eye tracking measures. Because of basically technological limitations, driving research has tended to use some pretty course eye tracking measures. They like to do, you know, big areas of interest on areas of like the windshield or the rear view mirror or your radio and, you know, that’s, that’s okay for some things, but if you want to know what the driver has actually seen and might, you know, have some representation of in their mind, that’s not gonna give you enough.

Mickes: No, that’s not gonna get you enough.

Wolfe: Part of the work I do is actually trying to push the driving, the driving research and traffic safety world, towards a more granular, more cognitive science understanding of eye tracking as a method.

Mickes: Oh,

Kosovicheva: I was just gonna circle back to – how do you cure the low prevalence effect, right?

Mickes: Yes.

Kosovicheva: I think that’s kind of, in some ways the million dollar question that everybody’s after right now is: how do you fix this in search? How do you fix this in, you know, these task like baggage screening and radiology, How do you fix this effect on the road? And there seems to be no straightforward solution, in my mind. I think one thing that people always ask though is what about the, the costs and payoffs of the responses that you make? So say, you know, in our study if participants miss a hazard, well, they just get a screen that says incorrect, right?

Mickes: Yes.

Kosovicheva: But in the real world, you know, if you miss the moose, then that’s a really bad day, right?

Mickes: Yeah.

Kosovicheva: So it’s something that is of course really impractical to kind of simulate in the lab. We could potentially, you know, try to pay people more, for example, for good performance in the task or something like that. Um, but in a lot of other studies, people have found no to minimal effects of these sorts of things. So people have tried all kinds of different approaches for manipulating these effects.

Mickes: Oh.

Kosovicheva: Yeah. And I think it’s, it’s possible that with when the costs and payoffs are sufficiently extreme, like they are in the real world, such as driving, that it’s possible these things could, could make a difference, but it’s impossible to know because it’s not something that we can kind of safely implement or at least things that are quite that extreme. Right?

Mickes: Right.

Kosovicheva: Um, so yeah, it’s something that we’re currently looking into.

Wolfe: Along those lines, like when people have looked at actual naturalistic data, for some of the real-world search tasks. So if you actually look at baggage screening, if you actually look at radiologists reading real medical images in situ, they show this effect. So even if you are a radiologist and your job is to look at chest CT or mammograms or choose, you know, choose your, a bit of radiological imaging and you know that there are consequences if you fail wherein you will fail to catch a diagnosis, someone will not get treated as quickly as they should with, you know, consequences for their own long term health. People will show this, this is not probably something where throw more money at the problem or, you know, scare, scare the pants off of your observers will actually change that.

Mickes: I see. So they’ve already got the highest cost, they’re already dealing with someone’s life.

Wolfe: Right. They’re already dealing with someone’s life and, and not just the patient’s life also, you know, you will get sued for malpractice,

Mickes: Right? Yeah. And your job is in jeopardy.

Wolfe: Higher risk than that. What, what, what higher stakes do you want?

Mickes: That’s right. No, a few extra bucks isn’t gonna do it, is it?

Wolfe: Nope. If we can find an answer, we can actually, you know, we can have a really big impact.

Mickes: Yeah, that’d be incredible.

Wolfe: The interesting place this lives in my mind is there’s this big push right now of let’s try to make roads safer for everybody. You know, not just, let’s not just focus on drivers. Drivers aren’t the only people on the road. You know, there are cyclists, there are motorcycle riders, there are pedestrians, everyone uses the road and areas around the road. Yeah. But if you just focus on drivers, that’s somewhere between silly and, you know, verging on evil. Um, so the question is how do you keep everybody safe? And so there’s a lot of work going on of, Hey, let’s pack technology into the car. Let’s, you know, give you advanced emergency breaking, let’s give you pedestrian detection. Let’s build all of these engineering solutions. Let’s redesign roadways to be safer for cyclists and pedestrians and drivers. And, you know, all of the, all of both the autonomy work in the car, all of the Vision Zero work on the roadway.

And that’s all great because it, it will keep people safer as you catch things from the car, as you keep vulnerable road users, which is the umbrella term for pedestrians and cyclists, away from cars. But what our results probably suggest is, all right, what you’re doing is you’re decreasing the prevalence of dangerous events.

Mickes: Oh. So it’ll be even harder.

Wolfe: There’s this nasty, there’s this nasty little rump that you probably can’t ever stop,

Mickes: Right.

Wolfe: At least not with infrastructure and technology. So essentially there’s a need to fill this cognitive gap. I tend to think of this as part of the larger space of how do we keep everybody safe.

Mickes: That’ll keep you busy. Yep. All right. I think we’ve covered everything. You have me thinking about so many different things. I’m sure the listeners will be too. Is there anything that I may have missed asking you or you don’t think you covered that you wanna be sure that we say here?

Kosovicheva: I don’t think so.

Mickes: No. Okay. All I can do is thank you. Thank you so much for describing that research and talking about the problems that we face.

Kosovicheva: Thank you for having us.

Wolfe: Yeah. Thanks for chatting.

Concluding statement

Rivers: Thank you for listening to All Things Cognition, a Psychonomic Society podcast. If you liked this episode, don’t forget to follow the channel using your favorite podcast player or app. All members of the Psychonomic Society receive free access to our seven journals and are invited to attend our annual conference at no charge. Learn more and become a member by visiting us online at http://www.psychonomic.org.

Featured Psychonomic Society article

Kosovicheva, A., Wolfe, J. M. & Wolfe, B. (2022). Taking prevalence effects on the road: Rare hazards are often missed. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-022-02159-0