In the past I’ve been accused of various forms of laziness—not taking advantage of a beautiful day, whiling away my childhood, not studying for my next test, whiling away my adolescence, not writing my dissertation, whiling away my 20s, 30s, etc.—when really I’ve been hard at work watching TV and movies. That the optimal method for this noblest of pastimes involves maximizing contact between the surface of your couch and your horizontal body should not, by any means, detract from the difficulty of translating images on a screen to a coherent narrative in your head. And while I won’t deny having a bit of a chip on my shoulder from these accusations of laziness (I mean, not to brag*, but I do have a Ph.D.), I’ll stand by the argument that watching movies really is harder than you think.

Watching a movie will unsurprisingly involve the visual system. This means that your eyes and your brain have the task of sorting through messy and noisy light waves coming from your television to identify shapes, colors, and contrasts. Then you have to connect these low-level visual details together to form perceptions of people, objects, and environments. From there, you need to connect these individual perceptions to your general knowledge stores about people and how they use different objects in different environments. On top of all that, when you’re watching a movie, you need to maintain this complex web of physical light signals, low-level visual processing, and higher-level processing, while also contextualizing all this information in the details of an unfolding movie plot, where multiple characters have different motivations and know different things and take various actions that can multiple, unexpected, chaotic consequences. See? Watching movies is hard!

In the realm of visual processing research, watching movies has an additional level of nobility, in that studying and identifying the ways that people have trouble with tracking certain kinds of information in movies can grant insights into the nuances of the underlying processes of vision.

The authors of the featured article of this post, Aditya Upadhyayula and John Henderson, have previously published a study demonstrating that movie-watchers are often insensitive to disruptions to the spatiotemporal continuity of a movie. In other words, people tend not to notice if a video skips slightly forward or backward in time. In this earlier work, participants watched video clips that occasionally jumped forward or backward when the participants moved their eyes. Sure enough, the results suggested that participants did not frequently notice any skips forward or backward, though they did tend to notice backward skips more often than forward skips.

In their featured article, “Spatiotemporal jump detection during continuous film viewing: Insights from a flicker paradigm,” published in Attention, Perception, & Psychophysics, Upadhyayula and Henderson extended this line of work to investigate whether this tendency not to notice abrupt spatiotemporal disruptions would hold in a paradigm that was not dependent on eye movements. While their earlier experiment allowed them to study this phenomenon with a technique that deliberately suppresses visual information (since our visual system doesn’t process any incoming information during the short moments when we move our eyes), the fact of the matter is that their stimuli were not distributed randomly to participants. Basing the presentation of spatiotemporal disruptions (forward or backward skips) on participants’ eye movements meant that at least part of their findings may have been driven by factors that cause people to move their eyes, like getting bored and losing focus—which could certainly lead to a tendency not to notice when a video skips ahead or backward.

To avoid these potential confounds, the experiments in the featured article used a different technique called a flicker paradigm. Participants watched one-minute clips without audio from the 2019 film 1917, a movie that was deliberately shot to appear as one long take (in that the camera does not appear to ever cut away from the present scene or angle**). Every two seconds, a white screen appeared over the video for a scant 150 milliseconds, creating the appearance of a “flicker”. Occasionally, immediately after a flicker, the video would skip either forward or backward for a short amount of time, and participants would press a key on their keyboards to indicate that they noticed a disruption. A demonstration of the experiments can be found by following this link.

The authors found that disruptions that skipped over longer windows of time tended to be more noticeable, including in their second experiment, which extended the maximum length of forward and backward skips. The authors also found that backward skips tended to be noticed more often than forward skips, just as they were in their previous study. But overall, participants were pretty dismal at noticing skips, detecting only about half of all skips in both experiments. Thus, the insensitivity to forward and backward skips that the authors observed in their previous work appears to extend beyond eye movements and their underlying motivations, suggesting more broadly that the visual system has a limited capacity for tracking spatiotemporal information during processing.

But why would our visual system have a cap on tracking spatiotemporal information? While the current studies can’t answer this directly, the authors do hypothesize that insensitivity to these kinds of disruptions could be tied to higher-level processing, such that disruptions of upcoming future events (i.e., forward skips) are less noticeable when they largely align with broad expectations about what we’re about to see. On the other hand, backward skips could be more noticeable if they’re more easily recognized from memory.

So while there are yet still questions to be resolved, there are still important lessons to learn from these studies. For one thing, we see that visual processing has deep complexities and subtle nuances in its mechanisms, many of which have yet to be fully explained. And by extension, we can argue even more strongly that lying around watching TV isn’t a waste—our visual systems are hard at work.

* (This was absolutely a brag.)

** Although—minor spoiler alert—technically there is one cut when a character falls unconscious.

Upadhyayula, A., & Henderson, J. M. (2024). Spatiotemporal jump detection during continuous film viewing: Insights from a flicker paradigm. Attention, Perception, & Psychophysics, 86(2). https://doi.org/10.3758/s13414-023-02837-8