This post was co-authored by Manish Singh of Rutgers University and Chetan Prakash of
California State University, San Bernardino.

A new theory in science, like the new offspring of an altricial species, needs an initial phase of development under the care of its progenitors. With the wane of this phase, its fitness must be tested in the rough and tumble of real life. Care and clash are both required for healthy maturation.

We are grateful for the incisive critiques of our interface theory of perception (ITP) offered by John Hummel, J. Scott Jordan, and Gary Lupyan. This is precisely the testing needed to hone the theory or find it unfit.

In what follows we respond to the critiques in alphabetical order.

John Hummel

Comment 1. Hummel states that “their argument, taken seriously, degenerates into solipsism—a problem that rears its head early and which raises its head higher and higher as the paper goes on.”

Reply 1. Metaphysical solipsism is the claim that nothing exists other than one’s own mind and its mental states. Epistemological solipsism is the claim that nothing can be known other than one’s own mind and its mental states. Hummel is probably referring to epistemological solipsism in his comment. But just to be complete, we discuss metaphysical solipsism in this reply, and epistemological solipsism in the next.

That ITP does not amount to metaphysical solipsism should be clear from the fact that its formal framework includes a set W that is distinct from the set X. In other words, ITP specifically postulates a “world” that is distinct from one’s perceptions. What ITP denies is any simplistic form of relation between W and X; e.g., the assumption that the predicates of our perceptions—species-specific as they are—somehow provide insight into the “true” predicates of the world. And it places strong limitations on what can be inferred about the structure of the world from the structure of our perceptions. Importantly, however, ITP does not preclude normal science, and makes falsifiable predictions of its own. We delve into these issues more deeply in our response to Comment 2.

Comment 2. Hummel argues that ITP “… amounts to the idea that we have no idea of what’s actually out in the world. (Lest you think I’m exaggerating, re- read their symmetry proof.) … The problem with this thesis is that, if it is true, then we can never know it (because its very truth implies that reality is beyond our grasp); neither can we know whether it’s false.”

Reply 2. ITP claims not just that some of our perceptions might be non-veridical, but rather that the very predicates of our perceptions—space-time, physical objects in space-time, and properties of objects such as shape, position, movement and color—are the wrong predicates to describe reality as it is. These are simply species-specific predicates that natural selection has produced. Veridical descriptions of reality simply cannot be formulated using the predicates of our perceptions. Hummel is indeed not exaggerating this aspect of ITP.

But, contrary to Hummel’s pessimism, this claim of ITP does not preclude the normal scientific process of building falsifiable theories about the nature of reality, and devising empirical tests of those theories. Indeed, as it stands, and without further augmentation, ITP makes four falsifiable predictions about the nature of reality.

1. Space-time is doomed. ITP predicts that space-time is not fundamental in physical theory, but instead emerges as a description from some more fundamental reality. Physicists will find that space-time descriptions are unnecessarily complex, and fail to reveal the deepest symmetries that govern reality. If this prediction proves false, ITP is wrong. Recent work on modeling scattering amplitudes using the amplituhedron by Arkani-Hamed and colleagues supports this prediction of ITP, as does earlier work in string theory by Giddings.
2. Physical objects in space-time are doomed. ITP predicts that no physical object has definite values of any dynamical physical properties—such as position, momentum, spin—when it is not observed. If any experiment demonstrates that, say, an electron has a definite value of spin or position when it is not observed, then ITP is wrong. Such tests are empirically possible, and have been repeated many times, so far with no result that contradicts the predictions of ITP. The latest test used entangled electron spins separated by 1.3 kilometers.
3. Causality of physical objects in space-time is doomed. If physical objects do not have positions, momenta, or spins when they are not observed, then the causal nexus of reality—if such there be—is not due to positions, momenta, or spin. If experiments demonstrate otherwise, ITP is wrong.
4. Causality of neurons is doomed. This follows as a special case of c. It entails the prediction that neurons cause none of our behaviors and none of our conscious experiences. If experiments demonstrate otherwise, ITP is wrong.

We hasten to add, however, that research in brain imaging and neuroscience is essential. We must take our data where we can. ITP simply entails that the standard causal interpretation of those data is too simplistic. Neurons are icons in the interface of H. sapiens, not an insight into the ultimate causal sources of our behavior and experiences.

These four predictions show that ITP makes nontrivial experimental predictions that can be tested, and thus is a falsifiable theory. Hummel also worries about ITP that “… if it is true, then we can never know it …” But the same can be said of every scientific theory. No scientific theory can ever be proven to be true. No matter how much data one has in favor of a particular scientific theory, it is straightforward to show that the data cannot logically entail the truth of that theory. Instead, there are, in general, countless different theories that are compatible with all the known data, but that make conflicting predictions about the outcomes of new experiments that have not yet been performed. This “underdetermination of theory by data” is a central topic in the philosophy of science. It is also what makes scientific theory building a creative and engaging process, rather than simply a mechanical deductive consequence of data.

Comment 3. Hummel argues that the versions of veridical perception rejected by ITP are straw men, not taken seriously by current perceptual researchers.

Reply 3. The evolutionary simulations we report were based on an exhaustive and mathematically precise classification of all possible definitions of “veridical perception”. None of them can compete successfully against interface perceptions that are sufficiently tuned to fitness. Thus we did not pick some particularly easy straw men and test them; we tested the whole gamut of possibilities.

A strong notion of veridicality runs deep in current perceptual theories. You can see this in the standard definition of “illusion”. Palmer, for instance, says “… veridical perception of the environment often requires heuristic processes based on assumptions that are usually, but not always, true. When they are true, all is well, and we see more or less what is actually there. When these assumptions are false, however, we perceive a situation that differs systematically from reality: that is, an illusion”.

Recall also that Pizlo says in his commentary, “…it is the shape of 3D objects that conveys the information we all use to perceive our world veridically … Specifically, the 3D symmetrical shapes of objects allow us not only to perceive the shapes, themselves, veridically, but also to perceive the sizes, positions, orientations and distances among the objects veridically.” And, as we noted in the target article, Pizlo and colleagues state in their recent book that “veridicality is an essential characteristic of perception and cognition. It is absolutely essential. Perception and cognition without veridicality would be like physics without the conservation laws” (p. 227).

Thus, when ITP says that the very predicates of shape, size, position, and orientation in space-time are the wrong predicates for describing reality, ITP is not pummeling straw men, it is rejecting the entire framework within which most current theories of perception are even articulated.

Hummel also argues that ITP is a special case of q-morphs proposed by Holland and colleagues. But as Hummel correctly notes, q-morphs are instances of hybrid-realist strategies. Evolutionary games show that such hybrid-realist strategies cannot compete successfully against interface perceptions that are sufficiently tuned to fitness. But even more to the point, ITP entails that the very language in which q-morphs are framed is the wrong language for describing reality. Thus ITP is not a special case of q-morphs. Hummel recognizes this point later in his commentary when he says, “… in this sense at least, their proposal diverges from simply being a special case of Holland’s q-morphs. They are going Full Monty and claiming an even weaker linkage between what’s in the world and what’s in the head …”

Current perceptual theories are still fundamentally caught in a classical physics theory of reality. ITP says that this classical model must be abandoned if our field is to move forward. It is time to catch up with the physicists who recognize that there is “No return to classical reality”.

Comment 4. Hummel argues, “The deeper problem is the authors’ confusing perception with action: They assume that what their simulated organism sees is identical to what it does.”

Reply 4. Not at all. The mathematical framework of ITP is based on the notion of a “PDA loop”, i.e., a “Perception-Decision-Action loop” in which perceptions, decisions and actions are treated separately and their interactions are studied. This is the framework in which we proved the Invention of Symmetry Theorem.

Our evolutionary games in which artificial organisms compete for resources in various territories do indeed focus on perception, rather than decisions and actions. But in this case, the games simply assumed that the decisions and actions were optimal, given the perceptual information available. Thus the games study just the effects of perception on evolutionary competitions.

However the genetic algorithms we studied in foraging games had “genes” for perception and action, and allowed both to evolve. The initial generation of organisms had comically stupid perceptions and stupid actions. They couldn’t forage to save their lives. Over hundreds of generations the perceptions and actions evolved and coordinated with each other, so that the resulting organisms were expert foragers. Never once did any kind of veridical perception make it into the last generation of organisms. In every case, only interface perceptions appeared in the last generation. Moreover, the pattern of evolution of the generations indicates to us that, in any situation in which there are more than just a few dozen “genes”, it is highly unlikely that any kind of veridical perception will appear on the stage in any generation of the algorithms. Since there are no selection pressures for any kind of veridical perceptions, they only appear by blind chance, and thus only with vanishingly small probability. In short, veridical perceptions are not even worth bothering with, they are so unfit. It is unlikely that veridical perceptions have ever appeared at any point in evolution. But if they did, they would be immediately rejected by natural selection.

Comment 5. Hummel argues, “Place the animals in a world where the payoff contingencies might change (today the best quantity of the resource is 50; tomorrow, it’s 75) and the “veridical” organism will eat the “interface” organism’s lunch.”

Reply 5. There is no a priori reason that interface strategies cannot adapt as quickly to changing payoffs as veridical strategies. So there is no reason to believe that veridical organism will eat the interface organism’s lunch.

But there is a different reason for believing that veridical strategies will not eat the interface organism’s lunch: Veridical strategies aren’t fit enough to get on the stage so that they can be lunch. But if a veridical strategy were placed on the stage, it would make a tasty morsel for an interface organism that rapidly adapts to changing payoffs.

Comment 6. Hummel argues, referring to our Figure 2, that “… if the “veridical” organism’s “yellow” perception straddled resource quantity 50 (as the “interface” organism’s “blue” perception does), then the “veridical” organism could guide its behavior as optimally as the “interface” organism does. But as illustrated in our Figure 2 (reproduced below), with yellow and green straddling 90% or more of the integral of the payoff function, the “veridical” organism simply cannot see the properties that matter in its environment. Such an organism can hardly be described as “veridical”.

Reply 6. This misses the point of Figure 2 (reproduced above). On the horizontal axis of that figure is a resource quantity in a presumed “real world”; the resource is a totally ordered quantity ranging from 0 to 100. A perceptual system that has only four perceptual categories (e.g., four colors) and that encodes as much information as possible about the true value of the resource quantity (i.e., is as veridical as possible), would have the mapping depicted in Figure 2. The order of the colors preserves the order of the quantities that map to them. And, assuming that the distribution of resources is uniform, the equal widths of the red, yellow, green and blue regions gives a minimum-variance best estimate of the true value of the resource. If one moves these colored regions around as Hummel proposes, the estimates of reality have higher variance, and are thus less tuned to veridicality. Thus, Hummel’s method of better tuning the perceptions to fitness has the necessary effect of more poorly tuning them for veridicality. Precisely a key point made by ITP.

Scott Jordan

Comment 7. Jordan says, “At first glance, ITP seems quite different from the above-mentioned realisms, for while they require veridical correspondence between perceptions and the objective world, ITP does not. Upon further reflection however, it turns out the two are actually quite similar, in that both believe perception corresponds to (i.e., represents) something. In short, both ITP and the realisms they critique constitute correspondence theories of perception.”

Reply 7. ITP is not a correspondence theory with respect to fitness. It does not claim that perceptions veridically represent fitness. It only claims that selection pressures are towards fitness (i.e. hill-climbing in the direction of increasing expected-fitness payouts), not towards truth. Evolution only leads to satisficing solutions. One only needs to be a bit more fit than the competition. One does not need to have a veridical representation of the fitness function. One’s perceptions only need to be a bit more tuned to fitness than the competition. Tuning, as we use the term, is not all or nothing. There are degrees of tuning. Selection pressures tune our perceptions to fitness only to the degree that is necessary to survive and reproduce in one’s niche. Selection pressures never tune our perceptions to reality, except as an accidental byproduct, a spandrel, of tuning to fitness.

Comment 8. Jordan argues, “This maneuver is problematic because it contradicts itself (i.e., it is logically incoherent). Perception cannot be coherently defined in contrast to reality because all perceptions are entailed within reality. There is no set we can refer to as “perception” that is not entailed within the set “reality.” Hoffman et al. however, attempt to define perception in contrast to reality in the equation P: W➔X, in which W refers to “measurable spaces” of the world, X refers to perceptual experiences, and P is a perceptual strategy. While describing this equation, the authors state, “…X is not a subset of W…”

Reply 8. There is no logical problem here. Let us denote the organism by O. Let W’ be the union of W and X, and call W’ “reality”. Then, since the perceptions X are part of the organism O, it follows that in the sense that Jordan wants, perception is a subset of reality. We agree. However we can still, with no hint of logical incoherence, ask if Xis also a subset of W, which we can call “objective reality”, i.e., reality outside the organism in which the organism is nested. The only objection left for Jordan is to say that X = W’. This amounts to metaphysical solipsism, viz., the claim that all that exists is me and my perceptions. We doubt that he would want to claim this.

Comment 9. Jordan says, “An advantage of conceptualizing organisms as embodiments of context is that it logically nests the organism, and all aspects of the organism, within reality. As a result, we need not generate epistemic gaps that lead us to assume the need for bridging principles, or “representations” that allow the organism to cross an epistemic gap.”

Reply 9. We agree that organisms are nested in reality and that understanding the interactions of the entire organism, not just its brain, with its environment is critical to a complete understanding of perception, cognition, decision, action and their interaction. We do not agree, however, that this entails that there are no epistemic gaps. Take a concrete example. In earlier work, Jordan argues that there is no epistemic gap between human observers in perceiving the mental contents of other people: “In short, we experience another’s mental contents directly because we directly experience the goals (i.e., distal effects) they generate and the means (i.e., actions) by which they generate them.”

But there is clear empirical evidence to the contrary. Consider this epistemic gap: Perilloux and colleagues report that women underestimate men’s sexual interest, and that “Men who were more oriented toward short-term mating strategies or who rated themselves more attractive were more likely to overperceive women’s sexual interest. The magnitude of men’s overperception of women’s sexual interest was predicted by the women’s physical attractiveness.”

Here is a case where evolution favors perceptions that misperceive the mental contents of others. This is no surprise to ITP. Our perceptions are about having kids, not about seeing truth, and this pattern of misperceptions is tuned to enhance the chance of having kids, with the tuning for each sex depending, e.g., on its parental investment.

More generally, the claim that there are no epistemic gaps, that perception is direct, has the problem of explaining misperceptions and illusions. What could it mean to have a direct misperception, or a misperception with no epistemic gap?

As described in our target article, the male jewel beetle tries to mate with a dimpled, glossy brown beer bottle, and will not stop until it dies. Its entire embodied being is involved in the process. But there is surely an epistemic gap here. The gap is this: The sensory information that normally allows it to successfully choose a mate is, in this case, misinforming the poor beetle.

The cosmetics industry depends for its existence on this epistemic gap: They create false cues that make a person look younger, firmer, less blemished, and so on. This false information creates, in the successful cases, the desired false beliefs. Success at creating an epistemic gap has made cosmetics a multi-billion dollar industry.

Gary Lupyan

Comment 10. Lupyan notes, “The philosopher Ludwig Wittgenstein, reportedly, once asked a friend, “Tell me, why do people always say that it’s natural for to assume that the sun went around the earth rather than the earth was rotating?”, “Well” the friend said: obviously, because it just looks as if the sun is going around the earth.” Wittgenstein responded: “Well, what would it look like [if it looked like] the earth was rotating?” (The answer, of course is that it would look exactly the same as it already looks!)”

Reply 10. Lupyan is responding to our statements in our target article such as “the pre-Socratic Greeks, and other ancient cultures, believed that the world is flat, in large part because it looks that way.” We agree. Wittgenstein’s point is well taken. But note its implication: What would the world look like if, as ITP claims, space-time and physical objects in space-time were not the nature of reality, and reality required utterly different predicates for its description than any predicate we use in our perceptions? Answer: It would look exactly the way it does. This, by the way, is the meaning of the Invention of Symmetry Theorem in our target article.

Comment 11. Lupyan, like Hummel, argues that we “… confuse perception with action and decision-making …”.

Reply 11. See our Reply 4 to Hummel on this point.

Comment 12. Referring to the figure in his commentary, Lupyan states that “We are indeed quite attuned to the stability of such towers, and yet at the same time we are sensitive to all kinds of similarities and differences between them. We can count the blocks, predict not just that it will fall, but which way, and so on. Hoffman et al. might say that this is because we do not live in a block-world. But what world do we live in?”

Reply 12. In our view, this has little to do with the block-world aspect of this example. Indeed, human observers are also quite sensitive to the physical stability of 3D objects with more complex shapes and smooth surfaces. Rather, the reason we are sensitive to all kinds of similarities and differences between the towers (and, indeed, other 3D shapes) is because we—or, more accurately, our ancestors—cared not justabout physical stability (as presumed in the current example), but also about many other things including recognition, categorization, manual interaction, etc. As Lupyan also notes (see his Comment 13), perceptual systems have to contend with multiple tasks, and it is this need to deal with multiple tasks that leads to sensitivity to many perceptual dimensions.

Comment 13. Lupyan argues that organisms “… have to contend with multiple tasks, and the way to deal with such a multitude of tasks is to evolve perceptual systems that have considerable representational flexibility. This is a problem for Hoffman et al., appear to imagine perceptual systems as very tightly constrained by highly specific problems like finding food and mates.”

Reply 13. Not at all. ITP does not entail representational inflexibility.

Here’s why. Each organism has many relevant fitness functions. Evolution has many different strategies available to deal with this (see the subsection entitled “Dedicated vs. general-purpose representations” in our target article).

At one extreme is a strategy that builds a unique representation for each relevant fitness function. This would have the advantage of allowing the organism to tune each representation as closely as needed to each fitness function. The obvious problem is that this would require enormous computational and representational resources.

At the other extreme is a strategy that builds a single representation that must deal with all of the relevant fitness functions. The advantage of this is reduced computational and representational demands. The obvious problem is that the organism cannot tune this representation as closely as needed to each fitness function. There will be tradeoffs. Tuning more to fitness function A will often force the representation to be less tuned to fitness functions B, C, and D.

It is likely that evolution settles for a compromise between these two extremes: It has more than one representation, but fewer representations than the number of fitness functions that must be dealt with. This requires that each representation handles acollection of fitness functions. The more similar the fitness functions within a collection are to each other, the more easily they can be handled by a single representation. Thus evolution probably ends up grouping fitness functions into different collections based on their similarities, and evolves a representation for each collection. If resources are at a premium, then there will be fewer representations, each handling more fitness functions. If resources are less costly, then there can be more representations, each handling more fitness functions, and each more easily tuned to its fitness functions.

These representations themselves might be further organized such that representations handling more similar collections of fitness functions have tighter interactions between them. This could lead to the evolution of perceptual modalities(e.g., vision, audition, olfaction, etc.), together with networks of representations within each modality. This does not, of course, preclude multimodal interactions as well.

This gives us a new way of thinking about object perception. The standard way is to think of object perception as the result of a process in which perception estimates the true properties of real objects in space-time that exist even if unperceived. ITP suggests something entirely different: Object perception is a satisficing solution to the problem, just described, of finding a perceptual representation that adequately, but economically, handles a variety of important fitness functions. There are no pre-existing observer-independent physical objects in an observer-independent space-time, and thus object perception is not about trying to estimate the properties of pre-existing objects. Instead, object perception is entirely about finding a tradeoff between economical use of resources and adequate tuning to a collection of fitness functions. Interesting questions arise from this new approach to object perception. What fitness functions are being represented by objects? Why are 3-D objects—with shapes, positions, colors, textures, velocities, attitudes in space—an adequate solution to the tradeoff problem?

What should be clear from this discussion is that ITP in no way requires representational inflexibility. To the contrary, it opens up a whole new way of thinking about representational flexibility and its evolutionary origins.

Another aspect of this issue is that many fitness functions are probably multidimensional, in the sense that they are non-separable functions of multiple aspects of the underlying reality. Our evolutionary game simulations indicate that in such a case, if the fitness function is non-monotonic with respect to the structures of these multiple aspects of the underlying reality, then the optimal perceptual maps will in general completely mix these aspects of the reality, making them unrecoverable from the perceptions. The more complex the fitness functions, the more aspects of reality that they are influenced by, the more likely it is that these aspects of reality are completely conflated and unrecoverable from the resulting perceptual map. How this might influence the evolution of object perception is another fascinating question.

Comment 14. Lupyan argues that perceptual systems “… converge on representational schemes that are broadly useful. We know this because these schemes have enabled us to go far beyond anything that could have been selected for by evolution: we have measured the speed of light, estimated the age of the universe, sequenced our genome, and discovered planets orbiting other stars. Maybe we did this with perceptual systems that are lousy guides to the truth, but maybe they are true enough.”

Reply 14. The IOS Theorem deals with precisely this point. Our ability to see structure in our perceptual worlds and to precisely predict how that structure changes as we act, entails nothing about the nature of the objective world except a lower bound on its cardinality. Our perceptual systems can be lousy guides to the truth, and still enjoy all this detailed and accurate prediction of structure and consequences of actions. There is no need for them to be true enough to reality as it is.

But, as we discussed in Reply 13, our representational schemes are broadly useful because they have evolved to be adequately tuned to many fitness functions. It is the adequate tuning to many fitness functions, not adequate tuning to truth, that endows our representational schemes with their broad usefulness.

Chris Fields

Finally, we close with a question raised by Chris Fields, one of the original ten commentators on the ITP paper in the Psychonomic Bulletin & Review.

Comment 15. Why are our perceptual experiences so complex?  Visual experience has megapixel bandwidth—we see every furl and crevice in the bark of a tree, every detail in a pattern of leaves, the intricate patterning on cloud tops from 35,000 feet. Why?  None of these seem at all connected with fitness, and we see them even when our activity is relaxation.  We burn loads of energy analyzing high-resolution perceptions when we seemingly don’t need to. So why isn’t our interface more like a computer’s interface?  Why all the gratuitous detail?  Why not save the energy when there’s no task at hand?

It is interesting, pleasurable, arousing of curiosity.  These all have something to do with fitness. Detail is useful in some contexts, even if not all.  We may need detail quickly when surprised.  Maybe it’s too hard to switch from high-resolution to some stripped-down low-resolution view of the world.

But none these are terribly satisfying.  There seems to be a gap here, a story that ITP needs to tell.

Reply 15. Change blindness demonstrations suggest that our perceptual experiences are richly detailed at most in a small region of visual space, perhaps only one or two degrees of visual angle in diameter. It might seem that we see much of the world in high resolution, but this is an illusion. Thus, according to ITP, our visual world is not just an interface, but an interface that we create piecemeal, on the fly, and only as much as we need in the moment. Peripheral vision is part of the interface, of course, but is computed at much lower resolution. Visual attention determines which parts of the interface we construct in greater detail at any moment. And attention itself is driven by the needs of fitness.

In addition, once a perceptual system has evolved in a specific way, it will of course continue to process information in that manner. High-resolution foveal vision clearly has huge fitness advantages (especially for predators and tree-dwelling creatures), but that does not mean that one can simply “turn off” this hard-wired mechanism in more relaxed situations where high resolution is not crucially needed for survival.

Conclusion

One of the hallmarks of a scientific theory is falsifiability, and among the strengths of science are its vigorous attempts to test and possibly falsify theories. Of course, it is human nature to prefer to falsify someone else’s theory; we are less inclined to be as vigorous in trying to falsify our own theories. But, for salubrious science, it must be done. So, for providing this essential service, we again thank John Hummel, J. Scott Jordan, Gary Lupyan, and the original ten commentators on our target article for their insightful and thought-provoking critiques.