Visual Perception as Construction

You are not seeing the world. You are seeing a model of the world that your brain constructs in real time, and the model is — by design — systematically wrong about what's actually out there. This isn't a bug. It's the whole point.

Motion That Isn't There

The clearest demonstration of visual construction comes from motion illusions. Jacob Yates's analysis of the famous Mario illusion — where Marios appear to traverse a level despite never leaving their positions — cuts through the mystification by showing exactly what's happening in space-time plots. Visual motion is orientation in space-time. Your visual system detects this orientation using local filters — spatiotemporal edge detectors that are descendants of the oriented receptive fields Hubel and Wiesel discovered in primary visual cortex back in the 1960s.¹

The critical detail: these motion detectors don't care about the sign of the edge. They respond to orientation regardless of whether the leading edge is light-on-dark or dark-on-light. The Adelson-Bergen motion energy model from 1985 explains this by combining pairs of oriented filters that are 90 degrees out of phase — a "quadrature pair" that gives phase-invariant responses. This is elegant engineering, but it means the system can be fooled whenever a stimulus creates oriented structure in space-time without anything actually moving. The Mario illusion does exactly this: colors shift at the edges of each Mario, creating oriented space-time structure that the motion detectors faithfully report as rightward motion.¹

What I find most instructive here is how the illusion reveals the mechanism. We don't see motion because something is moving. We see motion because oriented space-time structure activates a specific detection pathway. Motion perception is a conclusion the brain draws from limited evidence, not a direct readout of physics.

Ambiguity as Feature, Not Bug

Alex Holcombe's trio of illusions makes a complementary point about object recognition.² The Coffer Illusion — initially appearing as rectangular door panels before flipping to reveal sixteen circles — shows that the visual system is fundamentally in the business of grouping pixels into objects, and when the grouping is ambiguous, the brain picks one interpretation and commits to it. Most people see rectangles first, probably because rectangular shapes are more common in daily environments than circles. Your visual statistics shape what you see.

The face-sex illusion is even more revealing. The same androgynous face appears female when lightened and male when darkened, because the brain uses contrast between skin and features (lips, eyes) as a sex cue — a correlation that's statistically real but that nobody would consciously list as a defining characteristic of gender. The system is using information you don't know you have.²

Both of these connect to the broader predictive processing framework: the brain doesn't passively receive visual input but actively interprets it by applying prior knowledge — statistical regularities, learned categories, contextual expectations. Visual illusions aren't cases where perception fails. They're cases where the normal constructive process is made visible because the stimulus is specifically designed to exploit the brain's assumptions.

The Desktop Interface

Donald Hoffman pushes this much further than most vision scientists are comfortable with.³ His argument, grounded in evolutionary game theory, is that perception was never designed to show us reality. It was designed to show us fitness payoffs — the survival-relevant consequences of possible actions. The mathematical result, proved by his colleague Chetan Prakash, is stark: an organism tuned to fitness will always outcompete one tuned to truth, because fitness functions almost never align with the objective structure of reality. The canonical example: too little water kills you, too much drowns you — a bell curve over an objective linear axis. An organism that collapses both extremes into a single "danger" representation and marks the middle as "safe" will beat the truth-tracker every time.

Hoffman's metaphor is the computer desktop. The blue rectangular icon on your screen has color, position, and shape — but none of those properties describe the file itself, or anything in the computer's innards. The icon is useful precisely because it hides the underlying complexity. We can't form a true description of the computer's architecture using only desktop categories. Similarly, we can't describe objective reality using the categories of perception — space, time, objects — because those categories exist to guide adaptive behavior, not to represent truth.

This is a more radical claim than standard predictive processing. Predictive processing says the brain builds a model and checks it against sensory evidence. Hoffman says the model was never trying to approximate reality in the first place — it was always a fitness-guided interface. The snake you see isn't a slightly distorted version of a real snake. It's a symbol in your species-specific desktop, created by evolution to help you not die. The distinction between "imperfect representation" and "useful fiction" might seem academic, but it matters for how seriously we take the furniture of perception as evidence about fundamental reality.³

The Cultural Dimension

The mind-as-container metaphor that shapes how English speakers talk about consciousness — thoughts "entering" the head, ideas "held" in mind — is itself a perceptual construction projected onto experience. As Lakoff and Johnson showed, abstract thought is always metaphorical, built on physical interactions with the world. But the metaphor isn't universal. The Iban of Borneo don't have an elaborated concept of the mind as a container, which means that certain psychiatric symptoms (like thought insertion) simply aren't available to them as experiential categories.⁴

This connects to Barrett's work on constructed emotion: if you don't have the concept, you don't have the experience — not because you lack the raw sensations, but because your brain constructs a different experience from the same inputs. Perception is construction all the way up, from edge detection to emotional categorization to the felt sense of having a mind.

When Visual Construction Goes Missing

Perhaps the most striking evidence that visual perception is construction comes from what happens when the visual constructor never develops. A Frontiers in Psychology paper documents a remarkable finding: across the entire published literature, not a single case of a congenitally blind person developing schizophrenia has ever been reported.⁵ Blindness acquired later in life doesn't protect you. It has to be congenital — the visual cortex must never have learned to construct the visual world.

The "diametric model" of mental illness, proposed by the imprinted brain theory, offers a framework for why. Schizophrenia involves hyper-mentalism — an excess of the brain's tendency to construct models of other minds, find patterns, and attribute intentions. The visual system, which is the brain's most sophisticated constructive apparatus, may be a critical input to this hyper-construction. Without it, the brain develops compensatory strengths in other senses — congenitally blind people have enhanced pitch discrimination, touch sensitivity, and auditory processing — but the mentalistic modeling that can go haywire in schizophrenia never gets the visual scaffolding it needs to over-build.

Conversely, congenital blindness is associated with increased risk of autism, which the diametric model frames as the opposite pole: under-mentalism, reduced model-building, enhanced mechanistic processing. Blind children frequently show "marked impairments in interpersonal engagement" that overlap with autistic features. The combination of blindness, autism, and musical savantism is unusually common — as if the cortical resources that would have been devoted to visual construction get redirected toward auditory processing with extraordinary results.⁵

The connection to predictive processing is direct: if perception is controlled hallucination, then the visual cortex is the brain's most powerful hallucination engine. Remove it from birth, and the brain's capacity for certain kinds of constructive over-reach is permanently reduced — protecting against psychosis while increasing the risk of under-construction. The visual system isn't just how we see. It's a major component of how we build models of reality, and when it's absent, the entire model-building architecture develops differently.

What Remains

If perception is construction, what's left? Not nothing — the construction works. You catch the ball, avoid the cliff, recognize your friend. But the temptation to treat perceptual experience as a window onto reality, rather than a species-specific user interface, is probably the deepest cognitive bias there is. We're Truman, and we can't quite believe the sky is a set.