Pattern recognition explains how the visual system extracts features from the retinal image, but bottom-up feature detection alone cannot resolve the fundamental ambiguity inherent in perception. To interpret ambiguous sensory data, the brain must draw on prior knowledge, and Bayesian inference gives us the mathematical language to describe how.

We have seen that pattern recognition provides an elegant means of constructing representations from sensory input. Yet its bottom-up architecture cannot solve the inverse problem of perception: many different physical scenes can produce the same retinal image. To infer the most likely causes of sensory input, the brain must therefore incorporate prior knowledge about how the world typically behaves. Bayes’ theorem provides the mathematical framework for probabilistically combining this prior knowledge with sensory evidence:

The prior (expectation) encodes expectations about possible world states. It represents the probabilities assigned to possible world states before the current sensory evidence is considered. For example, light sources are usually white or yellowish, objects are solid, and faces are convex.

The likelihood (sensory data) captures how probable the observed sensory input is, given a particular world state.

Combining the prior and likelihood yields the posterior (the percept): our updated belief about the state of the world after taking the sensory evidence into account.

To illustrate the key intuition, take a look at the image below. In Bayesian inference, beliefs about the world are represented as probability distributions: rather than committing to a single answer, the model represents a spread of possibilities, each with an associated probability. The curves below illustrate this. The prior (purple) represents expectations about likely world states. The likelihood (teal) reflects how compatible the sensory signal is with each possible world state. The posterior (amber) is the outcome: in this Gaussian example, it is a narrower distribution that sits between the two, weighted toward the more precise source. What we actually perceive here is the posterior. The posterior is weighted by precision (the inverse of variance), reflected in how narrow the curve is. In our example, the precise prior dominates the broader likelihood, pulling the posterior toward the expectation.

Bayes’ theorem

Thomas Bayes, 1701–1761

The core idea has its roots in Hermann von Helmholtz’s work in the 1860s. Helmholtz proposed that perception operates via unconscious inference. Because the world causes our sensations, perception has to run this relationship in reverse, using sensations to infer what in the world most likely produced them. In this view, perception is active rather than passive: the mind does not simply receive the world but reconstructs it. Doing so requires knowledge of how world states give rise to sensations, and this mapping is called an internal model of the world.

Hermann von Helmholtz, 1821–1894

This intuition sat largely dormant as a formal idea until researchers in speech perception picked it up in the 1960s. Morris Halle and Kenneth Stevens proposed that recognizing spoken words might work in a similar way: a system generates candidate signals and compares them with the incoming sound until it finds the best match. They called this analysis-by-synthesis. The label stuck, and the idea gradually spread to vision and perception more broadly.

What made it powerful was the realization, spelled out by Yuille and Kersten in 2006, that this is more than just a metaphor. Generating and evaluating candidate interpretations is precisely what Bayesian inference computes. The two frameworks describe the same process from different angles: Bayesian inference specifies how prior knowledge and sensory evidence should be combined, while analysis-by-synthesis describes a possible process through which such inference might be carried out.

Bayesian inference therefore provides a computational account of how prior knowledge and sensory evidence can be combined to infer the causes of sensory input. But how might the brain implement this process? One influential answer is predictive coding, which we examine in the next step.

Authors: Fabian Mueller & William Palmer

References

Yuille, A., & Kersten, D. (2006). Vision as Bayesian inference: analysis by synthesis? Trends in Cognitive Sciences, 10(7), 301–308. https://doi.org/10.1016/j.tics.2006.05.002

Helmholtz, H. von (1867). Handbuch der physiologischen Optik [Handbook of Physiological Optics], Volume 3. Leipzig: Leopold Voss.

Griffiths, T. L., Chater, N., Kemp, C., Perfors, A., & Tenenbaum, J. B. (2010). Probabilistic models of cognition: Exploring representations and inductive biases. Trends in Cognitive Sciences, 14(8), 357–364. https://doi.org/10.1016/j.tics.2010.05.004