How do we take the representations extracted from perception and use them to guide thought and behavior? In this step, we turn to a second foundational mechanism that shapes cognition: learning. Across species, organisms rely on experienced consequences to form causal expectations. This step introduces the central ideas of reinforcement learning.

With an understanding of how the visual system extracts information from the environment, we now turn to how the mind uses this information to guide behavior. Conditioning paradigms show that animals can learn from regularities in the environment, such as rewards and punishments. But what computations could underlie these cognitive processes? Could we model the process of learning? This is the question that American psychologists Robert Rescorla and Allan Wagner sought to answer.

Their work led to the Rescorla–Wagner model of Pavlovian conditioning, which formalized a simple but powerful idea: learning occurs when the world delivers an outcome different from what the organism expected.

Let’s work through an example. Imagine yourself in the position of a fox.

Day 1: While foraging, you come across two rocks and suspect that there might be a tasty mouse beneath one of them. Since you are uncertain which to pick, both rocks start out with a low positive expected reward (blue bars).

You choose the right rock and indeed find a mouse (a strong reward; green bar). This triggers a positive prediction error δ (orange-outlined bar) because the outcome was substantially better than you had anticipated.

As a result, you update the expected value of the right rock (purple-outlined bar) in proportion to this prediction error. But you do not want your estimate to fluctuate dramatically after every single experience, so the update is only a fraction of the full prediction error, scaled by your learning rate α.

Reinforcement Learning: Day 1

Day 2: You return to the same rocks. Out of curiosity, you flip the left one and are bitten by a snake (a strong punishment; red bar). Because this outcome was much worse than you expected, it generates a large negative prediction error, which drives a downward update of the expected value for the left rock.

Reinforcement Learning: Day 2

Day 3: You have already learned that the right rock tends to yield food (positive expected value), whereas the left rock tends to hide an angry snake (negative expected value). You exploit this knowledge, choose the right rock, and capture another mouse.

Reinforcement Learning: Day 3

In the real world, outcomes are rarely this deterministic. Sometimes you would find food, sometimes nothing, and sometimes danger.

Under such uncertainty, prediction errors arise on each encounter, and your expected values continue to adjust. Over time, however, they fluctuate around the true statistical structure of the environment, reflecting the associations you uncover through experience.

In Marr’s terms, this finding is especially important because it links the computational level to the implementational level: the abstract idea of prediction-error learning appears to have a biologically plausible neural realisation in the brain.

This computational principle appears to have a direct neural counterpart. In 1997, Wolfram Schultz, Peter Dayan, and Read Montague showed that dopamine neurons in the primate midbrain fire in a pattern that closely resembles the prediction error signal of reinforcement learning models. When a reward is better than expected, dopamine neurons increase their firing; when a reward is worse than expected, they decrease it; and when a reward matches expectation, they do not respond at all. In Marr’s terms, this finding is especially significant because it links the computational and implementational levels: the Rescorla-Wagner model specifies the computational principle, and dopaminergic prediction-error responses suggest that the brain may implement something functionally analogous at the neural level.

Furthermore, what we can learn from this is that from a computational perspective, conditioning is not a trivial association-formation but a process of extracting causal structure from the environment. As Rescorla put it:

“Pavlovian conditioning is not a stupid process by which the organism willy-nilly forms associations between any two stimuli that happen to co-occur. Rather, the organism is better seen as an information seeker using logical and perceptual relations among events, along with its own preconceptions, to form a sophisticated representation of its world. Indeed, […] I favor an analogy between animals showing Pavlovian conditioning and scientists identifying the cause of a phenomenon.”

– Rescorla (1988)

Authors: Fabian Mueller & William Palmer

References

Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical Conditioning II (pp. 64–99). Appleton-Century-Crofts.

Soto, F. A., Vogel, E. H., Uribe-Bahamonde, Y. E., & Perez, O. D. (2023). Why is the Rescorla-Wagner model so influential? Neurobiology of Learning and Memory, 204, 107794.

Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press.

Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599.

O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 38(2), 329–337.

McClure, S. M., Berns, G. S., & Montague, P. R. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron, 38(2), 339–346. Rescorla, R. A. (1988). Pavlovian conditioning: It’s not what you think it is. American Psychologist, 43(3), 151–160.