1997 / 2014 · Gated Recurrent Memory
Do not rewrite every memory. Learn what to keep.
A vanilla recurrent network carries one hidden state and rewrites it at every step. That gives it memory in principle, but repeated multiplication makes old information—and the gradient needed to learn from it—fade. Long Short-Term Memory and the Gated Recurrent Unit add learned valves around the update.
An LSTM separates a long-lived cell state from the hidden state it exposes. Its input gate decides what enters, its forget gate decides what survives, and its output gate decides what becomes visible:
A GRU folds cell and hidden state together. Its update gate blends old state with a candidate; its reset gate controls how much of the past helps form that candidate. It is a smaller variation on the same insight:
The controlled experiment below stores one signed value, waits through blank distractors, then asks each architecture to recall it. The cells use explicit, interpretable gate values—what a trained network would have to learn—so the forward memory mechanism stays visible.
State magnitude across the sequence. The outlined first and last cells are store and recall; everything between them is a distractor. Darker means more of the original signed signal remains.
Try it — stretch the memory
Gates improve memory; they do not remove recurrence. Every step still waits for the previous one, so sequence processing cannot be fully parallelized.
The state is still a compression. However carefully it is gated, one fixed-size vector must summarize everything read so far.
This laboratory supplies the gate policy. Real LSTMs and GRUs learn their gates with backpropagation through time. Here the values are exposed and fixed to isolate what the architecture does after learning.
Long is not infinite. A forget gate close to one still leaks. Enough steps eventually erode the signal.