LSTM Networks

Why LSTMs matter

A plain recurrent neural network is, in principle, a universal sequence model: at each step it folds the current input into a hidden state and passes that state forward. In practice it forgets almost immediately. The reason is arithmetic, not architecture. Training uses backpropagation through time, which unrolls the recurrence and multiplies the same recurrent weight matrix W once per step. Gradients therefore scale like the powers of W's largest eigenvalue — they shrink toward zero (the vanishing gradient problem) or blow up (exploding gradients). After 10–20 steps the signal from a distant token has faded to noise, and the network cannot connect a pronoun to the noun it referred to fifteen words earlier.

The LSTM's contribution is a single, surgical change: alongside the ordinary hidden state h_t, keep a separate cell state c_t that is updated by addition, not repeated matrix multiplication. Because the dominant path from c_{t-1} to c_t is essentially the identity (scaled by a gate close to 1), the gradient along that path stays close to 1 across many steps. That protected highway is what makes long-range learning possible, and it is why LSTMs — not plain RNNs — became the workhorse of sequence modeling from roughly 2014 to 2018.

How an LSTM cell works, step by step

At each time step the cell receives the current input x_t and the previous hidden state h_{t-1}, concatenates them, and computes four things. Three are gates — sigmoid vectors in [0, 1] that act as soft, element-wise valves — and one is a candidate update squashed by tanh into [-1, 1]:

• Forget gate f_t = σ(W_f · [h_{t-1}, x_t] + b_f) — how much of the old cell state to keep. 0 erases a memory slot, 1 preserves it.
• Input gate i_t = σ(W_i · [h_{t-1}, x_t] + b_i) — how much of the new candidate to write.
• Candidate g_t = tanh(W_g · [h_{t-1}, x_t] + b_g) — the proposed new content for the cell.
• Output gate o_t = σ(W_o · [h_{t-1}, x_t] + b_o) — how much of the cell state to expose as the hidden state.

The state updates are the two lines that matter most:

• Cell update: c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t — forget part of the old memory, add part of the new. The ⊙ is element-wise (Hadamard) multiplication.
• Hidden update: h_t = o_t ⊙ tanh(c_t) — the exposed output is a gated, squashed view of the cell.

The whole magic lives in the cell-update line. Differentiate c_t with respect to c_{t-1} and you get f_t — no weight matrix, no saturating nonlinearity in the way. When the network learns to hold f_t ≈ 1 for a slot, the gradient through that slot is ≈ 1 for as long as the memory is held. This is the constant error carousel: error circulates through the cell state without decaying. The gates decide when to write, when to keep, and when to read — turning an otherwise leaky recurrence into a differentiable memory register.

The constant error carousel, precisely

The name comes from Hochreiter and Schmidhuber's original paper. Consider the gradient of a loss L at step T with respect to the cell state at step t < T. Following only the additive cell path:

∂c_T / ∂c_t  =  ∏_{k=t+1..T}  f_k        (element-wise)

Compare that to a vanilla RNN, where the analogous product is ∏ diag(σ')·Wᵀ — a chain of weight matrices and derivative terms whose magnitudes are almost never exactly 1. In the LSTM the product is just a product of forget gates. If the network wants to remember something for 200 steps, it drives the relevant coordinates of f_k to ≈ 1, and the product stays ≈ 1. That is a qualitatively different regime: a plain RNN's gradient must decay unless W is finely tuned, whereas the LSTM's gradient decays only if the network chooses to forget. Gradients can still explode along the additive path, which is why LSTM training almost always pairs the cell with gradient clipping by global norm.

LSTM vs GRU vs plain RNN vs Transformer

	LSTM	GRU	Vanilla RNN	Transformer
Gates	3 (forget, input, output)	2 (update, reset)	0	N/A (attention)
Separate cell state	Yes (c_t and h_t)	No (one state)	No	No (keys/values)
Weight matrices / unit	4	3	1	Q, K, V, O per head
Time per step / layer	O(H² + HD)	O(H² + HD)	O(H² + HD)	O(T²·H) attention
Parallel across time?	No (sequential)	No (sequential)	No (sequential)	Yes (all positions at once)
Long-range memory	Hundreds of steps	Hundreds of steps	~10–20 steps	Full context window
Introduced	1997	2014	1980s–1990	2017
Best for	Long sequences, streaming, tight memory	Similar, cheaper & faster	Toy / short sequences	Large-scale, GPU-parallel modeling

The GRU trades the LSTM's third gate and separate cell state for speed. It couples the forget and input decisions into a single update gate (what you keep is exactly what you don't overwrite) and adds a reset gate that controls how much past state feeds the candidate. With three weight matrices instead of four it is roughly 25% cheaper. Empirically the two are close; the LSTM's extra machinery tends to pay off on very long sequences and tasks that need precise, counting-like memory, while the GRU often wins on smaller datasets where fewer parameters means less overfitting.

LSTMs and sequence-to-sequence

The application that put LSTMs on the map was sequence-to-sequence (seq2seq) learning, introduced by Sutskever, Vinyals, and Le in 2014 for machine translation. The idea: run one LSTM (the encoder) over the source sentence and take its final cell/hidden state as a fixed-length "thought vector" summarizing the whole input; then run a second LSTM (the decoder), initialized from that state, to generate the target sentence one token at a time, feeding each predicted token back as the next input. Google Translate switched its production system to exactly this design (GNMT, 2016), stacking deep LSTMs with residual connections.

The fixed-length bottleneck — cramming an entire sentence into one vector — is also what motivated attention (Bahdanau et al., 2014), which lets the decoder look back at every encoder state instead of just the last one. Attention layered on top of LSTM seq2seq was the state of the art until the Transformer (2017) dropped recurrence entirely and kept only attention, winning on parallelism and scale.

Implementation in Python (NumPy forward pass)

A single LSTM cell forward pass makes the four equations concrete. This is deliberately un-vectorized across time so the recurrence is visible; production code batches and uses cuDNN's fused kernel.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def lstm_step(x_t, h_prev, c_prev, params):
    """One LSTM time step.
    x_t:    (D,)   input at time t
    h_prev: (H,)   previous hidden state
    c_prev: (H,)   previous cell state
    params: dict of weight matrices (H, H+D) and biases (H,)
    """
    z = np.concatenate([h_prev, x_t])          # (H + D,)

    f = sigmoid(params['W_f'] @ z + params['b_f'])   # forget gate
    i = sigmoid(params['W_i'] @ z + params['b_i'])   # input gate
    o = sigmoid(params['W_o'] @ z + params['b_o'])   # output gate
    g = np.tanh(params['W_g'] @ z + params['b_g'])   # candidate

    c_t = f * c_prev + i * g        # cell update — the additive carousel
    h_t = o * np.tanh(c_t)          # hidden update — gated readout
    return h_t, c_t

def lstm_forward(xs, params, H):
    """Run the cell over a sequence xs of shape (T, D)."""
    h = np.zeros(H)
    c = np.zeros(H)
    outputs = []
    for x_t in xs:                  # inherently sequential: h,c chain forward
        h, c = lstm_step(x_t, h, c, params)
        outputs.append(h)
    return np.stack(outputs)        # (T, H)

Note the forget-gate bias trick: initializing b_f to +1 instead of 0 starts the sigmoid near 1, so the cell state is retained by default and the carousel is intact from the very first update. Jozefowicz et al. (2015) found this to be one of the most reliable LSTM training improvements, and PyTorch/Keras apply it (or offer it) by default.

A short history

Hochreiter's 1991 diploma thesis was the first formal analysis of why gradients vanish in deep recurrent nets. The 1997 LSTM paper (Neural Computation 9(8):1735–1780) introduced the cell and the constant error carousel — but, surprisingly, had no forget gate: the original cell could only accumulate, never actively erase. Gers, Schmidhuber, and Cummins added the forget gate in 2000, which is why modern "LSTM" almost always means the forget-gated variant. Peephole connections (letting gates see the cell state directly) followed in 2000–2002. After a decade of quiet, GPUs and large datasets made LSTMs dominant: they set records in handwriting recognition (2009), then speech recognition and machine translation (2014–2016), before the Transformer era.

Common misconceptions and pitfalls

• "The gates output binary 0/1 decisions." No — they are sigmoids producing continuous values in [0, 1], applied element-wise. A gate value of 0.7 means "keep 70% of this coordinate." This continuity is exactly what makes the whole cell differentiable.
• "LSTMs never suffer from vanishing gradients." They mitigate it, not eliminate it. If the network learns forget gates well below 1, memory still decays; and gradients can still explode along the additive path — clip by global norm (typically 1–5).
• "c_t and h_t are the same thing." They are distinct. c_t is the internal, protected memory; h_t = o_t ⊙ tanh(c_t) is the gated, exposed view. Only h_t is passed to the next layer; both are passed to the next time step.
• "You can parallelize an LSTM across time on a GPU." You cannot — step t needs h_{t-1}. You batch across sequences and vectorize the four matmuls within a step, but the length-T critical path is sequential. This is the structural reason Transformers overtook LSTMs at scale.
• "More gates always beats fewer." Not reliably. GRUs match LSTMs on many benchmarks with fewer parameters; pick based on sequence length, data size, and latency budget, then validate empirically.