Cramér's Theorem and the Rate Function: Large Deviations of Sample Means

What It Is and Why It Matters

Cramér's theorem is the foundational statement of large deviation theory: it tells you the exponential rate at which the probability of an atypical sample mean decays as the sample size grows. If X₁, X₂, … are i.i.d. with mean μ = ⟨X⟩, the law of large numbers guarantees S̄ₙ → μ, and the central limit theorem describes the √n-scale Gaussian jitter around it. Cramér's theorem answers the harder question: how improbable is it that S̄ₙ sits a finite distance x away from μ?

The answer is startlingly clean. The probability decays exponentially in n, P(S̄ₙ ≈ x) ≍ e^−nI(x), and the entire tail is encoded in one convex function I(x), the rate function. This matters because rare events, not typical ones, drive the interesting physics: crystal nucleation, dielectric breakdown, protein folding pathways, and — through the deep dictionary Hugo Touchette formalized — the very entropy and free energy of equilibrium statistical mechanics.

The Mechanism: Tilting the Measure

The physics behind the rate function is exponential tilting (the Cramér transform / change of measure). To make a rare mean x typical, reweight each sample by the exponential factor e^kX, defining a tilted distribution dP_k ∝ e^kX dP. The normalization is the moment generating function M(k) = E[e^kX], and its logarithm λ(k) = log M(k) is the cumulant generating function. Choosing k so that the tilted mean ⟨X⟩_k = λ′(k) equals the target x makes x the new average — no longer rare under P_k.

The cost of this reweighting is exactly the rate function. The relative-entropy price paid to shift the mean from μ to x is I(x) = kx − λ(k) evaluated at the optimal k = k(x). Because λ is convex (a consequence of Hölder's inequality), the optimum is unique wherever λ is differentiable. Physically, tilting is the probabilistic mirror of adding a source term or a chemical potential: you bias the ensemble, then read off the entropic cost of the bias. This is the same machinery that produces the Boltzmann weight e^−βE from a maximum-entropy argument.

The Key Equation: A Legendre–Fenchel Transform

The central formula is that the rate function is the Legendre–Fenchel transform of the cumulant generating function:

I(x) = sup_k∈ℝ [ kx − λ(k) ],   λ(k) = log E[e^kX].

This is the exact large-deviation analogue of the Legendre transform linking entropy S and free energy F in thermodynamics — λ plays the role of a (dimensionless) free energy and I the role of an entropy. Key properties: I is convex and non-negative; it has a unique zero at the true mean, I(μ) = 0 (the typical value costs nothing); and its curvature at the mean reproduces the CLT, I(x) ≈ (x − μ)²/(2σ²) since λ″(0) = σ². For a fair coin (X ∈ {0,1}), I(x) = x log(2x) + (1−x) log(2(1−x)) in nats — a Kullback–Leibler divergence from ½. For a standard Gaussian, I(x) = x²/2 exactly. The Gärtner–Ellis theorem (Gärtner 1977, Ellis 1984) generalizes this to correlated variables: replace λ(k) with the scaled cumulant generating function λ(k) = lim_n→∞ (1/n) log E[e^nkS̄ₙ], provided that limit exists and is differentiable.

How It Is Realized and Measured

Large deviations are directly measurable wherever you can sample many independent trials or resolve fluctuations in a driven system. The cleanest laboratory is combinatorial: histogram the fraction of heads in n coin flips or the mean of n dice, and the log-frequency of tail bins traces −I(x) with slope n. In statistical physics, the signature is the empirical measure — Sanov's theorem states that the probability of observing an empirical distribution ν instead of the true μ decays as e^{−nD(ν‖μ)}, with the rate given by the relative entropy (KL divergence).

Experimentally, large deviation functions have been extracted from single-molecule pulling assays and colloidal particles in optical traps, where fluctuation theorems (Evans, Cohen, Morriss 1993; Gallavotti–Cohen 1995) constrain the rate function of entropy production, I(−s) − I(s) = s. In nonequilibrium transport, the large deviation function of the current is computed via the Macroscopic Fluctuation Theory of Bertini, De Sole, Gabrielli, Jona-Lasinio and Landim, and measured in electron counting statistics of quantum dots. In cosmology, the rate function of the matter density in spheres predicts non-Gaussian tails of the cosmic web.

Where It Operates and How It Differs from Related Results

Cramér's theorem in its original form requires i.i.d. variables with finite exponential moments — λ(k) must be finite in a neighborhood of k = 0 (the Cramér condition). When only power-law (heavy) tails exist, the exponential decay breaks down and a single large jump dominates the deviation (the 'big-jump' or subexponential regime), so the rate function is not defined and the LDP fails.

Distinguish it from three neighbors. The central limit theorem lives at scale 1/√n and sees only the variance; large deviations live at scale 1 and see the whole tail. The law of large numbers gives only the limit point, μ; large deviations give the exponential rate of convergence to it. And concentration inequalities (Hoeffding, Chernoff, Bernstein) provide non-asymptotic upper bounds; in fact the Chernoff bound P(S̄ₙ ≥ x) ≤ e^−nI(x) is exactly the finite-n shadow of Cramér's asymptotic. The Gärtner–Ellis theorem extends the reach to Markov chains, Gaussian processes, and mean-field spin systems where correlations persist.

Applications, Significance, and Open Questions

The deepest payoff is the large deviation reformulation of statistical mechanics (Ellis 1985; Touchette 2009): the Boltzmann entropy is a rate function, the free energy is a scaled cumulant generating function, and their Legendre duality is the Legendre transform of thermodynamics. A phase transition appears as a non-differentiable point of the free energy — equivalently, a linear (non-strictly-convex) stretch or a kink in the rate function — giving a rigorous, unified criterion for spontaneous symmetry breaking. Nonconvex rate functions, which the Legendre transform cannot recover, signal ensemble inequivalence and are studied via the tilted-ensemble and level-2.5 formalisms.

Practically, large deviations underpin importance sampling in Monte Carlo, error exponents in Shannon information theory, ruin probabilities in insurance mathematics, and rare-event forecasting for climate extremes and turbulence. Open frontiers include large deviations far from equilibrium (dynamical phase transitions in the current, computed by cloning/DMRG-like tensor methods), the geometry of rate functions for interacting particle systems, and extending the framework to quantum trajectories and open quantum systems where the tilted generator becomes a non-Hermitian operator.

Central limit theorem versus Cramér's large deviation principle for the sample mean of n i.i.d. variables

Feature	Central Limit Theorem	Cramér / Large Deviations
Scale probed	Fluctuations of order 1/√n around ⟨X⟩	Deviations of order 1 (fixed x ≠ ⟨X⟩)
Probability form	Gaussian: P ∝ exp(−n(x−μ)²/2σ²)	P ≍ exp(−nI(x)), I generally non-quadratic
Governing function	Mean μ and variance σ² only	Full cumulant generating function λ(k)
Behavior near mean	Exact	I(x) ≈ (x−μ)²/2σ² (recovers CLT locally)
Requirement	Finite variance	λ(k) < ∞ near k = 0 (exponential moments)
Information captured	Second moment	All moments / entire tail shape