Evolutionarily Stable Strategy

The core idea: uninvadable behavior

Imagine an entire population playing one strategy — say, every animal escalates every fight. Now drop in a single mutant doing something different. If that mutant leaves more offspring than the residents, it spreads, and the original strategy was not stable. If no possible mutant can out-reproduce the residents, the strategy is an evolutionarily stable strategy. That is the whole concept: a strategy that, once common, defends itself against every alternative by the simple fact that the alternative earns fewer descendants.

The decisive move John Maynard Smith and George Price made in 1973 was to measure payoffs in fitness — expected number of surviving offspring — rather than in money or utility. That swap matters enormously. Classical game theory (von Neumann and Morgenstern, 1944) assumed rational players who compute their best response. But a digger wasp, a stickleback, or a dung fly computes nothing. By denominating payoffs in offspring, Maynard Smith let natural selection do the optimizing automatically: whichever strategy reproduces fastest becomes common, and the ESS is wherever that race comes to rest. No brains required.

The formal definition and its two conditions

Write E(X,Y) for the expected fitness payoff to an individual playing strategy X against an opponent playing strategy Y. A resident strategy S is an ESS against a mutant T if one of two conditions holds:

1. Strict condition: E(S,S) > E(T,S). The resident does better against the population (which is almost entirely S) than any mutant does. This alone makes T unable to gain a foothold.
2. Tie-breaker: E(S,S) = E(T,S) and E(S,T) > E(T,T). If the mutant matches the resident against the common type, the contest is decided by the rare encounters in which the mutant meets a copy of itself — and there, the resident must win.

The second condition is exactly what separates ESS from Nash equilibrium. A Nash equilibrium only demands E(S,S) ≥ E(T,S), which permits neutral mutants to drift in for free. The ESS tie-breaker closes that loophole by checking the second-order encounters. Geometrically, an ESS is an attractor of the replicator dynamics — the differential equation that says a strategy's frequency grows in proportion to how far its payoff sits above the population average. Push the population a little away from an ESS and selection pushes it back; that local stability is the property a bare Nash point can lack.

The Hawk-Dove game, step by step

The canonical worked example is a contest over a resource worth V fitness units, where escalation risks an injury costing C. Each contestant uses one of two strategies. A Hawk escalates and fights until injured or victorious. A Dove displays, and retreats the instant the opponent escalates. Resolve every pairing:

• Hawk vs Hawk: they fight; on average one wins V and one is injured for −C, so each expects (V − C)/2.
• Hawk vs Dove: the Dove flees, the Hawk takes the whole resource: Hawk gets V, Dove gets 0.
• Dove vs Dove: they display, then share: each gets V/2.

Now test pure Hawk. In an all-Hawk population, every individual plays Hawk-vs-Hawk for (V − C)/2. A rare Dove playing against Hawks earns 0. Pure Hawk is an ESS only if (V − C)/2 > 0, i.e. V > C — the prize is worth more than the worst injury. When injury is cheap relative to the resource, ruthless aggression wins and stays.

But for almost any real contest C > V: serious injury (a torn flank, a broken jaw) is reproductively catastrophic, far costlier than one meal or one mating. Then (V − C)/2 is negative, a pure-Hawk population is invadable by Doves, and there is no pure ESS. The stable solution is a mixed strategy: play Hawk with probability p* = V/C. At that frequency the average payoff to Hawk and to Dove are exactly equal, so neither type can spread — the population is pinned. This is why field studies of red deer, fiddler crabs, and spiders find ritualized displays, roaring contests, and size assessments rather than fights to the death: the math of V and C forbids relentless aggression.

Hawk-Dove payoff matrix and the V/C threshold

Scenario (payoff to row player)	vs Hawk	vs Dove	ESS outcome
Hawk	(V − C)/2	V	If V > C: pure Hawk. If C > V: mixed, Hawk with probability V/C
Dove	0	V/2
Worked numbers below assume the resource is worth V = 50 fitness units
V = 50, C = 100 (injury twice the prize)	Mixed ESS: Hawk played with p* = 50/100 = 0.50		Half escalate, half bluff
V = 50, C = 250 (injury very costly)	Mixed ESS: p* = 50/250 = 0.20		Only 20% escalate
V = 50, C = 25 (cheap injury)	V > C → pure Hawk is an ESS		Everyone escalates

The players and the assumptions ESS requires

ESS is a clean idealization, and like any model it rests on conditions worth naming because real biology bends them:

• Large, well-mixed population. Encounters are random pairwise contests; the classic theory ignores spatial clustering. Adding spatial structure (lattice models, viscous populations) can stabilize cooperation that the well-mixed version forbids.
• Asexual, haploid replicators (to first order). Strategies breed true. Diploid genetics, dominance, and recombination can prevent a population from reaching the ESS phenotype even when it exists — the genetic ESS and phenotypic ESS need not coincide.
• Symmetric contests. Both players are interchangeable. When roles differ — resident versus intruder, large versus small — the relevant solution becomes an asymmetric ESS, and the famous result is that an arbitrary, payoff-irrelevant cue (such as "owner wins") can become a stable convention. This is the Bourgeois strategy: respect ownership, escalate as owner, retreat as intruder. Speckled wood butterflies (Pararge aegeria) defending sun-spots show almost exactly this rule.
• One-shot or memoryless encounters. Repeated games with memory (the Iterated Prisoner's Dilemma) open the door to reciprocal strategies like Tit-for-Tat, which Robert Axelrod's 1980 tournaments showed can be evolutionarily stable against many invaders — though no single strategy is unbeatable in the iterated game.

Where ESS shows up in real organisms

• Side-blotched lizards (Uta stansburiana). Barry Sinervo and Curtis Lively (1996, Nature) documented three male throat-color morphs locked in a rock-paper-scissors cycle: orange (ultradominant, large territories) beats blue (mate-guarders), blue beats yellow (sneaker mimics of females), and yellow beats orange. Each morph's frequency oscillates with a roughly 6-year period. There is no single-morph ESS — the only stable outcome is the cyclic polymorphism itself, the textbook case of a game with no static equilibrium.
• Fig-wasp sex ratios. W. D. Hamilton's 1967 local-mate-competition theory predicts that when brothers compete to inseminate their sisters inside one fig, the ESS sex ratio is strongly female-biased — observed broods are often only 8–20% male, sharply below the 50% Fisher predicted for outbred species. The fit between predicted and measured male fraction across dozens of wasp species is among the most quantitatively successful tests in evolutionary biology.
• Dung-fly mating times. Geoff Parker showed male yellow dung flies (Scathophaga stercoraria) copulate for an ESS duration near 36 minutes — the marginal-value point where extra fertilization gained by guarding longer just balances the offspring lost by not searching for a fresh female. The observed mean matches the predicted optimum closely.
• Bacterial cheaters and producers. In microbial populations, "producer" cells that secrete a costly public good (a siderophore that scavenges iron, or an invertase that digests sucrose) can be invaded by "cheater" mutants that consume the good without paying. The stable mixed frequency of producers and cheaters is a mixed-ESS analog and a live model system for the evolution of cooperation and its breakdown.
• Plant seed dormancy and germination timing. The fraction of seeds that germinate immediately versus stay dormant is a bet-hedging ESS tuned to the variance in good-versus-bad years — germinate too eagerly and a drought wipes out the cohort; stay dormant too long and competitors seize the soil.

ESS vs Nash equilibrium vs optimal strategy

Property	Evolutionarily stable strategy	Nash equilibrium	Group-optimal strategy
Who decides	Natural selection (no foresight)	Rational agents	A hypothetical planner
Payoff currency	Fitness (offspring)	Utility	Total population fitness
Stability requirement	Uninvadable by rare mutants (dynamic)	No profitable unilateral deviation (static)	Maximizes the average — often unstable
Defining test	E(S,S) > E(T,S), or tie + E(S,T) > E(T,T)	E(S,S) ≥ E(T,S) only	Maximize mean payoff over all players
Logical relationship	Strict subset of Nash equilibria	Superset; allows neutral drift in	Generally not a Nash point at all
Hawk-Dove result	Hawk with probability V/C	Same mixed point (and pure points)	All-Dove (sharing maximizes the average)
Why it's not group-optimal	All-Dove is invaded by a single Hawk	—	Cannot resist invasion — the tragedy of the commons

The last row is the philosophical sting of the theory. The group-best outcome in Hawk-Dove is everyone being a Dove and sharing peacefully (average payoff V/2). But that population is wide open to a single Hawk mutant who steals V from every neighbor. Selection drives the population away from the group optimum toward the lower-average ESS. ESS is therefore a precise statement of why evolution does not produce harmonious, welfare-maximizing populations — it produces uninvadable ones.

Common misconceptions and pitfalls

• "An ESS is the best strategy for the species." No — it is the strategy no mutant can beat, which is frequently worse for the group than an unstable cooperative alternative. ESS describes what selection settles on, not what would maximize collective welfare.
• "Animals must understand the game to play it." The opposite is the point. Payoffs are in offspring, so the strategy is encoded in genes and development, not reasoning. A wasp obeys an ESS the way a planet obeys gravity — without computing anything.
• "A mixed ESS means each animal must randomize." Not necessarily. A population fraction V/C of pure Hawks and the rest pure Doves gives identical average payoffs to individual randomization. Polymorphism and individual mixing are payoff-equivalent; genetics decides which appears.
• "Every game has an ESS." False. Rock-paper-scissors games (like the side-blotched lizard morphs) have none — the population cycles forever. When ESS predicts no static point, real systems often genuinely oscillate.
• "ESS and Nash equilibrium are the same." Every ESS is a Nash equilibrium, but many Nash equilibria are not ESSs because they fail the stability tie-breaker E(S,T) > E(T,T) and can be eroded by neutral drift.
• "The Bourgeois 'owner wins' rule reflects real fighting ability." It need not. In the symmetric Hawk-Dove-Bourgeois game, the convention can be completely arbitrary — "respect ownership" is stable purely because everyone follows it, even if owner and intruder are physically identical. The cue is a coordination device, not a measure of strength.