Reinforcement: How Selection Against Hybrids Strengthens Mating Barriers

What reinforcement is and where it happens

Reinforcement is the increase in prezygotic (premating) reproductive isolation between two taxa that results from natural selection against hybrids. It occurs at the final stage of speciation, typically in a zone of secondary contact where two populations that diverged in allopatry meet and begin to interbreed.

The logic is straightforward. Two populations diverge in isolation and evolve partial postzygotic incompatibilities — hybrids that are sterile, inviable, or ecologically unfit. When the populations reconnect:

• Individuals that mate across the boundary waste gametes on doomed hybrid offspring — a direct fitness cost.
• Individuals that mate only with their own type leave more surviving grand-offspring.
• Any heritable trait that biases mating toward conspecifics is therefore favored and spreads.

The outcome is a strengthening of assortative mating — 'like mates with like.' Reinforcement thus converts incidental, partial isolation into an actively selected, robust barrier, finishing the job of splitting one lineage into two. It is the one part of speciation where selection directly favors reproductive isolation itself, rather than isolation arising as a byproduct.

The mechanism, step by step

Reinforcement requires a specific sequence of conditions, and each step is a potential point of failure:

• 1. Partial premating isolation exists. Two populations diverge in allopatry and accumulate some genetic differences.
• 2. Costly hybrids form on contact. Hybrids suffer reduced fitness — Dobzhansky–Muller incompatibilities cause sterility/inviability, or hybrids are ecologically intermediate and outcompeted.
• 3. Heritable variation in mating traits is present. There must be genetic variation for signals or preferences (song, color, pheromone, flowering time) that affects who mates with whom.
• 4. Selection acts. Individuals bearing 'choosier' or more distinctive variants avoid the hybridization cost and out-reproduce indiscriminate individuals.
• 5. Prezygotic isolation strengthens. Choosy alleles rise in frequency specifically in the contact zone, sharpening the mating barrier.

The chief antagonists to this process are gene flow and recombination. Migration from allopatric populations keeps importing indiscriminate 'mater' alleles, and recombination breaks apart the association between the choosiness allele and the alleles it is trying to protect. Reinforcement therefore succeeds only when selection against hybrids is strong enough to overcome this homogenizing 'recombination load.'

Key molecules, genes, and a concrete example

The best-resolved molecular case is Phlox drummondii, a Texas wildflower studied by Robin Hopkins and Mark Rausher (Nature, 2011). Where P. drummondii overlaps its relative P. cuspidata, its flowers are dark red; in allopatry they are light blue. Hybrids between the two species are largely sterile, so cross-pollination is a dead loss.

The shift is controlled by cis-regulatory changes in just two anthocyanin-pathway genes:

• F3'5'h (flavonoid 3',5'-hydroxylase) — a branch-point enzyme. Reduced expression blocks the blue (delphinidin) branch, shunting flux toward red (cyanidin/pelargonidin-type) pigments. Result: hue changes blue → red.
• An R2R3-MYB transcription factor — a master regulator of the anthocyanin structural genes. Increased expression raises overall pigment intensity (light → dark).

The dark-red color exploits the floral constancy of the pipevine swallowtail Battus philenor: butterflies that visit a red flower tend to keep visiting red flowers, so they stop shuttling pollen between the two species. This behavioral filter cuts maladaptive interspecific pollen transfer by roughly 50% — a directly measured reinforcing benefit tied to specific mutations.

How reinforcement is studied and detected

Because reinforcement is a process, biologists usually infer it from its diagnostic pattern: reproductive character displacement, in which mating traits and mate discrimination are stronger where two species overlap (sympatry) than where each lives alone (allopatry).

Major lines of evidence include:

• Comparative datasets. Jerry Coyne and H. Allen Orr (1989; expanded 1997) compiled genetic-distance and isolation data across ~171+ Drosophila species pairs. They found that prezygotic isolation was markedly stronger in sympatric pairs than in equally-diverged allopatric pairs — while postzygotic isolation showed no such sympatry effect — the signature predicted if selection reinforces premating barriers.
• Geographic clines. Measuring how a signal (song, color, pheromone) changes along a transect through a contact zone.
• Laboratory selection / mate-choice assays. Rearing populations with enforced selection against hybrids and testing whether assortative mating evolves.
• Molecular population genetics. Detecting selection signatures at loci controlling the reinforced trait, as done for the Phlox flower-color genes.

A key control: alternatives such as differential fusion, ecological character displacement, or biased extinction of overlapping pairs must be ruled out, because they can mimic the same geographic pattern without hybrid-fitness selection.

How reinforcement differs from its close cousins

Reinforcement is often confused with related ideas; the distinctions matter:

• Reinforcement vs. reproductive character displacement (RCD). Reinforcement is the process (selection against hybrids); RCD is the observed pattern (greater trait/preference divergence in sympatry). RCD is strong evidence for reinforcement but can also arise from other kinds of costly interactions.
• Reinforcement vs. the general strengthening of isolation. Not all barrier buildup is reinforcement. If prezygotic isolation simply accumulates with time in allopatry, no selection against hybrids is involved.
• Reinforcement vs. reproductive interference / character displacement in ecology. Ecological character displacement involves competition for resources, not mating costs.
• Reinforcement vs. fusion. When hybrids are fit, gene flow can instead erase the boundary — the opposite outcome.

Historically, the process was proposed by Theodosius Dobzhansky (1937) and named reinforcement by W. Frank Blair (1955). It fell out of favor in the 1980s (theorists Felsenstein and Templeton stressed how gene flow and recombination oppose it), then was rehabilitated in the 1990s–2000s as clear empirical cases like Drosophila and Phlox accumulated.

Significance, applications, and open questions

Reinforcement matters because it is the mechanism most likely to complete speciation in the face of gene flow — it converts leaky, partial isolation into a durable barrier, and it is the one route where selection directly targets reproductive isolation. It helps explain why sister species are often most distinct exactly where they meet, and it links behavior (mate choice), development (pigment/signal genes), and population genetics.

Where it is relevant:

• Conservation and hybridization. Human-driven range shifts and habitat mixing create novel contact zones; whether reinforcement, fusion, or extinction follows affects biodiversity outcomes.
• Agriculture and vectors. Reinforcement-like assortative mating shapes species boundaries in pest and disease-vector complexes (e.g., Anopheles gambiae forms), relevant to gene-drive and control strategies.
• Model systems. Drosophila, Phlox, Timema walking sticks, and heliconiine butterflies remain workhorses.

Open questions remain: How often does reinforcement actually complete speciation versus stall? How commonly does a reinforced trait cascade to isolate conspecific populations (as in D. subquinaria)? And how strong must selection against hybrids be, relative to migration, for choosiness alleles to fix? Quantifying that threshold across taxa is an active frontier.

Reinforcement compared with other ways speciation can be completed after secondary contact

Process	What drives it	Barrier affected	Signature / evidence
Reinforcement	Selection against low-fitness hybrids	Prezygotic (mate choice, timing, signals)	Stronger premating isolation in sympatry than allopatry (reproductive character displacement)
Reproductive character displacement	Pattern produced by reinforcement (or by other conspecific interference)	Mating signals/preferences	Trait divergence greatest where ranges overlap
Reinforcement mimicry / 'cascade'	Reinforced trait incidentally isolates conspecific populations	Prezygotic within-species	Sympatric females also reject allopatric conspecific males (e.g. D. subquinaria)
Fusion (introgressive collapse)	Hybrids fit; gene flow homogenizes	None — species merge	Loss of distinctness in contact zone
Ecological/allopatric completion	Drift + divergent selection without contact	Any, no gene flow needed	Isolation accumulates with time, unrelated to sympatry