19.06.01 · eco-evo-bio / speciation

Speciation — allopatric and sympatric

draft3 tiersLean: nonepending prereqs

Anchor (Master): Coyne & Orr 2004 advanced sections; primary literature — Mayr 1942, Dobzhansky 1937, Orr 1995, Turelli & Orr 2000

Intuition [Beginner]

A species is a group of organisms that can interbreed and produce fertile offspring. Members of the same species share a gene pool — they exchange genes through mating. Members of different species cannot (or do not) interbreed successfully. This is the biological species concept.

Speciation is the process by which one species splits into two. The key requirement is reproductive isolation — something must prevent gene flow between two groups so they evolve independently. Once isolated, the groups accumulate genetic differences through natural selection, genetic drift, and mutation. Eventually they become so different that they can no longer interbreed, even if they meet again.

The most common route is allopatric speciation ("other homeland"): a geographic barrier splits a population into two physically separated groups. A mountain range, a river, or an ocean island can divide a species. With no gene flow between them, each group evolves independently. Given enough time, they become distinct species.

Sympatric speciation ("same homeland") is rarer but real. Here, a new species arises within the same geographic area as the parent species, without a physical barrier. This can happen through ecological specialisation (different food sources or habitats) or through polyploidy — a doubling of chromosome number that immediately creates reproductive isolation, common in plants.

The reason speciation matters: it is the engine of biodiversity. Every species on Earth, from bacteria to blue whales, originated through speciation from an ancestral population. Understanding how one species becomes two is the bridge between microevolution (changes within a species) and macroevolution (the origin of new species and higher taxa).

Visual [Beginner]

Imagine a population of beetles spread across a valley. A new river forms, splitting the population into east and west groups.

On the east side, the environment favours dark-coloured beetles (dark soil). On the west side, light-coloured beetles are favoured (sandy soil). Over thousands of generations, the two populations diverge in colour, size, and mating behaviour. Even if the river dries up and the populations meet again, they may not interbreed — they have become separate species. The diagram shows the three stages: one population, geographic split with independent evolution, and two species upon secondary contact.

Worked example [Beginner]

Darwin's finches on the Galapagos Islands are a textbook case of speciation. The 13 species of finches all descend from a single ancestral species that colonised the islands from mainland South America.

A small group of finches reached one island. Some individuals dispersed to other islands. Because each island had different food sources — large hard seeds, small soft seeds, insects, cactus flowers — natural selection favoured different beak shapes on each island.

On islands with large hard seeds, birds with larger, stronger beaks survived better. On islands with insects, birds with narrow, pointed beaks were favoured. Over many generations, the isolated populations on each island diverged in beak size and shape.

When individuals from different islands occasionally came into contact, they often did not interbreed because their beak differences were associated with different mating songs. The result: 13 species, each specialised for a different food source, all from one common ancestor. This is adaptive radiation — rapid speciation driven by ecological opportunity.

Step 1. One ancestral finch species colonises the Galapagos from South America.

Step 2. Small groups disperse to different islands with different food sources.

Step 3. Natural selection favours different beak shapes on each island; populations diverge over thousands of generations; beak differences produce different songs that prevent interbreeding upon secondary contact.

What this tells us: speciation can proceed rapidly when ecological opportunity (empty niches) and geographic isolation (separate islands) combine to prevent gene flow while selection drives divergence.

Check your understanding [Beginner]

Formal definition [Intermediate+]

The biological species concept

Under the biological species concept (Mayr, 1942), species are groups of actually or potentially interbreeding natural populations that are reproductively isolated from other such groups ^{[Mayr 1942]}. Reproductive isolation (RI) is measured as the proportion of gene flow prevented relative to random mating:

R I = 1 - \frac{H}{C},

where $H$ is the observed number of hybrids and $C$ is the expected number under random mating. $R I = 1$ means complete isolation; $R I = 0$ means no isolation. The RI index quantifies reproductive isolation on a continuous scale from 0 to 1, reflecting the biological reality that speciation is a gradual process, not an instantaneous event.

Pre-zygotic barriers

These prevent hybrid formation:

Temporal isolation: breeding at different times of year or day
Ecological (habitat) isolation: occupying different microhabitats within the same area
Behavioural isolation: different courtship signals, songs, or displays
Mechanical isolation: incompatible reproductive structures
Gametic isolation: sperm-egg incompatibility at the molecular level

Post-zygotic barriers

These reduce hybrid fitness:

Hybrid inviability: $F_{1}$ zygotes fail to develop or survive poorly
Hybrid sterility: $F_{1}$ adults are sterile (Haldane's rule: the heterogametic sex is affected first)
Hybrid breakdown: $F_{2}$ or backcross generations have reduced fitness

The Dobzhansky-Muller model

The Dobzhansky-Muller incompatibility model ^{[Dobzhansky 1937]} ^{[Muller 1940]} explains how reproductive isolation can evolve without either population ever passing through a fitness valley. Consider an ancestral population fixed for alleles $A_{1}$ and $B_{1}$ at two loci. Population 1 fixes $A_{2}$ (substituting for $A_{1}$ ) while population 2 fixes $B_{2}$ (substituting for $B_{1}$ ). Neither substitution is deleterious in its own genetic background. But when the two populations hybridise, the combination $A_{2} B_{2}$ has never been tested by selection and may be incompatible, causing hybrid dysfunction.

Mathematically, let the fitness of genotype $A_{i} B_{j}$ be $w_{ij}$ . The ancestral genotype $A_{1} B_{1}$ has fitness 1. Population 1 evolves $A_{2} B_{1}$ with fitness $w_{21} = 1$ (the substitution is neutral or beneficial). Population 2 evolves $A_{1} B_{2}$ with fitness $w_{12} = 1$ . But the hybrid genotype $A_{2} B_{2}$ has fitness $w_{22} < 1$ — a negative epistatic interaction that was never exposed to selection in either pure population. The model requires exactly two substitutions, one in each lineage, regardless of population size.

Counterexamples to common slips

Reproductive isolation is not all-or-nothing. RI ranges continuously from 0 to 1. Partial isolation ( $0 < R I < 1$ ) is the normal state during speciation; complete isolation is the endpoint, not the starting condition.
Speciation does not require either population to be less fit. The DM model shows that two populations can each fix beneficial or neutral substitutions while the hybrid combination is unfit. No fitness valley is crossed in either pure population.
Geographic isolation is not required for speciation. Sympatric speciation is documented in Rhagoletis fruit flies, cichlid fish, and polyploid plants. The requirement is reproductive isolation, not geographic isolation.

Key theorem with proof [Intermediate+]

Theorem (Orr 1995 — Snowball accumulation of DM incompatibilities). Consider two allopatric populations that split from a common ancestor at time $t = 0$ and evolve independently under a neutral molecular clock with substitution rate $α$ per generation in each population. Let $c$ be the probability that a random pair of derived alleles — one from each population — produces a Dobzhansky-Muller incompatibility. Then the expected number of pairwise incompatibilities between the populations at time $t$ is

E [I (t)] = c α^{2} t^{2} .

The expected number of incompatibilities therefore grows quadratically with time, not linearly.

Proof. Let $K_{1} (t)$ denote the number of substitutions fixed in population 1 by generation $t$ , and $K_{2} (t)$ the number fixed in population 2. Under the neutral molecular clock, substitutions in each population arrive as a Poisson process with rate $α$ per generation. The expected number of substitutions in each lineage after $t$ generations is

E [K_{1} (t)] = α t, E [K_{2} (t)] = α t .

Each of the $K_{1}$ substitutions in population 1 can potentially interact with each of the $K_{2}$ substitutions in population 2 to form a pairwise incompatibility. The total number of possible interacting pairs is $K_{1} (t) \times K_{2} (t)$ . By definition, each pair is incompatible with probability $c$ , independently of other pairs.

By linearity of expectation applied over all pairs:

E [I (t)] = c \cdot E [K_{1} (t) \cdot K_{2} (t)] .

Because the two populations evolve independently (allopatry, no gene flow), $K_{1} (t)$ and $K_{2} (t)$ are independent random variables. Independence of Poisson processes gives $E [K_{1} K_{2}] = E [K_{1}] \cdot E [K_{2}]$ , so

E [I (t)] = c \cdot E [K_{1} (t)] \cdot E [K_{2} (t)] = c \cdot (α t) (α t) = c α^{2} t^{2} .

The quadratic dependence on $t$ means that reproductive isolation accumulates slowly at first but accelerates: each new substitution in one population creates incompatibility opportunities with all substitutions already fixed in the other. $□$

This result is the theoretical foundation of the snowball theory ^{[Orr 1995]}. It makes a testable prediction: the strength of reproductive isolation between two species should increase faster than linearly with genetic distance. Data from Drosophila species pairs broadly support this pattern — the fraction of inviable or sterile $F_{1}$ hybrids increases more steeply between distantly related pairs than between closely related ones.

Bridge. The snowball effect is the foundational reason that speciation is inevitable given sufficient allopatric time: even under neutrality, incompatibilities accumulate without bound. This is exactly the mechanism that connects the microevolutionary process of allele substitution to the macroevolutionary pattern of species divergence. The quadratic accumulation rate builds toward 19.07.01 phylogenetics, where the timing of speciation events is inferred from genetic distances that themselves reflect the $t^{2}$ incompatibility clock. The central insight — that incompatibilities grow combinatorially, not additively — appears again in 19.05.01 pending quantitative genetics when predicting how rapidly multilocus trait divergence generates reproductive barriers between populations.

Exercises [Intermediate+]

Exercise 3 (medium, numeric).

A diploid plant species has $2 n = 14$ chromosomes. A mitotic error produces a tetraploid individual with $4 n = 28$ . If this tetraploid mates with a normal diploid, what is the chromosome number of the resulting hybrid? Is the hybrid expected to be fertile?

Hint

The tetraploid produces gametes with $2 n = 14$ chromosomes. The diploid produces gametes with $n = 7$ chromosomes.

Answer

The hybrid has $3 n = 21$ chromosomes and is expected to be sterile. The tetraploid produces $2 n = 14$ gametes and the diploid produces $n = 7$ gametes, giving a triploid ( $3 n = 21$ ) hybrid. Triploids are sterile because at meiosis the three homologous chromosomes cannot form balanced pairs — one set of seven chromosomes is unpaired, producing aneuploid gametes. This is why polyploidy causes instantaneous reproductive isolation in a single generation.

Exercise 4 (medium, multiple choice).

An ancestral population is fixed for alleles $A_{1}$ at locus A and $B_{1}$ at locus B. After geographic isolation, Population 1 fixes $A_{2}$ while Population 2 fixes $B_{2}$ . Hybrids carry genotype $A_{2} B_{2}$ and have reduced fitness. This is an example of:

A. Heterosis (hybrid vigour) B. A Dobzhansky-Muller incompatibility C. Directional selection D. Frequency-dependent selection

Hint

Neither $A_{2}$ nor $B_{2}$ is individually deleterious. The problem arises only in the combination.

Answer

B. This is the canonical Dobzhansky-Muller incompatibility. Neither population passed through a fitness valley — $A_{2}$ is neutral in the $B_{1}$ background and $B_{2}$ is neutral in the $A_{1}$ background. The incompatibility appears only in the hybrid combination $A_{2} B_{2}$ , which natural selection never tested in either pure population.

Exercise 6 (medium, short answer).

Explain how reinforcement strengthens pre-zygotic isolation when two partially diverged species come into secondary contact.

Hint

If hybrids have low fitness, compare the reproductive success of individuals that mate with their own species versus those that mate with the other species.

Answer

When two species meet after a period of allopatric divergence, individuals that mate with their own species produce viable offspring, while those that mate with the other species produce low-fitness hybrids. Natural selection favours traits that prevent hybridisation — more divergent mating signals, greater mate discrimination, or different breeding times. Over generations, pre-zygotic isolation strengthens as a direct result of selection against hybridisation. This predicts that sympatric populations of two species should show stronger pre-zygotic isolation than allopatric populations of the same pair — confirmed in Drosophila and tree frogs.

Exercise 7 (hard, symbolic).

The snowball model predicts $E [I (t)] = c α^{2} t^{2}$ . Suppose that incompatibilities involving three loci (rather than two) arise with probability $d$ per triplet of derived alleles. Derive the expected number of three-way incompatibilities as a function of $t$ , and state whether it grows faster or slower than the quadratic accumulation of pairwise incompatibilities.

Hint

A three-way incompatibility requires one derived allele from population 1 at each of two loci, and one from population 2 at a third locus — or any partition of three loci across the two populations. For the simplest case where all three loci diverge independently, count the number of triplets.

Answer

If each population has fixed $α t$ substitutions at independent loci, the number of triplets drawn from the combined set of $2 α t$ substitutions is $(3 2 α t)$ , but a biologically relevant three-way incompatibility typically requires that at least one derived allele comes from each population. The number of triplets with at least one from each population is $K_{1} \cdot K_{2} \cdot (K_{1} + K_{2} - 2) + K_{1}^{2} \cdot K_{2} + K_{1} \cdot K_{2}^{2} - 3 K_{1} K_{2}$ in general. Taking expectations: $E [triplets] \sim O ((α t)^{3}) = O (t^{3})$ . The expected number of three-way incompatibilities is $d \cdot O (t^{3})$ , which grows faster than the $t^{2}$ accumulation of pairwise incompatibilities. Orr and Turelli (2001) showed that higher-order incompatibilities accumulate at rate $t^{n}$ for $n$ -way interactions, meaning the snowball effect becomes even more pronounced for complex epistatic interactions.

Exercise 8 (hard, short answer).

Haldane's rule states that if one sex is absent, rare, or sterile in $F_{1}$ hybrids, it is the heterogametic sex (XY males in mammals, ZW females in birds). Explain how the dominance theory extends the DM model to account for this pattern.

Hint

Consider that recessive incompatible alleles on the X chromosome are fully expressed in XY males but masked by the dominant allele in XX females.

Answer

Under the DM model, many incompatible substitutions are partially recessive. In the homogametic sex (XX females in mammals), a recessive incompatible allele on one X chromosome is masked by the dominant compatible allele from the other species' X. In the heterogametic sex (XY males), the single X chromosome is expressed hemizygously — any incompatible recessive allele is fully exposed without a masking partner. The result: the heterogametic sex suffers disproportionate hybrid dysfunction. Turelli and Orr (1995) formalised this as the dominance theory ^{[Turelli & Orr 1995]}, showing that the large-X effect (disproportionate contribution of the X chromosome to reproductive isolation) and Haldane's rule both follow from the recessivity of DM incompatibilities. The theory predicts that X-linked loci contribute more to hybrid incompatibility per locus than autosomes — confirmed in Drosophila where the X chromosome carries roughly half of all identified speciation genes despite representing only about 20% of the genome.

Dobzhansky-Muller incompatibilities — the genetic architecture of speciation [Master]

The Dobzhansky-Muller model provides the theoretical backbone of speciation genetics. Its power lies in a simple combinatorial observation: when two populations fix different alleles at different loci, the derived combination in hybrids has never been screened by natural selection. The model requires no fitness valley in either parental population — each substitution can be neutral or beneficial within its own genetic background. What makes the hybrid unfit is the epistatic interaction between loci that diverged in different populations.

The mathematical framework is as follows. An ancestral population is fixed for alleles $a$ at locus 1 and $b$ at locus 2, with fitness $w (ab) = 1$ . Population 1 fixes derived allele $A$ (substituting for $a$ ), giving genotype $A b$ with fitness $w (A b) \geq 1$ . Population 2 fixes derived allele $B$ (substituting for $b$ ), giving genotype $a B$ with fitness $w (a B) \geq 1$ . The hybrid genotype $A B$ has fitness $w (A B) < 1$ — a negative epistatic interaction. The fitness decomposition is:

w (A B) = w (ab) + Δ w_{A} + Δ w_{B} + ϵ_{A B},

where $Δ w_{A}$ and $Δ w_{B}$ are the fitness effects of the individual substitutions (both $\geq 0$ in their own backgrounds) and $ϵ_{A B} < 0$ is the epistatic term — the DM incompatibility. The key insight is that $ϵ_{A B}$ is invisible to selection within either pure population because the combination $A B$ never arises there.

Proposition (Two-locus DM incompatibility is sufficient). Given any two allopatric populations each fixing one derived allele, a single negative epistatic interaction $ϵ_{A B} < 0$ is sufficient for partial post-zygotic isolation. No additional substitutions are required.

Proof. Under the fitness decomposition above, the fitness of $F_{1}$ hybrids (genotype $A a B b$ ) is determined by the dominance coefficients of $A$ and $B$ and the strength of the epistatic interaction. If $ϵ_{A B}$ is fully recessive in each background (i.e., it only reduces fitness in the double-homozygote $AA B B$ ), then $F_{1}$ hybrids ( $A a B b$ ) have normal fitness but $F_{2}$ segregants include the $AA B B$ class at frequency $1/16$ , producing hybrid breakdown. If $ϵ_{A B}$ is partially dominant, $F_{1}$ hybrids themselves suffer reduced fitness. In both cases, post-zygotic isolation is present without any fitness reduction in either parental population. $□$

Identified speciation genes

The DM model made concrete predictions about the genetic basis of speciation: there should exist specific loci whose derived alleles interact negatively when combined across species. Molecular genetics has identified several such speciation genes.

In Drosophila, the genes Hmr (Hybrid male rescue) and Lhr (Lethal hybrid rescue) were identified through genetic mapping of hybrid incompatibility between D. melanogaster and D. simulans. Hmr is a DNA-binding protein involved in heterochromatin regulation; Lhr is a heterochromatin protein. The D. melanogaster allele of Hmr combined with the D. simulans allele of Lhr causes hybrid male lethality via misregulation of heterochromatic repeats — a molecular confirmation of the DM model at the level of protein-protein interactions. The OdsH (Odysseus-related homeobox) gene causes hybrid male sterility between D. simulans and D. mauritiana through misexpression in the testes, driven by positive selection on DNA-binding specificity in the simulans lineage.

In mice, the Prdm9 gene (a meiotic recombination hotspot specifier) is the strongest-acting hybrid sterility gene known. Prdm9 alleles from different subspecies of house mouse (Mus musculus domesticus and M. m. musculus) interact to cause asymmetric failure of meiotic synapsis in $F_{1}$ males. The mechanism: Prdm9 determines where double-strand breaks form during meiosis; when two subspecies' alleles specify incompatible breakpoint sets, chromosomes fail to pair, and spermatogenesis arrests. Prdm9 is the first speciation gene shown to operate through a trans-acting DNA-binding mechanism, and it evolves under some of the fastest positive selection rates known in mammalian genomes (amino-acid substitution rate in the zinc-finger domain exceeds the genomic average by a factor of 20).

The snowball effect and empirical tests

Orr's 1995 snowball prediction — $E [I (t)] \propto t^{2}$ — generates a testable null hypothesis: the number of DM incompatibilities between two species should be proportional to the square of their genetic distance, not to the distance itself. Testing this requires comparing reproductive isolation between multiple species pairs of known genetic divergence.

Coyne and Orr's 1989 and 1997 compilations of Drosophila species pairs provided the first broad test. They scored pre-zygotic isolation (mate discrimination) and post-zygotic isolation (hybrid viability and fertility) for over 100 species pairs with estimated genetic distances from allozyme and DNA sequence data. The results showed: (a) post-zygotic isolation increases faster than linearly with genetic distance, consistent with the snowball prediction; (b) pre-zygotic isolation accumulates more rapidly than post-zygotic isolation in sympatric pairs, consistent with reinforcement; (c) total reproductive isolation reaches near-completeness ( $R I > 0.95$ ) at genetic distances corresponding to roughly 10-20 million years of divergence in Drosophila.

Matute et al. (2010 Evolution 64, 3164-3179) tested the snowball prediction directly by measuring hybrid sterility and inviability between three pairs of Drosophila species with different divergence times. The number of incompatibilities increased approximately as $D^{2}$ (where $D$ is Nei's genetic distance), supporting the quadratic accumulation predicted by Orr's model. However, Moehring et al. (2004 Genetics 168, 947-953) found that the number of loci contributing to hybrid sterility increases approximately linearly, not quadratically, suggesting that the snowball may apply to the severity of incompatibilities rather than to the number of contributing loci — a distinction that remains actively debated.

Allopatric speciation — vicariance, peripatry, and reinforcement [Master]

Allopatric speciation is the geographically simplest and historically best-documented mode. A physical barrier — mountain uplift, river formation, sea-level change, continental drift — splits a continuous population into two or more isolates that evolve independently. Mayr (1942) ^{[Mayr 1942]} championed allopatry as the dominant mode of speciation in animals, based on his studies of South Pacific birds. While Mayr's insistence on geographic isolation as necessary for speciation was overstated (sympatric speciation is now well documented), allopatry remains the most common route.

Vicariance: the Panama isthmus test case

Vicariance — the splitting of a continuous population by a newly formed geographic barrier — is the classic allopatric mode. The formation of the Isthmus of Panama approximately 3 million years ago is the most thoroughly studied vicariant event in evolutionary biology. Before the isthmus closed, marine populations were continuous between the Pacific and Atlantic (Caribbean) coasts of Central America. When the land bridge formed, populations of shrimp, fish, and other marine organisms were split into Pacific and Caribbean isolates.

Knowlton et al. (1993 Evolution 47, 1374-1387) studied 15 pairs of snapping shrimp (Alpheus) species — one Pacific, one Caribbean — that are morphological and genetic "snapshots" of the speciation process triggered by isthmus closure. The 15 pairs differ in genetic distance, reflecting different divergence times and possibly different colonisation histories. Key findings: (a) all 15 pairs show strong reproductive isolation in laboratory mating trials — Pacific and Caribbean members of each pair do not interbreed; (b) the strength of pre-zygotic isolation (mate discrimination) correlates with genetic distance; (c) post-zygotic isolation (hybrid viability) is essentially complete for all pairs, regardless of genetic distance — once the isthmus closed and gene flow stopped, hybrid inviability accumulated rapidly.

The Panama system provides a natural experiment: the barrier formed at a known geological time, creating replicated speciation events across multiple taxa. Molecular clock estimates place the divergence of most shrimp pairs at 3-10 million years, consistent with isthmus closure. The variation in genetic distance among pairs likely reflects different effective population sizes and selection regimes.

Peripatric speciation and founder effects

Peripatric speciation occurs when a small group of individuals colonises a new, isolated habitat at the periphery of the parent species' range. The founding population carries only a subset of the parental gene pool — a founder effect — and genetic drift in the small population can rapidly change allele frequencies, including fixing alleles that would be selected against in the larger parental population.

Mayr's (1954) model of genetic revolution proposed that founder effects could trigger a cascade of genetic changes that reorganise the gene pool, rapidly producing reproductive isolation. The model was influential but remains controversial. Templeton (2008 Evolution 62, 1555-1559) argued that founder-effect speciation is theoretically unlikely because small populations more commonly go extinct than speciate. However, empirical cases support the peripatric route: the Drosophila of Hawaii (hundreds of species on young volcanic islands, each founded by a small number of colonists), and the island foxes of the California Channel Islands (dwarfed relative to mainland grey foxes, with reproductive isolation following rapid body-size evolution).

The population-genetics of peripatric speciation involve three interacting forces: (a) drift in the small founding population, which can fix alleles regardless of fitness; (b) selection in the novel environment, which favours different alleles than in the parental range; and (c) reduced gene flow due to geographic isolation. When the founding population is very small ( $N_{e} < 50$ ), drift dominates and the population essentially restarts its genetic architecture from a random subset of the parental variation.

Reinforcement upon secondary contact

When two allopatrically diverged populations come back into geographic contact (secondary contact), two outcomes are possible: fusion back into a single population (if reproductive isolation is weak) or completion of speciation. Reinforcement — the strengthening of pre-zygotic isolation by natural selection against hybridisation — is the mechanism by which secondary contact can complete rather than reverse speciation.

The logic of reinforcement is as follows. If hybrids have reduced fitness, then individuals that mate with their own species produce more viable offspring than those that mate heterospecifically. Natural selection therefore favours alleles that increase assortative mating — traits that reduce the probability of hybrid matings. Over generations, pre-zygotic barriers strengthen, completing reproductive isolation.

Reinforcement makes a specific geographic prediction: sympatric populations of two species (those that co-occur and risk hybridisation) should show stronger pre-zygotic isolation than allopatric populations of the same pair (those that never meet and face no selection against hybridisation). This pattern — called reproductive character displacement — has been confirmed in multiple taxa. Hoskin et al. (2005 Nature 437, 1353-1356) documented reproductive character displacement in Australian rainforest frogs: sympatric populations of Litoria species show greater divergence in mating calls than allopatric populations, and the degree of call divergence correlates with the risk of hybridisation. Servedio and Noor (2003 Annu. Rev. Ecol. Evol. Syst. 34, 339-364) reviewed evidence across taxa and found that reinforcement is widespread but operates primarily when post-zygotic isolation is already moderate ( $R I \approx 0.5 - 0.8$ ): too little post-zygotic isolation and selection against hybrids is weak; too much and hybridisation is already rare, removing the selective pressure for stronger pre-zygotic barriers.

Sympatric and parapatric speciation — ecological divergence without geographic isolation [Master]

Sympatric speciation — the origin of reproductive isolation within a single geographic area — was considered theoretically implausible for much of the twentieth century. Mayr (1942) argued that gene flow in sympatry would prevent divergence unless geographic isolation intervened. The theoretical objection is sound: in sympatry, random mating continually homogenises gene pools, and any allele promoting assortative mating is diluted unless it is directly favoured by selection. However, three mechanisms have been documented to produce sympatric speciation in nature: host-shift speciation, polyploid speciation, and sexual-selection-driven divergence.

Host-shift speciation: Rhagoletis pomonella

The apple maggot fly Rhagoletis pomonella is the most thoroughly documented case of sympatric ecological speciation. The ancestral host of R. pomonella is the hawthorn (Crataegus spp.), native to North America. When apple trees were introduced to North America by European colonists in the 1800s, a subset of the hawthorn fly population shifted onto apples as a new host.

The host shift created two partially isolated host races: hawthorn flies and apple flies. Several forms of reproductive isolation accompany the shift. (a) Temporal isolation: apples fruit earlier than hawthorns, and flies emerge as adults timed to their host's fruiting phenology. Apple flies emerge 2-3 weeks earlier than hawthorn flies, reducing inter-race mating opportunities. (b) Habitat isolation: adults mate on or near the host fruit, so flies on different hosts encounter different potential mates. (c) Allochronic isolation: the different emergence times shift the entire mating season. Feder, Egan and Nosil (2012 J. Chem. Ecol. 38, 1003-1016) estimated that temporal and habitat isolation together reduce gene flow between the races by approximately 94%.

Genome-wide SNP data confirm that divergence between host races is concentrated in three genomic regions — on chromosomes 2, 3, and 4 — associated with host choice, diapause timing, and olfactory preference. The rest of the genome shows minimal differentiation ( $F_{S T} < 0.02$ ), consistent with ongoing gene flow outside the barrier loci. This heterogeneous genomic pattern is the hallmark of speciation with gene flow: a few loci under strong divergent selection maintain differentiation while the remainder of the genome homogenises.

Adaptive radiation: Lake Malawi cichlids

The cichlid fish of Lake Malawi represent the most spectacular sympatric radiation known. Approximately 500-1000 species have evolved from a common ancestor within the last 1-2 million years, with no geographic barriers within the lake. The radiation is driven by a combination of sexual selection and ecological specialisation.

Male cichlids display species-specific nuptial colour patterns — bright blues, yellows, reds, and oranges — that females use to identify conspecific mates. Female mate choice is based primarily on colour recognition, and colour is genetically determined by a small number of loci (primarily opsin genes for visual sensitivity and pigment-pathway genes for male coloration). Seehausen et al. (1999 Science 286, 1453-1455) showed that female Pundamilia nyererei and P. pundamilia — two sympatric cichlid species in Lake Victoria that differ in male colour (blue vs red) — interbreed readily under monochromatic light (where colour differences are invisible) but not under full-spectrum light. A single light manipulation eliminates assortative mating, demonstrating that colour-based mate choice alone maintains reproductive isolation between these species.

The sensory drive hypothesis (Endler 1992 Am. Nat. 139, S125-S153) explains how the underwater light environment drives divergence in both male colour and female visual sensitivity. At different depths in Lake Malawi, water filters different wavelengths: shallow waters transmit full-spectrum light; deeper waters are dominated by blue light. Species adapted to different depths evolve different visual sensitivities (via opsin gene expression shifts) and different male colour patterns (to maximise conspicuousness at their native depth). Depth-correlated divergence in both visual systems and male display creates assortative mating along a depth gradient — a form of ecological speciation that operates within a single lake.

Polyploid speciation: instantaneous isolation

Polyploid speciation is the only mechanism that produces reproductive isolation in a single generation. A polyploid individual arises when a mitotic or meiotic error doubles the chromosome set, producing a tetraploid ( $4 n$ ) from a diploid ( $2 n$ ) parent. When the tetraploid mates with a diploid, the hybrid is triploid ( $3 n$ ) and typically sterile because meiosis cannot properly segregate three sets of chromosomes — the univalent chromosomes fail to form balanced bivalents at metaphase I.

Polyploid speciation is common in flowering plants. Soltis et al. (2004 Trends Ecol. Evol. 19, 485-491) estimated that roughly 15% of angiosperm speciation events involve polyploidy, and that all flowering plants have experienced at least one round of ancient polyploidy (whole-genome duplication) in their evolutionary history. The genus Tragopogon (goatsbeard) provides a replicated natural experiment: tetraploid species T. mirus and T. miscellus have independently formed multiple times in the British Isles and New Zealand from diploid progenitors within the past 80 years, demonstrating that polyploid speciation is both rapid and repeatable.

Proposition (Polyploid speciation in one generation). An autotetraploid arising from a diploid population with $2 n$ chromosomes is reproductively isolated from the diploid parent by hybrid sterility: crosses produce sterile triploids with $3 n$ chromosomes.

Proof. The diploid produces gametes with $n$ chromosomes. The tetraploid produces gametes with $2 n$ chromosomes. The hybrid zygote has $3 n$ chromosomes. At meiosis in the triploid, three homologous chromosomes attempt to pair. Only two can form a bivalent; the third remains as a univalent. Because chromosome pairing is stochastic, the resulting gametes receive random numbers of each chromosome ( $0, 1,$ or $2$ copies) rather than the balanced $1$ of each. The probability of producing a balanced gamete with exactly $n$ chromosomes is $(1/2)^{n}$ , which becomes vanishingly small for biologically realistic chromosome numbers (e.g., $n = 7$ gives $1/128 \approx 0.8%$ ). Fertility is therefore effectively zero. $□$

Species concepts and the speciation continuum [Master]

The biological species concept (BSC) is the most widely used definition in speciation research, but it is not the only one, and it fails for several important categories of organisms. Understanding the alternatives — and why they matter — is essential for making sense of speciation as a real biological process rather than a taxonomic convention.

Alternative species concepts

The phylogenetic species concept (PSC) defines a species as the smallest monophyletic group on a phylogenetic tree — the smallest cluster of organisms that share a unique common ancestor not shared with any other group. The PSC does not require reproductive isolation; it requires only that the group be diagnosable by shared derived characters (synapomorphies). The advantage: the PSC applies to asexual organisms and fossils, where the BSC is inapplicable. The disadvantage: the PSC tends to split more finely than the BSC, recognising many "species" that interbreed freely, and the criterion of monophyly depends on which characters are examined and how many.

The ecological species concept (Van Valen 1976 Evol. Theory 1, 27-44) defines a species as a lineage occupying a unique adaptive zone — a distinctive ecological niche that maintains the lineage's identity through stabilising selection. Under this view, a species is maintained not by reproductive isolation per se but by selection against intermediate forms that fall between adaptive peaks. This concept is particularly relevant for sympatric speciation, where ecological divergence creates the reproductive barrier.

The genotypic cluster concept (Mallet 1995 Trends Ecol. Evol. 10, 294-299) defines species as groups that form distinct clusters in multivariate genetic or phenotypic space, separated by gaps that contain few or no individuals. This concept is operational — it can be applied to any dataset with genetic or morphological measurements — and it naturally accommodates hybrid zones, where the clusters are partially merged.

Genomic islands of divergence

Modern genomic data reveal that speciation is rarely an all-or-nothing process at the molecular level. Whole-genome sequencing of incipient species pairs shows genomic islands of divergence — regions of elevated differentiation ( $F_{S T} > 0.5$ ) interspersed with regions of low differentiation ( $F_{S T} < 0.05$ ) where gene flow continues. This heterogeneous pattern indicates that speciation proceeds via divergence at a subset of loci under strong divergent selection, while the rest of the genome homogenises through gene flow.

The speciation-with-gene-flow model predicts that loci under strong selection ( $s ≫ m$ , where $s$ is the selection coefficient and $m$ is the migration rate) can diverge even while neutral loci remain undifferentiated. The width of the genomic island around a selected locus scales as $σ / s$ , where $σ$ is the dispersal distance, reflecting the balance between selection (which maintains divergence) and recombination with migrant genomes (which erodes it). Charlesworth et al. (1997 Genet. Res. 69, 135-153) derived the expected cline width for a locus under selection in a hybrid zone.

Gene flow during speciation: Heliconius butterflies

Heliconius butterflies provide the clearest example of gene flow during speciation. H. melpomene and H. cydno are sympatric species in Central and South America that share Müllerian mimicry patterns — both display warning coloration signalling toxicity to predators. The two species hybridise at low frequency in nature ( $< 1%$ of individuals in most populations) and produce viable, partially fertile $F_{1}$ hybrids.

Genome-wide data reveal a striking pattern: the two species are undifferentiated across most of the genome (mean $F_{S T} \approx 0.03$ ), but a handful of genomic regions show near-complete divergence ( $F_{S T} > 0.8$ ). These islands contain the genes controlling wing colour pattern — the primary trait involved in mate choice and predator avoidance. The optix gene controls red pattern elements, and the WntA gene controls black pattern elements; both are located in divergent genomic islands. Heliconius females preferentially mate with males displaying the same wing pattern as themselves, creating strong assortative mating based on colour. The genomic islands are maintained by strong divergent selection on colour pattern (both ecological selection via mimicry and sexual selection via mate choice), while gene flow homogenises the rest of the genome.

Martin et al. (2019 Science 366, 735-739) showed that introgression — gene flow between species — has transferred adaptive colour-pattern alleles between Heliconius species repeatedly. A single haplotype at the optix locus has moved between H. melpomene, H. timareta, and H. elevatus multiple times, allowing each species to acquire new red wing patterns without evolving them independently. Adaptive introgression of colour-pattern alleles between species is a process that blurs the species boundary at the genomic level while maintaining it at the phenotypic level.

Ring species

A ring species is a chain of intergrading populations that forms a ring around a geographic barrier. Adjacent populations in the chain can interbreed, but the two end populations — which meet on the other side of the barrier — are reproductively isolated. Ring species provide a spatial snapshot of speciation in action, demonstrating that speciation can occur through the gradual accumulation of small changes without ever requiring a discrete speciation event.

The classic example is the greenish warbler (Phylloscopus trochiloides) ring around the Tibetan Plateau. Populations spread northward from the southern Himalayas along two routes — one westward through Central Asia, one eastward through China — meeting in Siberia. Adjacent populations along each route interbreed, but the two terminal populations in Siberia show strong reproductive isolation (different songs, minimal interbreeding). Irwin et al. (2001 Science 292, 2335-2336) demonstrated that song complexity increases gradually along both branches of the ring, driven by sexual selection, and the divergence in song between the terminal populations is sufficient to prevent interbreeding. The ring species demonstrates that speciation can be a purely continuous process — no discrete transition from "same species" to "different species" is required.

The Ensatina salamander ring in California is a second well-studied case. The species complex forms a ring around the Central Valley, with gradual morphological and genetic change around the ring. The southern contact zone between the eastern and western forms shows strong assortative mating and hybrid dysfunction. However, Wake and Schneider (1998 Int. J. Plant Sci. 159, S209-S219) noted that the ring is not perfectly continuous — some populations within the ring show sharp transitions, suggesting that geographic isolation has contributed to divergence in addition to the continuous chain.

Synthesis. The foundational reason that speciation is a gradual, quantitative process rather than a discrete event is that reproductive isolation accumulates through the combinatorial interaction of independently fixed substitutions, as the Dobzhansky-Muller model predicts. The central insight is that speciation occupies a continuum: populations diverge from $R I = 0$ to $R I = 1$ through the progressive accumulation of pre-zygotic and post-zygotic barriers, and partial isolation at intermediate stages is the normal condition. This is exactly what genomic islands of divergence reveal — a handful of barrier loci under strong selection embedded in a genomic background that remains undifferentiated through gene flow. Putting these together with the ecological mechanisms of divergence (host shifts in Rhagoletis, sensory drive in cichlids, mimicry in Heliconius) gives a unified picture: speciation is the process by which selection, drift, and recombination sort variation at barrier loci while gene flow continues elsewhere in the genome.

The bridge is between microevolutionary processes (allele frequency change at individual loci) and macroevolutionary patterns (the origin of reproductively isolated lineages), and the pattern recurs across every well-studied speciation system — from Panama shrimp to Galapagos finches to Lake Malawi cichlids. This continuum view generalises beyond any single species concept: whether one applies the biological, phylogenetic, ecological, or genotypic cluster concept, the underlying process is the same — the gradual accumulation and strengthening of reproductive barriers through the DM mechanism and its ecological amplifiers. The speciation continuum builds toward 19.07.01 phylogenetics, where the endpoint of complete isolation is the branching event that phylogenetic methods reconstruct.

Connections [Master]

Natural selection 19.03.01 pending. Divergent selection in different environments drives the phenotypic divergence between incipient species. Ecological speciation — where adaptation to different niches produces reproductive isolation as a byproduct — is the primary route connecting microevolutionary selection to macroevolutionary species divergence. The selection unit provides the mechanistic framework that this speciation unit invokes as the engine of divergence.
Genetic drift 19.04.01. Drift fixes incompatible alleles in the Dobzhansky-Muller model, particularly in small peripatric populations. The neutral molecular clock assumed by the snowball theorem is a drift-dominated substitution process. The drift unit formalises the stochastic process that the speciation unit treats as the source of raw incompatibility material.
Sexual selection 19.03.02. Female mate choice on male colour pattern drives sympatric divergence in cichlids and Heliconius butterflies. Sexual selection amplifies assortative mating without requiring geographic isolation, making it the key mechanism by which sympatric speciation overcomes gene flow. This unit's treatment of sensory drive and colour-based isolation directly extends the sexual-selection framework.
Mendelian genetics 19.01.01 pending. The allelic framework for DM incompatibilities — two loci, two derived alleles, epistatic fitness interaction — is Mendelian in structure. The identified speciation genes (Hmr, Lhr, Prdm9, OdsH) obey Mendelian inheritance with dominance and epistasis. The Mendelian unit provides the prerequisite genetic vocabulary.
Quantitative genetics 19.05.01 pending. The breeder's equation predicts how rapidly heritable ecological traits diverge under selection between isolated populations. Multivariate extensions predict the correlated evolution of ecological and mating traits that characterises ecological speciation.
Phylogenetics 19.07.01. Speciation events generate the branching structure that phylogenetic methods reconstruct. The timing and mode of speciation (allopatric vs sympatric, rapid vs gradual) leave detectable signatures in phylogenetic tree shape. The speciation continuum builds toward phylogenetic divergence as its macroevolutionary endpoint.

Historical & philosophical context [Master]

Charles Darwin titled his most famous work On the Origin of Species by Means of Natural Selection (1859), yet he struggled to explain how one species splits into two. He focused on gradual divergence through natural selection but did not articulate the role of geographic isolation. His treatment of speciation was largely associative — he demonstrated that varieties could diverge under selection and inferred by extrapolation that species must arise similarly, but he lacked a mechanistic model for how reproductive isolation evolves.

Theodosius Dobzhansky's Genetics and the Origin of Species (1937) ^{[Dobzhansky 1937]} provided the genetic framework. Dobzhansky formulated the concept of reproductive isolation as the defining feature of species and, independently with H. J. Muller, developed the two-locus incompatibility model showing that speciation requires no fitness valley in either parental population. Muller's 1940 chapter in Huxley's The New Systematics ^{[Muller 1940]} laid out the formal argument: with two loci, two substitutions, and epistatic fitness interactions, hybrid dysfunction arises inevitably without either lineage ever being less fit.

Ernst Mayr's Systematics and the Origin of Species (1942) ^{[Mayr 1942]} introduced the biological species concept in its modern form and championed allopatric speciation as the dominant mode. Mayr's fieldwork on South Pacific birds — where island populations showed consistent morphological divergence — convinced him that geographic isolation was necessary for speciation. His influence delayed recognition of sympatric speciation as a legitimate process for several decades.

The quantitative theory of speciation advanced with Orr's 1995 snowball paper ^{[Orr 1995]}, which derived the quadratic accumulation rate of DM incompatibilities under a neutral clock and generated testable predictions about the relationship between genetic distance and reproductive isolation. Coyne and Orr's 2004 monograph Speciation ^{[Coyne & Orr 2004]} synthesised the field, reviewing evidence from genetics, ecology, biogeography, and genomics. The genomic era (2005-present) has revealed the fine-scale architecture of speciation: genomic islands of divergence, adaptive introgression between species, and the identification of specific speciation genes whose molecular functions confirm the DM model at the protein-interaction level.

The species problem — the difficulty of defining what a species is — remains open. The BSC fails for asexual organisms, for fossils, and for partially interbreeding species complexes. Alternative concepts (phylogenetic, ecological, genotypic cluster) each work better for some taxa. The speciation-continuum view suggests that "species" may be a label applied to a point on a continuous process rather than a discrete natural kind — a position with implications for conservation law, taxonomy, and the definition of biodiversity.

Bibliography [Master]

@book{Darwin1859,
  author = {Darwin, Charles},
  title = {On the Origin of Species by Means of Natural Selection},
  publisher = {John Murray},
  year = {1859},
}

@book{Dobzhansky1937,
  author = {Dobzhansky, Theodosius},
  title = {Genetics and the Origin of Species},
  publisher = {Columbia University Press},
  year = {1937},
}

@incollection{Muller1940,
  author = {Muller, Hermann J.},
  title = {Bearing of the Drosophila Work on Systematics},
  booktitle = {The New Systematics},
  editor = {Huxley, Julian S.},
  publisher = {Clarendon Press},
  year = {1940},
  pages = {185--268},
}

@book{Mayr1942,
  author = {Mayr, Ernst},
  title = {Systematics and the Origin of Species},
  publisher = {Columbia University Press},
  year = {1942},
}

@article{Orr1995,
  author = {Orr, H. Allen},
  title = {The Population Genetics of Speciation: The Evolution of Hybrid Incompatibilities},
  journal = {Genetics},
  volume = {139},
  year = {1995},
  pages = {1805--1813},
}

@article{TurelliOrr1995,
  author = {Turelli, Michael and Orr, H. Allen},
  title = {The Dominance Theory of Haldane's Rule},
  journal = {Genetics},
  volume = {140},
  year = {1995},
  pages = {389--402},
}

@book{CoyneOrr2004,
  author = {Coyne, Jerry A. and Orr, H. Allen},
  title = {Speciation},
  publisher = {Sinauer Associates},
  year = {2004},
}

@book{GrantGrant2008,
  author = {Grant, Peter R. and Grant, B. Rosemary},
  title = {How and Why Species Multiply},
  publisher = {Princeton University Press},
  year = {2008},
}

@article{Knowlton1993,
  author = {Knowlton, Nancy and Weigt, Lee A. and Sol\'is, Anibal and Mills, D. Kent and Bermingham, Eldredge},
  title = {Divergence in Proteins, Mitochondrial {DNA}, and Reproductive Compatibility Across the {Isthmus of Panama}},
  journal = {Evolution},
  volume = {47},
  year = {1993},
  pages = {1374--1387},
}

@article{Seehausen1999,
  author = {Seehausen, Ole and van Alphen, Jacques J. M. and Witte, Frans},
  title = {Can Ancient Colour Polymorphisms Explain Why Some Cichlid Lineages Diversify Faster Than Others?},
  journal = {Science},
  volume = {286},
  year = {1999},
  pages = {1453--1455},
}

@article{Feder2012,
  author = {Feder, Jeffrey L. and Egan, Scott P. and Nosil, Patrik},
  title = {The Genomics of Speciation-with-Gene-Flow},
  journal = {Trends in Genetics},
  volume = {28},
  year = {2012},
  pages = {342--350},
}

@article{Martin2019,
  author = {Martin, Simon H. and Davey, John W. and Salazar, Camilo and Jiggins, Chris D.},
  title = {Recombination Rate Variation Shapes Barriers to Introgression Across Butterfly Genomes},
  journal = {Science},
  volume = {366},
  year = {2019},
  pages = {735--739},
}

@article{Irwin2001,
  author = {Irwin, Darren E. and Irwin, Jessica H. and Price, Trevor D.},
  title = {Ring Species as Bridges Between Microevolution and Speciation},
  journal = {Genetica},
  volume = {112--113},
  year = {2001},
  pages = {223--243},
}

@article{VanValen1976,
  author = {Van Valen, Leigh},
  title = {Ecological Species, Multispecies, and Oaks},
  journal = {Taxon},
  volume = {25},
  year = {1976},
  pages = {233--239},
}

@article{Mallet1995,
  author = {Mallet, James},
  title = {A Species Definition for the Modern Synthesis},
  journal = {Trends in Ecology and Evolution},
  volume = {10},
  year = {1995},
  pages = {294--299},
}

@book{Futuyma2017,
  author = {Futuyma, Douglas J.},
  title = {Evolution},
  edition = {4th},
  publisher = {Sinauer Associates},
  year = {2017},
}

@article{ServedioNoor2003,
  author = {Servedio, Maria R. and Noor, Mohamed A. F.},
  title = {The Role of Reinforcement in Speciation: Theory and Data},
  journal = {Annual Review of Ecology, Evolution, and Systematics},
  volume = {34},
  year = {2003},
  pages = {339--364},
}

@article{Endler1992,
  author = {Endler, John A.},
  title = {Signals, Signal Conditions, and the Direction of Evolution},
  journal = {American Naturalist},
  volume = {139},
  year = {1992},
  pages = {S125--S153},
}

@article{Hoskin2005,
  author = {Hoskin, Conrad J. and Higgie, Megan and McDonald, Keith R. and Moritz, Craig},
  title = {Reinforcement Drives Rapid Allopatric Speciation},
  journal = {Nature},
  volume = {437},
  year = {2005},
  pages = {1353--1356},
}

Prerequisites

19.01.01 pending
19.02.05 pending
19.03.02 pending

Tier anchors

beginner: Coyne Why Evolution Is True Ch. 5, 8; Campbell Biology 12th ed. Ch. 24; Crash Course Biology speciation episodes
intermediate: Futuyma Evolution 4th ed. Ch. 17-18; Coyne & Orr Speciation (Sinauer 2004)
master: Coyne & Orr 2004 advanced sections; primary literature — Mayr 1942, Dobzhansky 1937, Orr 1995, Turelli & Orr 2000

References

TODO_REF pending
Mayr, E. — Systematics and the Origin of Species (Columbia UP, 1942) · Originator monograph for the biological species concept and allopatric speciation · see docs/catalogs/NEED_TO_SOURCE.md#bio-mayr-1942
TODO_REF pending
Dobzhansky, T. — Genetics and the Origin of Species (Columbia UP, 1937) · Originator monograph for the genetic basis of speciation · see docs/catalogs/NEED_TO_SOURCE.md#bio-dobzhansky-1937
TODO_REF pending
Coyne, J. A. & Orr, H. A. — Speciation (Sinauer, 2004) · Ch. 1-12 Comprehensive review of speciation mechanisms · see docs/catalogs/NEED_TO_SOURCE.md#bio-coyne-orr-2004
TODO_REF pending
Futuyma, D. J. — Evolution, 4th ed. (Sinauer, 2017) · Ch. 17-18 Speciation · see docs/catalogs/NEED_TO_SOURCE.md#bio-futuyma-2017
TODO_REF pending
Grant, P. R. & Grant, B. R. — How and Why Species Multiply (Princeton UP, 2008) · Radiation of Darwin's finches as a model system for speciation · see docs/catalogs/NEED_TO_SOURCE.md#bio-grant-grant-2008
TODO_REF pending
Orr, H. A. — The population genetics of speciation: the evolution of hybrid incompatibilities · Genetics 139 (1995), 1805-1813; originator paper for the snowball theory · see docs/catalogs/NEED_TO_SOURCE.md#bio-orr-1995
TODO_REF pending
Turelli, M. & Orr, H. A. — Dominance theory of Haldane's rule · Genetics 140 (1995), 389-402 · see docs/catalogs/NEED_TO_SOURCE.md#bio-turelli-orr-1995
TODO_REF pending
Muller, H. J. — Bearing of the Drosophila work on systematics · in Huxley, J. S. (ed.), The New Systematics (Clarendon Press, Oxford, 1940), pp. 185-268; originator of the two-locus incompatibility model · see docs/catalogs/NEED_TO_SOURCE.md#bio-muller-1940
tong
raw/pdfs/mathbio/mathbio.pdf · Mathematical biology background — population divergence and isolation models

Reviewer

Tyler (pending external biology reviewer per BIOLOGY_PLAN §6)

Estimated time

beginner: 16m
intermediate: 40m
master: 75m