One of the mechanisms that can lead to the formation of new species occurs through the evolution of reproductive barriers. However, recent research has demonstrated that hybridization has been pervasive across the tree of life even in the presence of strong barriers. Swordtail fishes (genus Xiphophorus) are an emerging model system for studying the interface between these barriers and hybridization. We document overlapping mechanisms that act as barriers between closely related species, X. birchmanni and X. cortezi, by combining genomic sequencing from natural hybrid populations, artificial crosses, behavioral assays, sperm performance, and developmental studies. We show that strong assortative mating plays a key role in maintaining subpopulations with distinct ancestry in natural hybrid populations. Lab experiments demonstrate that artificial F1 crosses experience dysfunction: crosses with X. birchmanni females were largely inviable and crosses with X. cortezi females had a heavily skewed sex ratio. Using F2 hybrids we identify several genomic regions that strongly impact hybrid viability. Strikingly, two of these regions underlie genetic incompatibilities in hybrids between X. birchmanni and its sister species X. malinche. Our results demonstrate that ancient hybridization has played a role in the origin of this shared genetic incompatibility. Moreover, ancestry mismatch at these incompatible regions has remarkably similar consequences for phenotypes and hybrid survival in X. cortezi × X. birchmanni hybrids as in X. malinche × X. birchmanni hybrids. Our findings identify varied reproductive barriers that shape genetic exchange between naturally hybridizing species and highlight the complex evolutionary outcomes of hybridization.
Keywords: assortative mating; genetic incompatibilities; hybridization; reproductive isolation.