In pandemic and epidemic forms, influenza causes substantial, sometimes catastrophic, morbidity and mortality. Intense selection from the host immune system drives antigenic change in influenza A and B, resulting in continuous replacement of circulating strains with new variants able to re-infect hosts immune to earlier types. This 'antigenic drift' often requires a new vaccine to be formulated before each annual epidemic. However, given the high transmissibility and mutation rate of influenza, the constancy of genetic diversity within lineages over time is paradoxical. Another enigma is the replacement of existing strains during a global pandemic caused by 'antigenic shift'--the introduction of a new avian influenza A subtype into the human population. Here we explore ecological and immunological factors underlying these patterns using a mathematical model capturing both realistic epidemiological dynamics and viral evolution at the sequence level. By matching model output to phylogenetic patterns seen in sequence data collected through global surveillance, we find that short-lived strain-transcending immunity is essential to restrict viral diversity in the host population and thus to explain key aspects of drift and shift dynamics.