In global terms, respiratory viral infection is a major cause of morbidity and mortality. Infancy, in particular, is a time of increased disease susceptibility and severity. Early-life viral infection causes acute illness and can be associated with the development of wheezing and asthma in later life. The most commonly detected viruses are respiratory syncytial virus (RSV), rhinovirus (RV), and influenza virus. In this review we explore the complete picture from epidemiology and virology to clinical impact and immunology. Three striking aspects emerge. The first is the degree of similarity: although the infecting viruses are all different, the clinical outcome, viral evasion strategies, immune response, and long-term sequelae share many common features. The second is the interplay between the infant immune system and viral infection: the immaturity of the infant immune system alters the outcome of viral infection, but at the same time, viral infection shapes the development of the infant immune system and its future responses. Finally, both the virus and the immune response contribute to damage to the lungs and subsequent disease, and therefore, any prevention or treatment needs to address both of these factors.