Most antibodies isolated from COVID-19 patients are specific to SARS-CoV-2. COVA1-16 is a relatively rare antibody that also cross-neutralizes SARS-CoV. Here we determined a crystal structure of COVA1-16 Fab with the SARS-CoV-2 RBD, and a negative-stain EM reconstruction with the spike glycoprotein trimer, to elucidate the structural basis of its cross-reactivity. COVA1-16 binds a highly conserved epitope on the SARS-CoV-2 RBD, mainly through a long CDR H3, and competes with ACE2 binding due to steric hindrance rather than epitope overlap. COVA1-16 binds to a flexible up conformation of the RBD on the spike and relies on antibody avidity for neutralization. These findings, along with structural and functional rationale for the epitope conservation, provide a blueprint for development of more universal SARS-like coronavirus vaccines and therapies.