Molecular-level understanding of human neutralizing antibody responses to SARS-CoV-2 could accelerate vaccine design and facilitate drug discovery. We analyzed 294 SARS-CoV-2 antibodies and found that IGHV3-53 is the most frequently used IGHV gene for targeting the receptor binding domain (RBD) of the spike (S) protein. We determined crystal structures of two IGHV3-53 neutralizing antibodies +/- Fab CR3022 ranging from 2.33 to 3.11 Å resolution. The germline-encoded residues of IGHV3-53 dominate binding to the ACE2 binding site epitope with no overlap with the CR3022 epitope. Moreover, IGHV3-53 is used in combination with a very short CDR H3 and different light chains. Overall, IGHV3-53 represents a versatile public VH in neutralizing SARS-CoV-2 antibodies, where their specific germline features and minimal affinity maturation provide important insights for vaccine design and assessing outcomes.