SARS-CoV-2 entry into host cells is orchestrated by the spike (S) glycoprotein that contains an immunodominant receptor-binding domain (RBD) targeted by the largest fraction of neutralizing antibodies (Abs) in COVID-19 patient plasma. Little is known about neutralizing Abs binding to epitopes outside the RBD and their contribution to protection. Here, we describe 41 human monoclonal Abs (mAbs) derived from memory B cells, which recognize the SARS-CoV-2 S N-terminal domain (NTD) and show that a subset of them neutralize SARS-CoV-2 ultrapotently. We define an antigenic map of the SARS-CoV-2 NTD and identify a supersite recognized by all known NTD-specific neutralizing mAbs. These mAbs inhibit cell-to-cell fusion, activate effector functions, and protect Syrian hamsters from SARS-CoV-2 challenge. SARS-CoV-2 variants, including the 501Y.V2 and B.1.1.7 lineages, harbor frequent mutations localized in the NTD supersite suggesting ongoing selective pressure and the importance of NTD-specific neutralizing mAbs to protective immunity.