Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is currently causing a global pandemic. The antigen specificity and kinetics of the antibody response mounted against this novel virus are not understood in detail. Here, we report that subjects with a more severe SARS-CoV-2 infection exhibit a larger antibody response against the spike and nucleocapsid protein and epitope spreading to subdominant viral antigens, such as open reading frame 8 and non-structural proteins. Subjects with a greater antibody response mounted a larger memory B cell response against the spike, but not the nucleocapsid protein. Additionally, we revealed that antibodies against the spike are still capable of binding the D614G spike mutant and cross-react with the SARS-CoV-1 receptor binding domain. Together, this study reveals that subjects with a more severe SARS-CoV-2 infection exhibit a greater overall antibody response to the spike and nucleocapsid protein and a larger memory B cell response against the spike.