COVID-19 patients exhibit differential disease severity after SARS-CoV-2 infection. It is currently unknown as to the correlation between the magnitude of neutralizing antibody (NAb) responses and the disease severity in COVID-19 patients. In a cohort of 59 recovered patients with disease severity including severe, moderate, mild, and asymptomatic, we observed the positive correlation between serum neutralizing capacity and disease severity, in particular, the highest NAb capacity in sera from the patients with severe disease, while a lack of ability of asymptomatic patients to mount competent NAbs. Furthermore, the compositions of NAb subtypes were also different between recovered patients with severe symptoms and with mild-to-moderate symptoms. These results reveal the tremendous heterogeneity of SARS-CoV-2-specific NAb responses and their correlations to disease severity, highlighting the needs of future vaccination in COVID-19 patients recovered from asymptomatic or mild illness.