Monoclonal antibodies targeting a variety of epitopes have been isolated from individuals previously infected with SARS-CoV-2, but the relative contributions of these different antibody classes to the polyclonal response remains unclear. Here we use a yeast-display system to map all mutations to the viral spike receptor-binding domain (RBD) that escape binding by representatives of three potently neutralizing classes of anti-RBD antibodies with high-resolution structures. We compare the antibody-escape maps to similar maps for convalescent polyclonal plasma, including plasma from individuals from whom some of the antibodies were isolated. The plasma-escape maps most closely resemble those of a single class of antibodies that target an epitope on the RBD that includes site E484. Therefore, although the human immune system can produce antibodies that target diverse RBD epitopes, in practice the polyclonal response to infection is dominated by a single class of antibodies targeting an epitope that is already undergoing rapid evolution.