SARS-CoV-2 is difficult to contain because many transmissions occur during the pre-symptomatic phase of infection. Moreover, in contrast to influenza, while most SARS-CoV-2 infected people do not transmit the virus to anybody, a small percentage secondarily infect large numbers of people. We designed mathematical models of SARS-CoV-2 and influenza which link observed viral shedding patterns with key epidemiologic features of each virus, including distributions of the number of secondary cases attributed to each infected person (individual R0) and the duration between symptom onset in the transmitter and secondarily infected person (serial interval). We identify that people with SARS-CoV-2 or influenza infections are usually contagious for fewer than one day congruent with peak viral load several days after infection, and that transmission is unlikely below a certain viral load. SARS-CoV-2 super-spreader events with over 10 secondary infections occur when an infected person is briefly shedding at a very high viral load and has a high concurrent number of exposed contacts. The higher predisposition of SARS-CoV-2 towards super-spreading events is not due to its 1-2 additional weeks of viral shedding relative to influenza. Rather, a person infected with SARS-CoV-2 exposes more people within equivalent physical contact networks than a person infected with influenza, likely due to aerosolization of virus. Our results support policies that limit crowd size in indoor spaces and provide viral load benchmarks for infection control and therapeutic interventions intended to prevent secondary transmission.