The utility of recombinant adenovirus serotype 5 (rAd5) vector-based vaccines for HIV-1 and other pathogens will likely be limited by the high prevalence of pre-existing Ad5-specific neutralizing Abs (NAbs) in human populations. However, the immunodominant targets of Ad5-specific NAbs in humans remain poorly characterized. In this study, we assess the titers and primary determinants of Ad5-specific NAbs in individuals from both the United States and the developing world. Importantly, median Ad5-specific NAb titers were >10-fold higher in sub-Saharan Africa compared with the United States. Moreover, hexon-specific NAb titers were 4- to 10-fold higher than fiber-specific NAb titers in these cohorts by virus neutralization assays using capsid chimeric viruses. We next performed adoptive transfer studies in mice to evaluate the functional capacity of hexon- and fiber-specific NAbs to suppress the immunogenicity of a prototype rAd5-Env vaccine. Hexon-specific NAbs were remarkably efficient at suppressing Env-specific immune responses elicited by the rAd5 vaccine. In contrast, fiber-specific NAbs exerted only minimal suppressive effects on rAd5 vaccine immunogenicity. These data demonstrate that functionally significant Ad5-specific NAbs are directed primarily against the Ad5 hexon protein in both humans and mice. These studies suggest a potential strategy for engineering novel Ad5 vectors to evade dominant Ad5-specific NAbs.