Increasing body and brain size constitutes a key macro-evolutionary pattern in the hominin lineage, yet the mechanisms behind these changes remain debated. Hypothesized drivers include environmental, demographic, social, dietary, and technological factors. Here we test the influence of environmental factors on the evolution of body and brain size in the genus Homo over the last one million years using a large fossil dataset combined with global paleoclimatic reconstructions and formalized hypotheses tested in a quantitative statistical framework. We identify temperature as a major predictor of body size variation within Homo, in accordance with Bergmann's rule. In contrast, net primary productivity of environments and long-term variability in precipitation correlate with brain size but explain low amounts of the observed variation. These associations are likely due to an indirect environmental influence on cognitive abilities and extinction probabilities. Most environmental factors that we test do not correspond with body and brain size evolution, pointing towards complex scenarios which underlie the evolution of key biological characteristics in later Homo.