The selection pressures that have shaped the evolution of complex traits in humans remain largely unknown, and in some contexts highly contentious, perhaps above all where they concern mean trait differences among groups. To date, the discussion has focused on whether such group differences have any genetic basis, and if so, whether they are without fitness consequences and arose via random genetic drift, or whether they were driven by selection for different trait optima in different environments. Here, we highlight a plausible alternative: that many complex traits evolve under stabilizing selection in the face of shifting environmental effects. Under this scenario, there will be rapid evolution at the loci that contribute to trait variation, even when the trait optimum remains the same. These considerations underscore the strong assumptions about environmental effects that are required in ascribing trait differences among groups to genetic differences.