Previous research has identified morphological differences between the brains of Neanderthals and anatomically modern humans (AMHs). However, studies using endocasts or the cranium itself are limited to investigating external surface features and the overall size and shape of the brain. A complementary approach uses comparative primate data to estimate the size of internal brain areas. Previous attempts to do this have generally assumed that identical total brain volumes imply identical internal organization. Here, we argue that, in the case of Neanderthals and AMHs, differences in the size of the body and visual system imply differences in organization between the same-sized brains of these two taxa. We show that Neanderthals had significantly larger visual systems than contemporary AMHs (indexed by orbital volume) and that when this, along with their greater body mass, is taken into account, Neanderthals have significantly smaller adjusted endocranial capacities than contemporary AMHs. We discuss possible implications of differing brain organization in terms of social cognition, and consider these in the context of differing abilities to cope with fluctuating resources and cultural maintenance.