Social group size has been shown to correlate with neocortex size in primates. Here we use comparative analyses to show that social group size is independently correlated with the size of non-V1 neocortical areas, but not with other more proximate components of the visual system or with brain systems associated with emotional cueing (e.g. the amygdala). We argue that visual brain components serve as a social information 'input device' for socio-visual stimuli such as facial expressions, bodily gestures and visual status markers, while the non-visual neocortex serves as a 'processing device' whereby these social cues are encoded, interpreted and associated with stored information. However, the second appears to have greater overall importance because the size of the V1 visual area appears to reach an asymptotic size beyond which visual acuity and pattern recognition may not improve significantly. This is especially true of the great ape clade (including humans), that is known to use more sophisticated social cognitive strategies.