Face-derived information on trustworthiness and attractiveness crucially influences social interaction. It is, however, unclear to what degree the functional neuroanatomy of these complex social judgments on faces reflects genuine social versus basic emotional and cognitive processing. To disentangle social from nonsocial contributions, we assessed commonalities and differences between the functional networks activated by judging social (trustworthiness, attractiveness), emotional (happiness), and cognitive (age) facial traits. Relative to happiness and age evaluations, both trustworthiness and attractiveness judgments selectively activated the dorsomedial prefrontal cortex and inferior frontal gyrus, forming a core social cognition network. Moreover, they also elicited a higher amygdalar response than even the emotional control condition. Both social judgments differed, however, in their top-down modulation of face-sensitive regions: trustworthiness judgments recruited the posterior superior temporal sulcus, whereas attractiveness judgments recruited the fusiform gyrus. Social and emotional judgments converged and, therefore, likely interact in the ventromedial prefrontal cortex. Social and age judgments, on the other hand, commonly engaged the anterior insula, inferior parietal cortex, and dorsolateral prefrontal cortex, which appear to subserve more cognitive aspects in social evaluation. These findings demonstrate the modularity of social judgments on human faces by separating the neural correlates of social, face-specific, emotional, and cognitive processing facets.