Cumulative evidence of gray matter abnormalities in semantic dementia (SD) has been reported using voxel-based morphometry (VBM). However, these studies have not been reviewed quantitatively. To estimate gray matter changes in SD quantitatively, we systematically searched whole-brain VBM studies comparing SD patients with healthy controls in the PubMed, ISI Web of Science, and EMABSE databases from January 1990 to August 2011. Coordinates with significant differences between the gray matter volumes of SD patients and healthy controls were extracted from clusters. Meta-analysis was performed using anatomic likelihood estimation. Seven studies, with 68 SD patients and 167 healthy controls, were included. Gray matter volume reductions were found in bilateral fusiform and inferior temporal gyri, extending to the medial portion of the temporal lobes (including amygdala and parahippocampal gyri), left temporal pole, middle temporal gyrus, and caudate. No significant increase in gray matter volume was found. Our findings provide strong evidence of atrophy in bilateral temporal lobes with predominate impairment on the left side, parahippocampal gyrus/amygdala, and left caudate, representing the pathophysiology of SD.