The impact of genetic and environmental factors on human brain structure is of great importance for understanding normative cognitive and brain aging as well as neuropsychiatric disorders. However, most studies of genetic and environmental influences on human brain structure have either focused on global measures or have had samples that were too small for reliable estimates. Using the classical twin design, we assessed genetic, shared environmental, and individual-specific environmental influences on individual differences in the size of 96 brain regions of interest (ROIs). Participants were 474 middle-aged male twins (202 pairs; 70 unpaired) in the Vietnam Era Twin Study of Aging (VETSA). They were 51-59 years old, and were similar to U.S. men in their age range in terms of sociodemographic and health characteristics. We measured thickness of cortical ROIs and volume of other ROIs. On average, genetic influences accounted for approximately 70% of the variance in the volume of global, subcortical, and ventricular ROIs and approximately 45% of the variance in the thickness of cortical ROIs. There was greater variability in the heritability of cortical ROIs (0.00-0.75) as compared with subcortical and ventricular ROIs (0.48-0.85). The results did not indicate lateralized heritability differences or greater genetic influences on the size of regions underlying higher cognitive functions. The findings provide key information for imaging genetic studies and other studies of brain phenotypes and endophenotypes. Longitudinal analysis will be needed to determine whether the degree of genetic and environmental influences changes for different ROIs from midlife to later life.