Up to 25% of patients with nonalcoholic fatty liver disease (NAFLD) are not obese but may have a fat or muscle composition that predisposes them to NAFLD. Our aim was to determine whether body composition parameters associate with NAFLD and to identify genetic contributors to this association. This study included two cohorts. The first included 2,249 participants from the Framingham Heart Study who underwent a computed tomography scan to evaluate hepatic steatosis, dual-energy x-ray absorptiometry testing to assess body composition, and clinical examination. Body composition parameters were normalized to total body weight. A subset of participants underwent genotyping with an Affymetrix 550K single-nucleotide polymorphism array. The second cohort, Michigan Genomics Initiative, included 19,239 individuals with genotyping on the Illumina HumanCoreExome v.12.1 array and full electronic health record data. Using sex-stratified multivariable linear regression, greater central body fat associated with increased hepatic steatosis while greater lower extremity body fat associated with decreased hepatic steatosis. Greater appendicular lean mass was associated with decreased hepatic steatosis in men but not in women. A polygenic risk score for lipodystrophy (regional or global loss of adipose tissue) was associated with increased hepatic steatosis, increased liver fibrosis, and decreased lower extremity fat mass. Conclusion: Greater central body fat associated with increased hepatic steatosis, while greater lower extremity body fat and, in men, greater appendicular lean mass were associated with decreased hepatic steatosis. A genetic risk score for lipodystrophy was associated with NAFLD and liver fibrosis. Our results suggest that buffering of excess energy by peripheral fat and muscle may protect against NAFLD and liver fibrosis in the general population.