Sexual dimorphism in common disease is pervasive, including a dramatic male preponderance in autism spectrum disorders (ASDs). Potential genetic explanations include a liability threshold model requiring increased polymorphism risk in females, sex-limited X-chromosome contribution, gene-environment interaction driven by differences in hormonal milieu, risk influenced by genes sex-differentially expressed in early brain development, or contribution from general mechanisms of sexual dimorphism shared with secondary sex characteristics. Utilizing a large single nucleotide polymorphism (SNP) dataset, we identify distinct sex-specific genome-wide significant loci. We investigate genetic hypotheses and find no evidence for increased genetic risk load in females, but evidence for sex heterogeneity on the X chromosome, and contribution of sex-heterogeneous SNPs for anthropometric traits to ASD risk. Thus, our results support pleiotropy between secondary sex characteristic determination and ASDs, providing a biological basis for sex differences in ASDs and implicating non brain-limited mechanisms.