"BRCAX" refers breast cancers occurring in women with a family history predictive of being a BRCA1/2 mutation carrier, but BRCA1/2 genetic screening has failed to find causal mutations. In this study, we report the findings of the genetic architecture of BRCAX with novel and redefined candidate loci and their potential impacts on preventive strategy. We performed a genome-wide association study involving 1,469 BRCAX cases from the Korean Hereditary Breast Cancer study, and high-risk breast cancer cases (1,482 Asians and 9,902 Europeans) from the Breast Cancer Association Consortium. We also evaluated the previously reported susceptibility loci for their roles in the high-risk breast cancers. We have identified three novel loci (PDE7B, UBL3, and a new independent marker in CDKN2B-AS1) associated with BRCAX, and replicated previously reported SNPs (24 of 92) and moderate/high-penetrance (seven of 23) genes for Korean BRCAX. For the novel candidate loci, evidence supported their roles in regulatory function. We estimated that the common low-penetrance loci might explain a substantial part of high-risk breast cancer (39.4% for Koreans and 24.0% for Europeans). Our study findings suggest that common genetic markers with lower penetrance constitute a part of susceptibility to high-risk breast cancers, with potential implications for a more comprehensive genetic screening test.