Higher levels of circulating estrogens and estrogen metabolites (EMs) have been associated with higher breast cancer risk. In breast tissues, reduced levels of terminal duct lobular unit (TDLU) involution, as reflected by higher numbers of TDLUs and acini per TDLU, have also been linked to elevated breast cancer risk. However, it is unknown whether reduced TDLU involution mediates the risk associated with circulating EMs. In a cross-sectional analysis of 94 premenopausal and 92 postmenopausal women referred for clinical breast biopsy at an academic facility in Vermont, we examined the associations of 15 EMs, quantified using liquid chromatography-tandem mass spectrometry, with the number of TDLUs and acini count/TDLU using zero-inflated Poisson regression with a robust variance estimator and ordinal logistic regression models, respectively. All analyses were stratified by menopausal status and adjusted for potential confounders. Among premenopausal women, comparing the highest vs. the lowest tertiles, levels of unconjugated estradiol (risk ratio (RR) = 1.74, 95 % confidence interval (CI) = 1.06-2.87, p trend = 0.03), 2-hydroxyestrone (RR = 1.74, 95 % CI = 1.01-3.01, p trend = 0.04), and 4-hydroxyestrone (RR = 1.74, 95 % CI = 0.99-3.06, p trend = 0.04) were associated with significantly higher TDLU count. Among postmenopausal women, higher levels of estradiol (RR = 2.09, 95 % CI = 1.01-4.30, p trend = 0.04) and 16α-hydroxyestrone (RR = 2.27, 95 % CI = 1.29-3.99, p trend = 0.02) were significantly associated with higher TDLU count. Among postmenopausal women, higher levels of EMs, specifically conjugated estrone and 2- and 4-pathway catechols, were also associated with higher acini count/TDLU. Our data suggest that higher levels of serum EMs are generally associated with lower levels of TDLU involution.