High birth weight is associated with increased breast cancer risk and, less consistently, with higher mammographic density. In contrast, adolescent body size has been consistently, negatively associated with both MD and breast cancer risk. It is unclear when the direction of these associations changes and whether weight gain in infancy is associated with MD. We evaluated the associations of birth weight and postnatal weight (measured at 4 months, 1 year, and 4 years) by absolute and velocity measures (relative within-cohort percentile changes) with adult mammographic density, assessed using a computer-assisted thresholding program (Cumulus), using linear regression models with generalized estimating equations to account for correlation between siblings in the Early Determinants of Mammographic Density study (1959-2008; n = 700 women with 116 sibling sets; mean age = 44.1 years). Birth weight was positively associated with dense area (per 1-kg increase, β = 3.36, 95% confidence interval (CI): 0.06, 6.66). Weight gains from 0 months to 4 months and 1 year to 4 years were negatively associated with dense area (for 10-unit increase in weight percentile, β = -0.65, 95% CI: -1.23, -0.07, and β = -1.07, 95% CI: -1.98, -0.16, respectively). Findings were similar in the sibling subset. These results support the hypothesis that high birth weight is positively associated with increased breast density and suggest that growth spurts starting in early infancy reduce mammographic dense area in adulthood.