The phenomenon of contrasting color perceptions of "the dress" photograph has gained scientific interest. The mechanism underlying why individuals differ is yet to be fully explained. We use the powerful twin model design to ascertain the relative contribution of genetic and environmental factors on perception variation. A sample of 466 twins from the British TwinsUK registry were invited to report what color they saw in a standard image of the dress in standard illumination. The mean age of the participants was 49.5 (SD = 17.8) years, and 85% were female. When asked to choose between white and gold (WG) or blue and black (BB), 328 reported WG (70.4%) and 135 (29.0%) reported BB. Subjects choosing WG were significantly older (p < 0.01), but there was no significant difference in gender. Monozygotic (MZ) twins were more concordant in their responses than dizygotic (DZ) twins (0.46 vs. 0.36). Twin modeling revealed that genetic factors accounted for 34% (95% confidence interval, 5%-59%) of variation in the reported color of the dress when adjusted for age, whereas environmental factors contributed 66% (95% CI, 41%-95%). This study suggests environmental factors play a significant role in how an individual perceives the color of "the dress."