The study aim was to estimate the genetic contribution to individual differences in different forms of memory in a large family-based group of older adults. As part of the Late Onset Alzheimer's Disease Family Study, 899 persons (277 with Alzheimer's disease, 622 unaffected) from 325 families completed a battery of memory tests from which previously established composite measures of episodic memory, semantic memory, and working memory were derived. Heritability in these measures was estimated using the maximum likelihood variance component method, controlling for age, gender, and education. In analyses of unaffected family members, the adjusted heritability estimates were 0.62 for episodic memory, 0.49 for semantic memory, and 0.72 for working memory, where a heritability estimate of 1 indicates that genetic factors explain all of the phenotypic variance and a heritability of 0 indicates that genetic factors explain none. Adjustment for APOE genotype had little effect on these estimates. When analyses included affected and unaffected family members, adjusted heritability estimates were lower (0.47 for episodic memory, 0.32 for semantic memory, 0.42 for working memory). Adjusting for APOE slightly reduced the estimate for episodic memory (0.40) but had no effect on the remaining estimates. The results indicate that memory functions are under strong genetic influence in older persons with and without AD, and are only partly attributable to APOE. This suggests that genetic analyses of memory endophenotypes may help to identify genetic variants associated with AD.