The e4 allele of the apolipoprotein E (APOE) gene confers risk of Alzheimer's disease and, in some studies, relates to cognitive ability and decline in older people without Alzheimer's disease. Its relationship with processing speed, a contributor to cognitive decline with age, is largely unknown. This study tests the association of APOE with cognition and speed, with and without covarying childhood mental ability. The 1,013 participants were tested on cognitive ability at age 11 as part of the Scottish Mental Survey of 1947 and, at age 70, were tested on reasoning, working memory, information processing speed, and executive function. The results showed that APOE was associated with the general cognitive factor, 2 nonverbal tests, and choice reaction time (RT) variability; as expected, the e4 allele was the risk allele. RT measures and a general speed factor were nonlinearly related to APOE when factoring childhood ability (p < .05): The correlation between childhood ability and speed was lower in e4 allele carriers. APOE has an influence on nonverbal cognition in old age and interacts with childhood IQ to influence processing speed.