Active lifestyles might protect cognitive abilities; however, studies rarely consider the reverse causal direction. Activity-cognition associations might reflect stable intelligence differences rather than a protective effect of activity. The Lothian Birth Cohort 1936 (n = 1091) completed cognitive tests aged 70, having taken an intelligence test aged 11. Activity (assessed by participation in 15 activities that produced a socio-intellectual activity factor, and by physical activity) was positively associated with cognition (r = .08 to .32, p ≤ .05). When age-11 IQ and adult social class were controlled, only physical activity remained significantly associated with general cognitive ability and processing speed.