The apolipoprotein E (APOE) ε4 allele is strongly linked with cerebral β-amyloidosis, but its relationship with tauopathy is less established. We investigated the relationship between APOE ε4 carrier status, regional amyloid-β (Aβ), magnetic resonance imaging (MRI) volumetrics, tau positron emission tomography (PET), APOE messenger RNA (mRNA) expression maps, and cerebrospinal fluid phosphorylated tau (CSF ptau181). Three hundred fifty participants underwent imaging, and 270 had ptau181. We used computational models to evaluate the main effect of APOE ε4 carrier status on regional neuroimaging values and then the interaction of ε4 status and global Aβ on regional tau PET and brain volumes as well as CSF ptau181. Separately, we also examined the additional interactive influence of sex. We found that, for the same degree of Aβ burden, APOE ε4 carriers showed greater tau PET signal relative to noncarriers in temporal regions, but no interaction was present for MRI volumes or CSF ptau181. This potentiation of tau aggregation irrespective of sex occurred in brain regions with high APOE mRNA expression, suggesting local vulnerabilities to tauopathy. There were greater effects of APOE genotype in females, although the interactive sex effects did not strongly mirror mRNA expression. Pathology is not homogeneously expressed throughout the brain but mirrors underlying biological patterns such as gene expression.