Attentional set-shifting ability, commonly assessed with the Trail Making Test (TMT), decreases with increasing age in adults. Since set-shifting performance relies on activity in widespread brain regions, deterioration of the white matter tracts that connect these regions may underlie the age-related decrease in performance. We used an automated fiber tracking method to investigate the relationship between white matter integrity in several cortical association tracts and TMT performance in a sample of 24 healthy adults, 21-80 years. Diffusion tensor images were used to compute average fractional anisotropy (FA) for five cortical association tracts, the corpus callosum (CC), and the corticospinal tract (CST), which served as a control. Results showed that advancing age was associated with declines in set-shifting performance and with decreased FA in the CC and in association tracts that connect frontal cortex to more posterior brain regions, including the inferior fronto-occipital fasciculus (IFOF), uncinate fasciculus (UF), and superior longitudinal fasciculus (SLF). Declines in average FA in these tracts, and in average FA of the right inferior longitudinal fasciculus (ILF), were associated with increased time to completion on the set-shifting subtask of the TMT but not with the simple sequencing subtask. FA values in these tracts were strong mediators of the effect of age on set-shifting performance. Automated tractography methods can enhance our understanding of the fiber systems involved in performance of specific cognitive tasks and of the functional consequences of age-related changes in those systems.