The goal of the current study was to examine cognitive change in both healthy controls (n = 229) and individuals with mild cognitive impairment (MCI) (n = 397) from the Alzheimer's Disease Neuroimaging Initiative (ADNI). We applied latent growth modeling to examine baseline and longitudinal change over 36 months in five cognitive factors derived from the ADNI neuropsychological test battery (memory, executive function/processing speed, language, attention and visuospatial). At baseline, MCI patients demonstrated lower performance on all of the five cognitive factors when compared to controls. Both controls and MCI patients declined on memory over 36 months; however, the MCI patients declined at a significantly faster rate than controls. The MCI patients also declined over 36 months on the remaining four cognitive factors. In contrast, the controls did not exhibit significant change over 36 months on the non-memory cognitive factors. Within the MCI group, executive function declined faster than memory, while the other factor scores changed slower than memory over time. These findings suggest different patterns of cognitive change in healthy older adults and MCI patients. The findings also suggest that, when compared with memory, executive function declines faster than other cognitive factors in patients with MCI. Thus, decline in non-memory domains may be an important feature for distinguishing healthy older adults and persons with MCI.