Maintenance of physical activity and good physical fitness is important for functional independence. This study had two aims: examine the physical fitness level in older persons with mild cognitive impairment (MCI) or dementia, and examine the relationship between the components of physical fitness and cognitive domains in this group. The cross-sectional study included community-living older people ≥ 65 years of age with MCI or dementia. Physical fitness and cognition were assessed using the Senior Fitness Test and five cognitive tests. Most of the participants scored below the criteria for maintaining physical independence in later years. There were significant associations between the components of physical fitness and cognition, except flexibility. Declines in executive function were most related to declines in physical fitness. These factors should receive more attention in people with MCI and dementia because they risk losing independence.