Sustained progression on the expanded disability status scale (EDSS) is a common outcome measure of disease progression in clinical studies of MS. Unfortunately, this outcome may not accurately measure long-term and irreversible disease progression. To assess the performance of definitions of sustained progression, patients with relapsing-remitting MS (RRMS) or a clinically isolated syndrome with evidence of lesions on a brain MRI were included in our study. Fifteen definitions of sustained progression using both the EDSS and the functional system (FS) scales were investigated. The impact of both relapses and changes in provider on the probability of maintaining progression was also evaluated. Although the provider scoring the EDSS sometimes changed during followup, the provider had access to previous EDSS scores. Between 15.8% and 42.2% of patients experienced sustained progression based on the definitions using EDSS as the outcome, but nearly 50% of these patients failed to maintain sustained progression for the duration of followup. When FS scales were used, progression was most common on the pyramidal and sensory scales. Unfortunately, progression on specific FS scales failed to be more sensitive to irreversible disability. Relapses or changes in provider did not explain the poor performance of the measures. Short-term changes in the EDSS or FS scores may not be an accurate marker of irreversible change in RRMS.