Purpose: Registers derived from administrative datasets are valuable tools in psychosis research, but diagnostic accuracy can be problematic. We sought to compare the relative performance of four methods for assigning a single diagnosis from longitudinal administrative clinical records when compared with reference diagnoses.
Methods: Diagnoses recorded in inpatient and community mental health records were compared to research diagnoses of psychotic disorders obtained from semi-structured clinical interviews for 289 persons. Diagnoses were derived from administrative datasets using four algorithms; 'At least one' diagnosis, 'Last' or most recent diagnosis, 'Modal' or most frequently occurring diagnosis, and 'Hierarchy' in which a diagnostic hierarchy was applied. Agreements between algorithm-based and reference diagnoses for overall presence/absence of psychosis and for specific diagnoses of schizophrenia, schizoaffective disorder, and affective psychosis were examined using estimated prevalence rates, overall agreement, ROC analysis, and kappa statistics.
Results: For the presence/absence of psychosis, the most sensitive and least specific algorithm ('At least one' diagnosis) performed best. For schizophrenia, 'Modal' and 'Last' diagnoses had greatest agreement with reference diagnosis. For affective psychosis, 'Hierarchy' diagnosis performed best. Agreement between clinical and reference diagnoses was no better than chance for diagnoses of schizoaffective disorder. Overall agreement between administrative and reference diagnoses was modest, but may have been limited by the use of participants who had been screened for likely psychosis prior to assessment.
Conclusion: The choice of algorithm for extracting a psychosis diagnosis from administrative datasets may have a substantial impact on the accuracy of the diagnoses derived. An 'Any diagnosis' algorithm provides a sensitive measure for the presence of any psychosis, while 'Last diagnosis' is more accurate for specific diagnosis of schizophrenia and a hierarchical diagnosis is more accurate for affective psychosis.