Rationale: Computerized severity assessment for community-acquired pneumonia could improve consistency and reduce clinician burden. Objectives: To develop and compare 30-day mortality-prediction models using electronic health record data, including a computerized score with all variables from the original Pneumonia Severity Index (PSI) except confusion and pleural effusion ("ePSI score") versus models with additional variables. Methods: Among adults with community-acquired pneumonia presenting to emergency departments at 117 Veterans Affairs Medical Centers between January 1, 2006, and December 31, 2016, we compared an ePSI score with 10 novel models employing logistic regression, spline, and machine learning methods using PSI variables, age, sex and 26 physiologic variables as well as all 69 PSI variables. Models were trained using encounters before January 1, 2015; tested on encounters during and after January 1, 2015; and compared using the areas under the receiver operating characteristic curve, confidence intervals, and patient event rates at a threshold PSI score of 970. Results: Among 297,498 encounters, 7% resulted in death within 30 days. When compared using the ePSI score (confidence interval [CI] for the area under the receiver operating characteristic curve, 0.77-0.78), performance increased with model complexity (CI for the logistic regression PSI model, 0.79-0.80; CI for the boosted decision-tree algorithm machine learning PSI model using the Extreme Gradient Boosting algorithm [mlPSI] with the 19 original PSI factors, 0.83-0.85) and the number of variables (CI for the logistic regression PSI model using all 69 variables, 0.84-085; CI for the mlPSI with all 69 variables, 0.86-0.87). Models limited to age, sex, and physiologic variables also demonstrated high performance (CI for the mlPSI with age, sex, and 26 physiologic factors, 0.84-0.85). At an ePSI score of 970 and a mortality-risk cutoff of <2.7%, the ePSI score identified 31% of all patients as being at "low risk"; the mlPSI with age, sex, and 26 physiologic factors identified 53% of all patients as being at low risk; and the mlPSI with all 69 variables identified 56% of all patients as being at low risk, with similar rates of mortality, hospitalization, and 7-day secondary hospitalization being determined. Conclusions: Computerized versions of the PSI accurately identified patients with pneumonia who were at low risk of death. More complex models classified more patients as being at low risk of death and as having similar adverse outcomes.
Keywords: clinical prediction models; decision support; machine learning; pneumonia.