Optimal sampling in derivation studies was associated with improved discrimination in external validation for heart failure prognostic models

Abstract

Objectives: The objective of the study was to identify determinants of external validity of prognostic models.

Study design and setting: We systematically searched for studies reporting prognostic models of heart failure (HF) and examined their performance for predicting 30-day death in a cohort of consecutive 3,452 acute HF patients. We applied published critical appraisal tools and examined whether bias or other characteristics of original derivation studies determined model performance.

Results: We identified 224 models from 6,354 eligible studies. The mean c-statistic in the cohort was 0.64 (standard deviation, 0.07). In univariable analyses, only optimal sampling assessed by an adequate and valid description of the sampling frame and recruitment details to collect the population of interest (total score range: 0-2, higher scores indicating lower risk of bias) was associated with high performance (standardized β = 0.25, 95% CI: 0.12 to 0.38, P < 0.001). It was still significant after adjustment for relevant study characteristics, such as data source, scale of study, stage of illness, and study year (standardized β = 0.24, 95% CI: 0.07 to 0.40, P = 0.01).

Conclusion: Optimal sampling representing the gap between the population of interest and the studied population in derivation studies was a key determinant of external validity of HF prognostic models.

Keywords: External validation; Heart failure; Mortality; Prediction; Prognosis; Prognostic model; Study bias; Systematic review.