Skip to main content

Research Repository

Advanced Search

Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets.

Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets. Thumbnail


Abstract

OBJECTIVE: To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. STUDY DESIGN AND SETTING: We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices. RESULTS: Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. CONCLUSION: In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.

Acceptance Date Mar 29, 2021
Publication Date Apr 6, 2021
Publicly Available Date Mar 28, 2024
Journal Journal of Clinical Epidemiology
Print ISSN 0895-4356
Publisher Elsevier
Pages 83 - 91
DOI https://doi.org/10.1016/j.jclinepi.2021.03.025
Keywords Prediction model, Calibration, Discrimination, Validation, Heterogeneity, Model comparison
Publisher URL https://www.sciencedirect.com/science/article/pii/S0895435621001074

Files




Downloadable Citations