Takada, T, Nijman, S, Denaxas, S, Snell, KIE, Uijl, A, Nguyen, T-L, Asselbergs, FW and Debray, TPA (2021) Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets. Journal of Clinical Epidemiology, 137. 83 - 91. ISSN 1878-5921

[thumbnail of Internal-external cross-validation helped....pdf]
Internal-external cross-validation helped....pdf - Published Version

Download (568kB) | Preview


OBJECTIVE: To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. STUDY DESIGN AND SETTING: We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices. RESULTS: Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. CONCLUSION: In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.

Item Type: Article
Additional Information: © 2021 The Authors. Published by Elsevier Inc. The published version of this article and all relevant information related to it can be found online at; https://www.jclinepi.com/article/S0895-4356(21)00107-4/fulltext https://www.sciencedirect.com/science/article/pii/S0895435621001074
Uncontrolled Keywords: Prediction model, Calibration, Discrimination, Validation, Heterogeneity, Model comparison
Subjects: R Medicine > R Medicine (General)
R Medicine > R Medicine (General) > R735 Medical education. Medical schools. Research
Related URLs:
Depositing User: Symplectic
Date Deposited: 12 May 2021 08:46
Last Modified: 21 Oct 2021 14:28
URI: https://eprints.keele.ac.uk/id/eprint/9550

Actions (login required)

View Item
View Item