Martin, G, Mamas, M, Peek, N, Buchan, I and Sperrin, M (2017) Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing models. BMC Medical Research Methodology, 17 (1). pp. 1-11. ISSN 1471-2288

[thumbnail of Clinical Prediction in Defined Populations Manuscript.docx] Text
Clinical Prediction in Defined Populations Manuscript.docx - Accepted Version
Available under License Creative Commons Attribution.

Download (1MB)
[thumbnail of martin_ 12874_2016_Article_277.pdf]
martin_ 12874_2016_Article_277.pdf - Published Version
Available under License Creative Commons Attribution.

Download (1MB) | Preview


Clinical prediction models (CPMs) are increasingly deployed to support healthcare decisions but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.

Simulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.

While redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.

This study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.

Item Type: Article
Uncontrolled Keywords: clinical prediction models, model aggregation, validation, computer simulation, contextual 8 heterogeneity
Subjects: R Medicine > R Medicine (General)
Divisions: Faculty of Medicine and Health Sciences > Institute for Science and Technology in Medicine
Depositing User: Symplectic
Date Deposited: 15 Dec 2016 16:17
Last Modified: 02 Apr 2019 15:19

Actions (login required)

View Item
View Item