Martin, G, Mamas, M ORCID: https://orcid.org/0000-0001-9241-8890, Peek, N, Buchan, I and Sperrin, M (2017) Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing models. BMC Medical Research Methodology, 17 (1). pp. 1-11.

[img] Text
Clinical Prediction in Defined Populations Manuscript.docx - Accepted Version
Available under License Creative Commons Attribution.

Download (1MB)
[img]
Preview
Text
martin_ 12874_2016_Article_277.pdf - Published Version
Available under License Creative Commons Attribution.

Download (1MB) | Preview

Abstract

Background
Clinical prediction models (CPMs) are increasingly deployed to support healthcare decisions but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.

Methods
Simulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.

Results
While redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.

Conclusion
This study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.

Item Type: Article
Uncontrolled Keywords: clinical prediction models, model aggregation, validation, computer simulation, contextual 8 heterogeneity
Subjects: R Medicine > R Medicine (General)
Divisions: Faculty of Medicine and Health Sciences > Institute for Science and Technology in Medicine
Depositing User: Symplectic
Date Deposited: 15 Dec 2016 16:17
Last Modified: 02 Apr 2019 15:19
URI: http://eprints.keele.ac.uk/id/eprint/2670

Actions (login required)

View Item View Item