Mamas Mamas m.mamas@keele.ac.uk
Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing models
Mamas
Authors
Abstract
Background
Clinical prediction models (CPMs) are increasingly deployed to support healthcare decisions but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.
Methods
Simulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.
Results
While redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.
Conclusion
This study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.
Acceptance Date | Dec 15, 2016 |
---|---|
Publication Date | Jan 6, 2017 |
Journal | BMC Medical Research Methodology |
Print ISSN | 1471-2288 |
Publisher | Springer Verlag |
Pages | 1-11 |
DOI | https://doi.org/10.1186/s12874-016-0277-1 |
Keywords | clinical prediction models, model aggregation, validation, computer simulation, contextual 8 heterogeneity |
Publisher URL | http://dx.doi.org/10.1186/s12874-016-0277-1 |
Files
Clinical Prediction in Defined Populations Manuscript.docx
(1.1 Mb)
Document
Publisher Licence URL
https://creativecommons.org/licenses/by/4.0/
martin_ 12874_2016_Article_277.pdf
(1.4 Mb)
PDF
Publisher Licence URL
https://creativecommons.org/licenses/by/4.0/
You might also like
Downloadable Citations
About Keele Repository
Administrator e-mail: research.openaccess@keele.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search