Keele Research Repository
Explore the Repository
Riley, R, Snell, K, Martin, G, Whittle, R, Archer, L, Sperrin, M and Collins, G (2020) Penalisation and shrinkage methods can produce unreliable clinical prediction models especially when sample size is small. Journal of Clinical Epidemiology, 132. pp. 88-96. ISSN 1878-5921
![[thumbnail of Shrinkage uncertainty - revised SUBMITTED.docx]](https://eprints.keele.ac.uk/style/images/fileicons/text.png)
Shrinkage uncertainty - revised SUBMITTED.docx - Accepted Version
Download (7MB)
Abstract
Objectives
When developing a clinical prediction model, penalization techniques are recommended to address overfitting, as they shrink predictor effect estimates toward the null and reduce mean-square prediction error in new individuals. However, shrinkage and penalty terms (‘tuning parameters’) are estimated with uncertainty from the development data set. We examined the magnitude of this uncertainty and the subsequent impact on prediction model performance.
Study Design and Setting
This study comprises applied examples and a simulation study of the following methods: uniform shrinkage (estimated via a closed-form solution or bootstrapping), ridge regression, the lasso, and elastic net.
Results
In a particular model development data set, penalization methods can be unreliable because tuning parameters are estimated with large uncertainty. This is of most concern when development data sets have a small effective sample size and the model's Cox-Snell is low. The problem can lead to considerable miscalibration of model predictions in new individuals.
Conclusion
Penalization methods are not a ‘carte blanche’; they do not guarantee a reliable prediction model is developed. They are more unreliable when needed most (i.e., when overfitting may be large). We recommend they are best applied with large effective sample sizes, as identified from recent sample size calculations that aim to minimize the potential for model overfitting and precisely estimate key parameters.
Item Type: | Article |
---|---|
Uncontrolled Keywords: | Risk prediction models, Penalization, Shrinkage, Overfitting, Sample size |
Subjects: | R Medicine > RA Public aspects of medicine |
Divisions: | Faculty of Medicine and Health Sciences > School of Medicine |
Related URLs: | |
Depositing User: | Symplectic |
Date Deposited: | 10 Dec 2020 16:54 |
Last Modified: | 11 Jun 2021 11:30 |
URI: | https://eprints.keele.ac.uk/id/eprint/9004 |