Riley, RD, Debray, TPA, Collins, GS, Archer, L, Ensor, J, van Smeden, M and Snell, KIE (2021) Minimum sample size for external validation of a clinical prediction model with a binary outcome. Statistics in Medicine. ISSN 0277-6715

[thumbnail of sim.9025.pdf]
sim.9025.pdf - Published Version
Available under License Creative Commons Attribution.

Download (2MB) | Preview


In prediction model research, external validation is needed to examine an existing model's performance using data independent to that for model development. Current external validation studies often suffer from small sample sizes and consequently imprecise predictive performance estimates. To address this, we propose how to determine the minimum sample size needed for a new external validation study of a prediction model for a binary outcome. Our calculations aim to precisely estimate calibration (Observed/Expected and calibration slope), discrimination (C-statistic), and clinical utility (net benefit). For each measure, we propose closed-form and iterative solutions for calculating the minimum sample size required. These require specifying: (i) target SEs (confidence interval widths) for each estimate of interest, (ii) the anticipated outcome event proportion in the validation population, (iii) the prediction model's anticipated (mis)calibration and variance of linear predictor values in the validation population, and (iv) potential risk thresholds for clinical decision-making. The calculations can also be used to inform whether the sample size of an existing (already collected) dataset is adequate for external validation. We illustrate our proposal for external validation of a prediction model for mechanical heart valve failure with an expected outcome event proportion of 0.018. Calculations suggest at least 9835 participants (177 events) are required to precisely estimate the calibration and discrimination measures, with this number driven by the calibration slope criterion, which we anticipate will often be the case. Also, 6443 participants (116 events) are required to precisely estimate net benefit at a risk threshold of 8%. Software code is provided.

Item Type: Article
Additional Information: © 2021 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
Subjects: R Medicine > R Medicine (General)
R Medicine > R Medicine (General) > R735 Medical education. Medical schools. Research
Divisions: Faculty of Medicine and Health Sciences > School of Medicine
Related URLs:
Depositing User: Symplectic
Date Deposited: 11 Jun 2021 08:38
Last Modified: 11 Jun 2021 08:38

Actions (login required)

View Item
View Item