Snell, KI, Ensor, J ORCID: 0000-0001-7481-0282, Debray, TP, Moons, KG and Riley, RD ORCID: 0000-0001-8699-0735 (2018) Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures? Stat Methods Med Res, 27 (11). pp. 3505-3522. ISSN 1477-0334

[img]
Preview
Text
R Riley - Meta-analysis of prediction model performance across multiple studies....pdf - Published Version
Available under License Creative Commons Attribution.

Download (542kB) | Preview

Abstract

If individual participant data are available from multiple studies or clusters, then a prediction model can be externally validated multiple times. This allows the model's discrimination and calibration performance to be examined across different settings. Random-effects meta-analysis can then be used to quantify overall (average) performance and heterogeneity in performance. This typically assumes a normal distribution of 'true' performance across studies. We conducted a simulation study to examine this normality assumption for various performance measures relating to a logistic regression prediction model. We simulated data across multiple studies with varying degrees of variability in baseline risk or predictor effects and then evaluated the shape of the between-study distribution in the C-statistic, calibration slope, calibration-in-the-large, and E/O statistic, and possible transformations thereof. We found that a normal between-study distribution was usually reasonable for the calibration slope and calibration-in-the-large; however, the distributions of the C-statistic and E/O were often skewed across studies, particularly in settings with large variability in the predictor effects. Normality was vastly improved when using the logit transformation for the C-statistic and the log transformation for E/O, and therefore we recommend these scales to be used for meta-analysis. An illustrated example is given using a random-effects meta-analysis of the performance of QRISK2 across 25 general practices.

Item Type: Article
Additional Information: This is the accepted author manuscript (AAM). The final published version (version of record) is available online via Sage at https://doi.org/10.1177/0962280217705678. Please refer to any applicable terms of use of the publisher.
Uncontrolled Keywords: validation, performance statistics, C-statistic, discrimination, calibration, meta-analysis, between-study distribution, heterogeneity, simulation
Subjects: R Medicine > RA Public aspects of medicine
Divisions: Faculty of Medicine and Health Sciences > Primary Care Health Sciences
Related URLs:
Depositing User: Symplectic
Date Deposited: 01 Jun 2017 09:10
Last Modified: 06 Nov 2018 13:36
URI: http://eprints.keele.ac.uk/id/eprint/3520

Actions (login required)

View Item View Item