Willis, BH and Riley, RD (2017) Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice. Statistics in Medicine, 36 (21). 3283 -3301. ISSN 1097-0258

[thumbnail of riley_Willis_sim_2017.pdf]
riley_Willis_sim_2017.pdf - Published Version
Available under License Creative Commons Attribution.

Download (408kB) | Preview


An important question for clinicians appraising a meta-analysis is: are the findings likely to be valid in their own practice-does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity-where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple ('leave-one-out') cross-validation technique, we demonstrate how we may test meta-analysis estimates for statistical validity using a new validation statistic, Vn, and derive its distribution. We compare this with the usual approach of investigating heterogeneity in meta-analyses and demonstrate the link between statistical validity and homogeneity. Using a simulation study, the properties of Vn and the Q statistic are compared for univariate random effects meta-analysis and a tailored meta-regression model, where information from the setting (included as model covariates) is used to calibrate the summary estimate to the setting of application. Their properties are found to be similar when there are 50 studies or more, but for fewer studies Vn has greater power but a higher type 1 error rate than Q. The power and type 1 error rate of Vn are also shown to depend on the within-study variance, between-study variance, study sample size, and the number of studies in the meta-analysis. Finally, we apply Vn to two published meta-analyses and conclude that it usefully augments standard methods when deciding upon the likely validity of summary meta-analysis estimates in clinical practice. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Item Type: Article
Additional Information: This is the final published version of the article (version of record). It first appeared online via Wiley at http://dx.doi.org/10.1002/sim.7372 - please refer to any applicable terms of use of the publisher.
Uncontrolled Keywords: data interpretation, decision making, meta-analysis, models, statistical, validity
Subjects: H Social Sciences > HA Statistics
Divisions: Faculty of Medicine and Health Sciences > Primary Care Health Sciences
Related URLs:
Depositing User: Symplectic
Date Deposited: 24 Aug 2017 13:02
Last Modified: 13 Aug 2018 10:13
URI: https://eprints.keele.ac.uk/id/eprint/3940

Actions (login required)

View Item
View Item