Andaur Navarro, CL, Damen, JAA, Takada, T, Nijman, SWJ, Dhiman, P, Ma, J, Collins, GS, Bajpai, R, Riley, RD ORCID: https://orcid.org/0000-0001-8699-0735, Moons, KGM and Hooft, L (2021) Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ, 375. n2281 - ?.

[img]
Preview
Text
bmj.n2281.full.pdf - Published Version
Available under License Creative Commons Attribution.

Download (252kB) | Preview

Abstract

OBJECTIVE: To assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties. DESIGN: Systematic review. DATA SOURCES: PubMed from 1 January 2018 to 31 December 2019. ELIGIBILITY CRITERIA: Articles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes. REVIEW METHODS: Methodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall). RESULTS: 152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively. CONCLUSION: Most studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice. SYSTEMATIC REVIEW REGISTRATION: PROSPERO CRD42019161764.

Item Type: Article
Additional Information: This article is Open Access, published under the terms of a Creative Commons licence. Please refer back to the article to check the applicable licence (details are in the article Footnotes). Articles published under CC-BY-NC permit non-commercial use, distribution and reproduction in any medium, provided the original work is properly cited. Permission only needs to be obtained for commercial use and can be done via the RightsLink system below. Articles published under the CC-BY permit unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited.
Subjects: R Medicine > R Medicine (General)
Divisions: Faculty of Medicine and Health Sciences > School of Medicine
Related URLs:
Depositing User: Symplectic
Date Deposited: 28 Oct 2021 15:51
Last Modified: 28 Oct 2021 15:51
URI: https://eprints.keele.ac.uk/id/eprint/10203

Actions (login required)

View Item View Item