Andaur Navarro, CL, Damen, JAA, Takada, T, Nijman, SWJ, Dhiman, P, Ma, J, Collins, GS, Bajpai, R, Riley, RD, Moons, KGM and Hooft, L (2021) Completeness of reporting of clinical prediction models developed using supervised machine learning: A systematic review. ISSN 0004-6256 (Unpublished)

[thumbnail of 2021.06.28.21259089v1.full.pdf]
2021.06.28.21259089v1.full.pdf - Draft Version

Download (689kB) | Preview


<jats:title>ABSTRACT</jats:title><jats:sec><jats:title>Objective</jats:title><jats:p>While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.</jats:p></jats:sec><jats:sec><jats:title>Study design and setting</jats:title><jats:p>We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields (PROSPERO, CRD42019161764). We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (<jats:ext-link xmlns:xlink="" ext-link-type="uri" xlink:href=""></jats:ext-link>). We measured the overall adherence per article and per TRIPOD item.</jats:p></jats:sec><jats:sec><jats:title>Results</jats:title><jats:p>Our search identified 24 814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4) of TRIPOD items. No articles fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model’s predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).</jats:p></jats:sec><jats:sec><jats:title>Conclusion</jats:title><jats:p>Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.</jats:p></jats:sec><jats:sec><jats:title>What is new?</jats:title><jats:list list-type="bullet"><jats:list-item><jats:p><jats:bold>Key findings:</jats:bold> Similar to prediction model studies developed using regression techniques, machine learning (ML)-based prediction model studies adhered poorly to the TRIPOD statement, the current standard reporting guideline.</jats:p></jats:list-item><jats:list-item><jats:p><jats:bold>What this adds to what is known?</jats:bold> In addition to efforts to improve the completeness of reporting in ML-based prediction model studies, an extension of TRIPOD for these type of studies is needed.</jats:p></jats:list-item><jats:list-item><jats:p><jats:bold>What is the implication, what should change now?</jats:bold> While TRIPOD-AI is under development, we urge authors to follow the recommendations of the TRIPOD statement to improve the completeness of reporting and reduce potential research waste of ML-based prediction model studies.</jats:p></jats:list-item></jats:list></jats:sec>

Item Type: Article
Additional Information: medRxiv preprint doi:; this version posted July 1, 2021. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC 4.0 International license . The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC 4.0 International license.
Subjects: R Medicine > RA Public aspects of medicine
Divisions: Faculty of Medicine and Health Sciences > School of Medicine
Depositing User: Symplectic
Date Deposited: 21 Jul 2021 11:33
Last Modified: 21 Jul 2021 11:33

Actions (login required)

View Item
View Item