Keele Research Repository

<jats:title>ABSTRACT</jats:title><jats:sec><jats:title>Objective</jats:title><jats:p>While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.</jats:p></jats:sec><jats:sec><jats:title>Study design and setting</jats:title><jats:p>We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields (PROSPERO, CRD42019161764). We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (<jats:ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="http://www.TRIPOD-statement.org">www.TRIPOD-statement.org</jats:ext-link>). We measured the overall adherence per article and per TRIPOD item.</jats:p></jats:sec><jats:sec><jats:title>Results</jats:title><jats:p>Our search identified 24 814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4) of TRIPOD items. No articles fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model’s predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).</jats:p></jats:sec><jats:sec><jats:title>Conclusion</jats:title><jats:p>Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.</jats:p></jats:sec><jats:sec><jats:title>What is new?</jats:title><jats:list list-type="bullet"><jats:list-item><jats:p><jats:bold>Key findings:</jats:bold> Similar to prediction model studies developed using regression techniques, machine learning (ML)-based prediction model studies adhered poorly to the TRIPOD statement, the current standard reporting guideline.</jats:p></jats:list-item><jats:list-item><jats:p><jats:bold>What this adds to what is known?</jats:bold> In addition to efforts to improve the completeness of reporting in ML-based prediction model studies, an extension of TRIPOD for these type of studies is needed.</jats:p></jats:list-item><jats:list-item><jats:p><jats:bold>What is the implication, what should change now?</jats:bold> While TRIPOD-AI is under development, we urge authors to follow the recommendations of the TRIPOD statement to improve the completeness of reporting and reduce potential research waste of ML-based prediction model studies.</jats:p></jats:list-item></jats:list></jats:sec>