Olorisade, BK, Brereton, OP and Andras, PE (2017) Reporting Statistical Validity and Model Complexity in Machine Learning based Computational Studies. In: EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE'17). ACM, New York, pp. 128-133. ISBN 9781450348041

[thumbnail of EASE_17_short_paper.pdf]
EASE_17_short_paper.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial.

Download (507kB) | Preview


Background:: Statistical validity and model complexity are both important concepts to enhanced understanding and correctness assessment of computational models. However, information about these are often missing from publications applying machine learning.

Aim: The aim of this study is to show the importance of providing details that can indicate statistical validity and complexity of models in publications. This is explored in the context of citation screening automation using machine learning techniques.

Method: We built 15 Support Vector Machine (SVM) models, each developed using word2vec (average word) features --- and data for 15 review topics from the Drug Evaluation Review Program (DERP) of the Agency for Healthcare Research and Quality (AHRQ).

Results: The word2vec features were found to be sufficiently linearly separable by the SVM and consequently we used the linear kernels. In 11 of the 15 models, the negative (majority) class used over 80% of its training data as support vectors (SVs) and approximately 45% of the positive training data.

Conclusions: In this context, exploring the SVs revealed that the models are overly complex against ideal expectations of not more than 2%-5% (and preferably much less) of the training vectors.

Item Type: Book Section
Additional Information: © The Authors, ACM, 2017. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE'17), http://doi.org/10.1145/3084226.3084283
Uncontrolled Keywords: computer science
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Faculty of Natural Sciences > School of Computing and Mathematics
Depositing User: Symplectic
Date Deposited: 03 Aug 2017 09:24
Last Modified: 28 Feb 2019 14:32
URI: https://eprints.keele.ac.uk/id/eprint/3873

Actions (login required)

View Item
View Item