Kitchenham, BA, Madeyski, L and Brereton, OP (2019) Problems with Statistical Practice in Software Engineering Research. In: EASE '19 Proceedings of the Evaluation and Assessment on Software Engineering. ACM, pp. 134-143. ISBN 978-1-4503-7145-2

[thumbnail of EASE19CameraReadyAkaSEProblems.pdf]
EASE19CameraReadyAkaSEProblems.pdf - Accepted Version

Download (550kB) | Preview


Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices.

Our goal is to highlight issues contributing to poor statistical practice in human-centric SE experiments.

We reviewed the statistical analysis practices used in the 13 papers that reported families of human-centric SE experiments and were published in high quality journals.

Reviewed papers related to 45 experiments and involved a total of 1303 human participants. We searched for issues that were related to questionable statistical practice that were found in more than one paper. We observed three types of bad practice: incorrect use of terminology, incorrect analysis of repeated measures designs, and post-hoc power testing. We also found two analysis practices (i.e., multiple testing and pre-testing for normality) where statisticians disagree about good practice.

Identified issues pose a problem because readers may expect the statistical methods used in papers published in
top quality, peer-reviewed journals to be correct. We explain why the practices are problematic and provide recommendations for improved practice.

Item Type: Book Section
Additional Information: © The Authors, ACM 2019. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published by ACM at
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Q Science > QA Mathematics > QA76 Computer software
Divisions: Faculty of Natural Sciences > School of Computing and Mathematics
Depositing User: Symplectic
Date Deposited: 01 Apr 2019 13:12
Last Modified: 31 May 2019 01:30

Actions (login required)

View Item
View Item