Kitchenham, BA, Madeyski, L and Brereton, OP (2019) Problems with Statistical Practice in Software Engineering Research. In: EASE '19 Proceedings of the Evaluation and Assessment on Software Engineering. ACM, pp. 134-143.

[img] Text
EASE19CameraReadyAkaSEProblems.pdf - Accepted Version
Restricted to Repository staff only until 31 May 2019.

Download (550kB)

Abstract

Background
Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices.

Objective
Our goal is to highlight issues contributing to poor statistical practice in human-centric SE experiments.

Method
We reviewed the statistical analysis practices used in the 13 papers that reported families of human-centric SE experiments and were published in high quality journals.

Results
Reviewed papers related to 45 experiments and involved a total of 1303 human participants. We searched for issues that were related to questionable statistical practice that were found in more than one paper. We observed three types of bad practice: incorrect use of terminology, incorrect analysis of repeated measures designs, and post-hoc power testing. We also found two analysis practices (i.e., multiple testing and pre-testing for normality) where statisticians disagree about good practice.

Conclusions
Identified issues pose a problem because readers may expect the statistical methods used in papers published in
top quality, peer-reviewed journals to be correct. We explain why the practices are problematic and provide recommendations for improved practice.

Item Type: Book Section
Additional Information: © The Authors, ACM 2019. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published by ACM at http://doi.org/10.1145/3319008.3319009
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Q Science > QA Mathematics > QA76 Computer software
Divisions: Faculty of Natural Sciences > School of Computing and Maths
Depositing User: Symplectic
Date Deposited: 01 Apr 2019 13:12
Last Modified: 16 Apr 2019 10:53
URI: http://eprints.keele.ac.uk/id/eprint/6124

Actions (login required)

View Item View Item