Skip to main content

Research Repository

Advanced Search

Problems with Statistical Practice in Software Engineering Research

Kitchenham, Barbara; Madeyski, Lech; Brereton, Pearl

Problems with Statistical Practice in Software Engineering Research Thumbnail


Authors

Barbara Kitchenham

Lech Madeyski

Pearl Brereton



Abstract

Background
Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices.

Objective
Our goal is to highlight issues contributing to poor statistical practice in human-centric SE experiments.

Method
We reviewed the statistical analysis practices used in the 13 papers that reported families of human-centric SE experiments and were published in high quality journals.

Results
Reviewed papers related to 45 experiments and involved a total of 1303 human participants. We searched for issues that were related to questionable statistical practice that were found in more than one paper. We observed three types of bad practice: incorrect use of terminology, incorrect analysis of repeated measures designs, and post-hoc power testing. We also found two analysis practices (i.e., multiple testing and pre-testing for normality) where statisticians disagree about good practice.

Conclusions
Identified issues pose a problem because readers may expect the statistical methods used in papers published in
top quality, peer-reviewed journals to be correct. We explain why the practices are problematic and provide recommendations for improved practice.

Conference Name EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering
Conference Location Copenhagen, Denmark
Start Date Apr 15, 2019
End Date Apr 17, 2019
Acceptance Date Feb 28, 2019
Online Publication Date Apr 15, 2019
Publication Date Feb 28, 2019
Publicly Available Date Mar 29, 2024
Pages 134-143
Series Title Evaluation and Assessment in Software Engineering
Book Title EASE '19 Proceedings of the Evaluation and Assessment on Software Engineering
ISBN 978-1-4503-7145-2
DOI https://doi.org/10.1145/3319008.3319009
Publisher URL http://doi.org/10.1145/3319008.3319009

Files




You might also like



Downloadable Citations