Madeyski, L and Kitchenham, BA (2017) Effect sizes and their variance for AB/BA crossover design studies. EMPIRICAL SOFTWARE ENGINEERING, 23 (4). 1982 - 2017. ISSN 1382-3256

[thumbnail of B Kitchenham - Effect sizes and their variance....pdf]
B Kitchenham - Effect sizes and their variance....pdf - Published Version
Available under License Creative Commons Attribution.

Download (2MB) | Preview


Vegas et al. IEEE Trans Softw Eng 42(2):120:135 (2016) raised concerns about the use of AB/BA crossover designs in empirical software engineering studies. This paper addresses issues related to calculating standardized effect sizes and their variances that were not addressed by the Vegas et al.’s paper. In a repeated measures design such as an AB/BA crossover design each participant uses each method. There are two major implication of this that have not been discussed in the software engineering literature. Firstly, there are potentially two different standardized mean difference effect sizes that can be calculated, depending on whether the mean difference is standardized by the pooled within groups variance or the within-participants variance. Secondly, as for any estimated parameters and also for the purposes of undertaking meta-analysis, it is necessary to calculate the variance of the standardized mean difference effect sizes (which is not the same as the variance of the study). We present the model underlying the AB/BA crossover design and provide two examples to demonstrate how to construct the two standardized mean difference effect sizes and their variances, both from standard descriptive statistics and from the outputs of statistical software. Finally, we discuss the implication of these issues for reporting and planning software engineering experiments. In particular we consider how researchers should choose between a crossover design or a between groups design.

Item Type: Article
Additional Information: © The Author(s) 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Uncontrolled Keywords: Empirical software engineering, Crossover designs, Effect size estimation, Effect size variance estimation, Meta-analysis
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Faculty of Medicine and Health Sciences > Primary Care Health Sciences
Related URLs:
Depositing User: Symplectic
Date Deposited: 18 Sep 2018 14:36
Last Modified: 18 Mar 2019 15:43

Actions (login required)

View Item
View Item