Skip to main content

Research Repository

Advanced Search

Effect sizes and their variance for AB/BA crossover design studies

Effect sizes and their variance for AB/BA crossover design studies Thumbnail


Abstract

Vegas et al. IEEE Trans Softw Eng 42(2):120:135 (2016) raised concerns about the use of AB/BA crossover designs in empirical software engineering studies. This paper addresses issues related to calculating standardized effect sizes and their variances that were not addressed by the Vegas et al.’s paper. In a repeated measures design such as an AB/BA crossover design each participant uses each method. There are two major implication of this that have not been discussed in the software engineering literature. Firstly, there are potentially two different standardized mean difference effect sizes that can be calculated, depending on whether the mean difference is standardized by the pooled within groups variance or the within-participants variance. Secondly, as for any estimated parameters and also for the purposes of undertaking meta-analysis, it is necessary to calculate the variance of the standardized mean difference effect sizes (which is not the same as the variance of the study). We present the model underlying the AB/BA crossover design and provide two examples to demonstrate how to construct the two standardized mean difference effect sizes and their variances, both from standard descriptive statistics and from the outputs of statistical software. Finally, we discuss the implication of these issues for reporting and planning software engineering experiments. In particular we consider how researchers should choose between a crossover design or a between groups design.

Acceptance Date Dec 6, 2017
Publication Date Dec 6, 2017
Publicly Available Date Mar 29, 2024
Journal EMPIRICAL SOFTWARE ENGINEERING
Print ISSN 1382-3256
Publisher Springer Verlag
Pages 1982 - 2017
DOI https://doi.org/10.1007/s10664-017-9574-5
Keywords Empirical software engineering, Crossover designs, Effect size estimation, Effect size variance estimation, Meta-analysis
Publisher URL https://link.springer.com/article/10.1007%2Fs10664-017-9574-5#copyrightInformation

Files




Downloadable Citations