Kitchenham, BA, Madeyski, L, Scanniello, G and Gravino, C (2022) The Importance of the Correlation in Crossover Experiments. IEEE Transactions on Software Engineering, 48 (8). 2802 - 2813. ISSN 0098-5589

[thumbnail of The_Importance_of_the_Correlation_in_Crossover_Experiments.pdf]
The_Importance_of_the_Correlation_in_Crossover_Experiments.pdf - Published Version

Download (971kB) | Preview


Context: In empirical software engineering, crossover designs are popular for experiments comparing software engineering techniques that must be undertaken by human participants. However, their value depends on the correlation ( r ) between the outcome measures on the same participants. Software engineering theory emphasizes the importance of individual skill differences, so we would expect the values of r to be relatively high. However, few researchers have reported the values of r . Goal: To investigate the values of r found in software engineering experiments. Method: We undertook simulation studies to investigate the theoretical and empirical properties of r . Then we investigated the values of r observed in 35 software engineering crossover experiments. Results: The level of r obtained by analysing our 35 crossover experiments was small. Estimates based on means, medians, and random effect analysis disagreed but were all between 0.2 and 0.3. As expected, our analyses found large variability among the individual r estimates for small sample sizes, but no indication that r estimates were larger for the experiments with larger sample sizes that exhibited smaller variability. Conclusions: Low observed r values cast doubts on the validity of crossover designs for software engineering experiments. However, if the cause of low r values relates to training limitations or toy tasks, this affects all Software Engineering (SE) experiments involving human participants. For all human-intensive SE experiments, we recommend more intensive training and then tracking the improvement of participants as they practice using specific techniques, before formally testing the effectiveness of the techniques.

Item Type: Article
Additional Information: CCBY - IEEE is not the copyright holder of this material. Please follow the instructions via to obtain full-text articles and stipulations in the API documentation.
Subjects: Q Science > Q Science (General)
T Technology > T Technology (General)
Divisions: Faculty of Natural Sciences > School of Computing and Mathematics
Depositing User: Symplectic
Date Deposited: 22 Sep 2022 14:17
Last Modified: 22 Sep 2022 14:17

Actions (login required)

View Item
View Item