Assessing the usability of virtual learning environments in higher education

Abstract

Context: E-learning is an integral part of the modern higher education system, and therefore it is essential that students and staff are able to use systems that support E-learning, such as Virtual Learning Environments (VLEs), effectively. Usability is essential to ensure effective use of these systems and is often assessed by means of subjective questions. Although developed mainly for industry use, the Technology Acceptance Model (TAM) and System Usability Scale (SUS) questionnaire are often used to assess E-learning systems.
Goal: The main goal of this thesis is to assess the usability of a VLE platform currently used in universities (Blackboard) and identify the most common and appropriate methods used to assess a VLE platform’s usability. Another aim is to investigate whether there are extensions to common usability models and methods (such as the SUS and TAM) that could improve their accuracy, including the potential of combining them with more objective measures such as number of clicks, time taken and open-ended questions.
Method: The literature on VLE usability evaluation was reviewed using a mapping study methodology to identify the usability methods and factors that have been used previously. Informed by the findings of this study, a set of usability questionnaires have been developed, used and evaluated, with 101 student respondents recruited from all the Schools at Keele University participating in the first study (Chapter Four) and 162 in the second study (Chapter Five). A standard usability questionnaire and a novel form of observation were then combined to record 25 participants’ interactions with the VLE (Chapter Six) while they completed a set of representative tasks in two sessions that were held eight weeks apart. These interactions were then compared.
Results: The results indicate that the VLE performed below the average usability expectation score (SUS score of 62.52) but is still considered as ‘acceptable’. Twenty-seven free text responses were also obtained in the first study and a thematic analysis of comments revealed very negative views of the VLE as well as areas for improvement. In the second study, it was found that perceived enjoyment (PE) and usability were jointly related to the perceived usefulness (PU), although the association was relatively weak. Perceived enjoyment and learnability were jointly associated with perceived ease of use (PEOU), with the association accounting for 39% of the variation in PEOU. Usability was related to PE but learnability was not. Overall, the original TAM can be improved by the addition of learnability, PE and usability as they have a positive effect on TAM. In the final study, the task success rate was relatively high (i.e. 82.3% in session 1); however, an average participant took 3.6 times longer to complete the set of tasks than a competent user. Furthermore, task time, clicks and success rate improved only marginally in the second session (which was at the end of the semester). However, when compared with the analysis of the results from the standard usability questionnaires (subjective measures), participants stated that they were satisfied with the usability of the system, contradicting the objective measures (number of clicks, task time and success rate).
Conclusions: Using subjective measures alone, in the form of standard usability questionnaires, to assess the usability of a complex system can conceal significant issues. Usability assessment should therefore be based on actual performance against a defined baseline and combined with forms of qualitative feedback such as free text responses. Evaluating the effect of usability on E-learning is complicated. The studies conducted in this thesis have provided valuable guidance on how to measure the usability of VLEs. Suggestions for future work on the usability of VLEs as well as appropriate recommendations are provided.