Yeates, P ORCID: https://orcid.org/0000-0001-6316-4051, Moult, A ORCID: https://orcid.org/0000-0002-9424-5660, Cope, N ORCID: https://orcid.org/0000-0003-2053-0474, Mccray, G ORCID: https://orcid.org/0000-0002-0728-5171, Fuller, R and McKinley, RK ORCID: https://orcid.org/0000-0002-3684-3435 (2021) Determining influence, interaction and causality of Contrast and Sequence effects in OSCEs. Medical Education, 56 (3). pp. 292-302.

[img] Text
Medical Education - 2021 - Yeates - Determining influence interaction and causality of Contrast and Sequence effects in.pdf - Accepted Version
Restricted to Repository staff only until 10 December 2022.
Available under License Creative Commons Attribution Non-commercial.

Download (1MB)

Abstract

INTRODUCTION: Differential rater function over time (DRIFT) and contrast effects (examiners' scores biased away from the standard of preceding performances) both challenge the fairness of scoring in Objective Structured Clinical Exams (OSCEs). This is important as, under some circumstances, these effects could alter whether some candidates pass or fail assessments. Benefitting from experimental control, this study investigated the causality, operation and interaction of both effects simultaneously for the first time in an OSCE setting.

METHODS: We used secondary analysis of data from an OSCE in which examiners scored embedded videos of student performances interspersed between live students. Embedded video position varied between examiners (early vs late) whilst the standard of preceding performances naturally varied (previous high or low). We examined linear relationships suggestive of DRIFT and contrast effects in all within-OSCE data before comparing the influence and interaction of "Early" vs "Late" and "PreviousHigh" vs "PreviousLow" conditions on embedded video scores.

RESULTS: Linear relationships data did not support the presence of DRIFT or contrast effects. Embedded videos were scored higher early (19.9(19.4-20.5)) vs late (18.6(18.1-19.1), p<0.001) but scores did not differ between PreviousHigh and PreviousLow conditions. The interaction term was non-significant.

CONCLUSIONS: In this instance, the small DRIFT effect we observed on embedded video can be causally attributed to examiner behaviour. Contrast effects appear less ubiquitous than some prior research suggests. Possible mediators of these finding include: OSCE context, detail of task specification, examiners' cognitive load and the distribution of learners' ability. As the operation of these effects appears to vary across contexts, further research is needed to determine the prevalence and mechanisms of contrast and DRIFT effects, so that assessments may be designed in ways which are likely to avoid their occurrence. Quality assurance should monitor for these contextually variable effects in order to ensure OSCE equivalence.

Item Type: Article
Additional Information: The final version of this article and all relevant information related to it, including copyrights, can be found on the publisher website at; https://onlinelibrary.wiley.com/doi/10.1111/medu.14713
Subjects: L Education > LB Theory and practice of education
L Education > LB Theory and practice of education > LB2300 Higher Education
R Medicine > R Medicine (General) > R735 Medical education. Medical schools. Research
Related URLs:
Depositing User: Symplectic
Date Deposited: 21 Dec 2021 16:15
Last Modified: 21 Feb 2022 14:17
URI: https://eprints.keele.ac.uk/id/eprint/10428

Actions (login required)

View Item View Item