Yeates, P and Sebok-Syer, SS (2016) Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students' scores using Many Facet Rasch Modeling. Medical Teacher. 1 -8. ISSN 1466-187X

[img] Text
Hawks Doves Rasch paper Accepted manuscript for Keele repository.pdf - Accepted Version
Restricted to Repository staff only until 29 November 2017.
Available under License Creative Commons Attribution Non-commercial.

Download (281kB)

Abstract

INTRODUCTION: OSCEs are commonly conducted in multiple cycles (different circuits, times, and locations), yet the potential for students' allocation to different OSCE cycles is rarely considered as a source of variance-perhaps in part because conventional psychometrics provide limited insight. METHODS: We used Many Facet Rasch Modeling (MFRM) to estimate the influence of "examiner cohorts" (the combined influence of the examiners in the cycle to which each student was allocated) on students' scores within a fully nested multi-cycle OSCE. RESULTS: Observed average scores for examiners cycles varied by 8.6%, but model-adjusted estimates showed a smaller range of 4.4%. Most students' scores were only slightly altered by the model; the greatest score increase was 5.3%, and greatest score decrease was -3.6%, with 2 students passing who would have failed. DISCUSSION: Despite using 16 examiners per cycle, examiner variability did not completely counter-balance, resulting in an influence of OSCE cycles on students' scores. Assumptions were required for the MFRM analysis; innovative procedures to overcome these limitations and strengthen OSCEs are discussed. CONCLUSIONS: OSCE cycle allocation has the potential to exert a small but unfair influence on students' OSCE scores; these little-considered influences should challenge our assumptions and design of OSCEs.

Item Type: Article
Additional Information: This is the author accepted manuscript (AAM) published in final form as Yeates, P. & Sebok-Syer, S.S., 2016. Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling. Medical Teacher, pp.1–8 and available from Taylor & Francis at http://dx.doi.org/10.1080/0142159x.2017.1248916
Uncontrolled Keywords: medical education research, management, assessment, OSCE, psychometrics
Subjects: R Medicine > R Medicine (General)
Divisions: Faculty of Medicine and Health Sciences > Institute for Science and Technology in Medicine
Related URLs:
Depositing User: Symplectic
Date Deposited: 12 Dec 2016 12:02
Last Modified: 12 Dec 2016 12:04
URI: http://eprints.keele.ac.uk/id/eprint/2606

Actions (login required)

View Item View Item