Abstract

DE BEER, V et al. An evaluation of the inter-rater reliability in a clinical skills objective structured clinical examination. Afr. J. Health Prof. Educ. (Online) [online]. 2023, vol.15, n.2, pp.2-6. ISSN 2078-5127.  http://dx.doi.org/10.7196/AJHPE.2023.v15i2.1574.

BACKGROUND: An objective structured clinical examination (OSCE) is a performance-based examination used to assess health sciences students and is a well-recognised tool to assess clinical skills with or without using real patients OBJECTIVES: To determine the inter-rater reliability of experienced and novice assessors from different clinical backgrounds on the final mark allocations during assessment of third-year medical students' final OSCE at the University of the Free State METHODS: This cross-sectional analytical study included 24 assessors and 145 students. After training and written instructions, two assessors per station (urology history taking, respiratory examination and gynaecology skills assessment) each independently assessed the same student for the same skill by completing their individual checklists. At each station, assessors could also give a global rating mark (from 1 to 5) as an overall impression RESULTS: The urology history-taking station had the lowest mean score (53.4%) and the gynaecology skills station the highest (71.1%). Seven (58.3%) of the 12 assessor pairs differed by >5% regarding the final mark, with differences ranging from 5.2% to 12.2%. For two pairs the entire confidence interval (CI) was within the 5% range, whereas for five pairs the entire CI was outside the 5% range. Only one pair achieved substantial agreement (weighted kappa statistic 0.74 - urology history taking). There was no consistency within or across stations regarding whether the experienced or novice assessor gave higher marks. For the respiratory examination and gynaecology skills stations, all pairs differed for the majority of students regarding the global rating mark. Weighted kappa statistics indicated that no pair achieved substantial agreement regarding this mark CONCLUSION: Despite previous experience, written instructions and training in the use of the checklists, differences between assessors were found in most cases

        · text in English     · English ( pdf )