Freedman–Bernstein musculoskeletal competence testing of South African intern doctors: is there a difference between health science faculties?

Background: Basic competency in musculoskeletal medicine is necessary for all graduating doctors due to the growing burden of disease. Globally and nationally research has shown deficiencies in musculoskeletal knowledge according to the Freedman–Bernstein test. In South Africa, different health science faculties show different approaches to training; this article considers if any of these demonstrate adequate training and whether significant differences exist between the faculties’ results. The aim of the study is to determine whether there are significant differences between musculoskeletal learning outcomes between graduates from different health science faculties.


Introduction
The prevalence and cost of treating musculoskeletal disease have increased dramatically over the last several decades. 1,2 Musculoskeletal pathology also accounts for a substantial proportion of the global burden of disease. 3 Patients attending primary healthcare facilities in Cape Town showed a 36% prevalence of musculoskeletal conditions. 4 With this increasing burden of disease, basic competence in the management of musculoskeletal conditions should be considered an essential prerequisite for graduating South African medical students. Deficiencies have been highlighted in the ability of junior doctors to manage this musculoskeletal burden. Naidoo showed the poor quality of orthopaedic referrals to a tertiary hospital in KwaZulu-Natal. Basic principles of acute orthopaedic intervention were shown to be lacking, including splinting, analgesia and the handling of compound fractures. 5 A study of community service doctors in rural KwaZulu-Natal showed skills and knowledge in orthopaedics to be inadequate, and management of major joint dislocations to be poor. 6 Undergraduate orthopaedic training was also shown to be deficient in South Africa when a study showed junior doctors requested further basic orthopaedic training to fill the voids in their skill set. Half of the topics chosen by South African doctors were orthopaedic essential practical skills such as wrist and ankle fracture reduction techniques, and management of open fractures. 3 Globally, deficiencies in musculoskeletal training and knowledge have also been highlighted. In 1998 Freedman and Bernstein designed and validated a musculoskeletal competency test. 7 This 25 mark short-answer question paper was developed in the United States of America to test how well medical school graduates understood basic musculoskeletal problems and has a pass mark of 73% or 70%, validated by the chairs of both orthopaedic and internal medicine residency programmes respectively. 7 Multiple global studies have subsequently shown failure rates of 74-94% with this test. [8][9][10][11] Dachs et al. showed that deficiencies in undergraduate musculoskeletal training were evident when 72 of the 79 interns (91%) from South African health science faculties failed the Freedman-Bernstein test. 12 This deficiency was perceived to be due to insufficient training time allocation at undergraduate level, with Dachs et al. showing only 2% of training time (four to six weeks) dedicated to musculoskeletal medicine in South African health science faculties, 13 a sentiment shared globally in multiple studies. 7,14,15 South African universities differ with regard to curricular approach (systems approach/problem-based learning/mixed), programme length, entrance criteria, resources, setting (rural/urban) and teaching methods. [16][17][18] The common goal is to create competent doctors to serve our population. The aim of this study was to determine whether the different faculties' training programmes resulted in different levels of musculoskeletal knowledge.

Methods
A multi-centre cross-sectional study was performed by administering the Freedman-Bernstein musculoskeletal competence test to qualified South African intern doctors prior to starting their formal orthopaedic rotation, with the aim of testing knowledge acquired as an undergraduate. Interns were enrolled at Frere Hospital over a three-year period from January 2017 to January 2020 by the author and co-author after their orthopaedic block orientation session. Similarly, interns were also enrolled in January 2020 from other training hospitals in Gauteng, KwaZulu-Natal and the Eastern Cape. This was done to increase sample size and overcome selection bias. Intern doctors are allocated to hospitals by the national intern allocation process and we obtained representation from all health science faculties at each site. Invigilators were recruited from other centres by approaching the intern curator and orthopaedic head of department and arranging with a consultant or registrar to act as an invigilator. After a verbal informed consent, all tests were administered under invigilated examination conditions with no time limit. The tests were anonymous and voluntary, asking only for health science faculty name and year of qualification. Only graduates from South Africa's eight health science faculties were enrolled. The population consisted of first-and second-year interns. Once the tests were completed, and invigilators had couriered their tests to East London, they were marked by the author with the validated scoring system and answer key published in Freedman and Bernstein's original article (Table I), and results were recorded on Excel data sheet tables. The pass mark of 70% was chosen as suggested by the chairs of internal medicine for our study. 7 Statistical analysis was done using IBM SPSS Statistics version 25 to estimate means and their binomial exact 95% confidence intervals. A one-way analysis of variance (ANOVA) was then performed to determine whether statistically significant differences existed between the health science faculty groups' mean scores. Once significance was confirmed then multiple comparisons between each faculty and the rest were done, with use of post hoc tests and the Bonferroni adjustment, to check for significance of differences between specific faculties (p<0.05).

Combined test analysis
A total of 262 intern doctors completed tests; three of the doctors had graduated from international medical schools and were excluded. Of the 259 included tests, 203 were from Frere Hospital and 56 were from the other centres. The mean score for the 259 interns was 46% (95% CI 44-48%). Only 15 out of 259 interns (6%) achieved a score of 70% or more and therefore showed basic competence in the test, thus demonstrating a 94% failure rate.

Comparison between health science faculties
The health science faculties were anonymised by numbering them 1 to 8 in order of ascending sample sizes. Sample sizes ranged from 12 interns to 67 interns. The mean percentage of each health science faculty's graduate group was then calculated (Table II, Figure 1). The lowest performing health science faculty group mean was 34% (95% CI 28-40%) and the highest 60% (95% CI 55 -64%). In terms of failure rates, half of the health science faculties recorded 100% failure rates. Of the health science faculties that recorded passing scores by graduates, the pass rate ranged from 2% to 22% ( Figure 2). The ANOVA revealed a statistically significant difference between health science graduate groups (p<0.001). The post hoc tests and Bonferroni correction do an additional 28 comparisons of the health science faculties against one another to check for significance. University 4 had a mean score significantly higher than each of the other seven faculties and health science faculty 3 had a significantly lower mean than each of the top three performing faculties (Figure 3).

Discussion
The results of our study show that there has not been a demonstrable improvement in musculoskeletal competence among graduates from our health science faculties from previous literature. Our mean score of 46% (95% CI 44-48%) and failure rate of 94% are consistent with the work of Dachs et al., ten years ago, of 45% The primary aim of our study was to determine whether significant differences existed between the musculoskeletal knowledge of health science faculty graduates. We have shown that statistically significant differences exist, with a difference between highest and lowest averages reaching 25%. Of the interns who passed the test, 13 out of 15 (87%) came from the top two health science faculties in our study.
The recently published Academy of Science of South Africa (ASSAf) report suggests that core competencies need to be developed by regulatory bodies, and that competency-based education should be subscribed to by health science faculties. 19 Botha, in her dissertation on curricular mapping of undergraduate medical programmes, showed that inter-institution differences were recognised and there is a need for collaboration and transparency between health science faculties to improve the curriculum. 17 Marais and Dunn also noted that with the concurrent service demands and training time constraints, we need to review the way we educate to compensate for the expanding field of orthopaedics. 20 We have found in the literature that there is a drive to align, standardise and improve undergraduate training programmes.
We therefore performed a comparison of the training programmes with the highest and lowest mean scores to investigate possible reasons for these significant differences. We conducted telephonic interviews with the two lowest scoring health science faculties (mean scores 34% and 37%) and the two highest scoring faculties (60% and 49%) to obtain details on their undergraduate orthopaedic training programmes and look for similarities and differences which   Figure 2. Pass/fail percentages of each health science faculty's graduates could possibly account for differences in their test scores. All four of these faculties generally followed a similar structure of exposing students to theoretical training in the preclinical years, followed by clinical rotations in orthopaedics at training hospitals. The faculty with the highest mean value and pass rate was interestingly a five-year programme. There were small differences in total musculoskeletal training time in the clinical years between the four faculties, all consisting of between four to six weeks, either running concurrently or divided up between the last three years of study. The preclinical musculoskeletal exposure time differed between faculties. The top two performing faculties, which both had traditional teaching models, had longer formal preclinical rotations; one consisted of a six-week consecutive block, the other a full year of regular dissection-based anatomy training and didactic lectures, more in keeping with the recommendations of Woolf et al. 21 When looking at how the clinical teaching programme was structured, all four programmes generally followed a similar framework and timetable of teaching. All four programmes consisted of varying amounts of exposure to the essential elements of orthopaedic departments such as morning meetings, clinics, theatre, ward rounds, patient examination, after-hours duty, and all had small group tutorials scheduled. With similar teaching structures, one must consider that other factors may account for test score differences, such as perhaps enforcement of the programme or the quality of teaching received, as suggested in the discussion of Weiss et al.'s article. 22 An interesting distinguishing feature of the top performing faculty was the frequency with which they assessed their students, which was significantly more than any other faculty. Their students complete orthopaedic blocks in each of their final three years (two weeks each) and need to pass examinations in the form of objective structured clinical examinations (OSCEs), multiple-choice questions (MCQs) and clinical exams, occurring at the end of each block and again at the end of the year. They are therefore examined eight times on musculoskeletal course work before graduation. This requires revision of course material multiple times and may account for their higher test scores. With examinations occurring frequently and at different stages in their training, students who are struggling can be easily identified. These students are then either approached by faculty members or they can identify themselves as needing more help. Those students are then given additional individual tutorials to assist them with their learning, a method which has proven to increase musculoskeletal competence in the literature. 3,23 The ASSAF report supports this method by emphasising the need for adequate assessment tools to be incorporated into the syllabus that will guide the development of core competencies and are essential to enable best training practices to be developed. 19 With regard to time dedicated to musculoskeletal training, the literature shows that increasing training time does not always result in improved competency scores. 22 Woolf et al. suggested that the ideal musculoskeletal training programme should consist of 12 weeks of clinical time and sufficient non-clinical background given before that. 21 With all departments jostling for time in resourceconstrained undergraduate programmes, it may not be possible to extend clinical training to beyond six weeks. Our health science faculties' results are poor and major reform is needed. Based on analysis of the top performing faculties in our study, we feel their teaching strategies may improve competence levels and help standardise results if adopted by other institutions, and should be the focus of further study.
South Africa's universities differ with regard to setting, resources, demographics, training time, entrance criteria for students and curricular approach. [16][17][18] Some subscribe to a problem-based learning approach, others a more traditional systems approach or a hybrid version of the two. 18 These differences are what make each faculty unique, and they have their advantages and disadvantages. These inherent differences may, however, result in differences in the mean test score of the faculties. Botha highlighted these considerable differences between faculties and indicated the need for collaboration between health science faculties in order to formulate a plan to standardise the outcome of training, the common goal being to produce competent doctors with core competencies who can best serve their community. 17 We feel the significant differences in the outcome of musculoskeletal training shown in our study should prompt efforts to collectively improve training regardless of these inter-institutional differences.
There are limitations to our study. Even though we tried to get a true representation of South African health science faculty graduates by our multi-centred approach and large sample size, there exists the possibility that our sample is not a true representation of the graduate population of the faculty. Bias may have been introduced by the fact that interns select their hospitals of choice and therefore the sample is not truly random. We recruited interns over three years and the possibility exists that changes to the curriculum may have occurred during this time, which could affect our results. Nearly half of our intern sample was obtained from new interns in January 2020; we did not feel a sub-group analysis was necessary to show similarity between these groups. Our tests were administered under invigilated conditions to eliminate cheating, but the result obtained could be biased by interns having been previously exposed to the test questions. Our sample was a mixture of first and second year interns which may have caused bias by the fact that first years were more recently trained and second years may have gained knowledge in other blocks; however, we do not

Conclusion
Our study has confirmed that undergraduate training in South Africa continues to produce graduates without the necessary basic musculoskeletal knowledge to deal with the rising burden of disease. We have furthermore provided evidence that statistically significant differences exist between the undergraduate training programmes. We feel this evidence strengthens the argument for inter-institutional collaboration to reform and standardise musculoskeletal curricula.