SciELO - Scientific Electronic Library Online

 
vol.19 número3The short-term outcome of hip revision arthroplasty with Trabecular Metal™ components and augmentsInfantile Blount's disease índice de autoresíndice de assuntospesquisa de artigos
Home Pagelista alfabética de periódicos  

Serviços Personalizados

Artigo

Indicadores

Links relacionados

  • Em processo de indexaçãoCitado por Google
  • Em processo de indexaçãoSimilares em Google

Compartilhar


SA Orthopaedic Journal

versão On-line ISSN 2309-8309
versão impressa ISSN 1681-150X

SA orthop. j. vol.19 no.3 Centurion Ago./Set. 2020

http://dx.doi.org/10.17159/2309-8309/2020/v19n3a7 

TEACHING AND TRAINING

 

Freedman-Bernstein musculoskeletal competence testing of South African intern doctors: is there a difference between health science faculties?

 

 

Coetzee KPI; Gibson NWII

IMBChB(UCT); Registrar, Orthopaedic Surgery, Frere Hospital, East London; Walter Sisulu University, Mthatha, South Africa
IIMBBCh(Wits), FCOrth(SA); Associate Professor and Head of Department, Orthopaedic Surgery, Frere Hospital, East London; Walter Sisulu University, Mthatha, South Africa

Correspondence

 

 


ABSTRACT

BACKGROUND: Basic competency in musculoskeletal medicine is necessary for all graduating doctors due to the growing burden of disease. Globally and nationally research has shown deficiencies in musculoskeletal knowledge according to the Freedman-Bernstein test. In South Africa, different health science faculties show different approaches to training; this article considers if any of these demonstrate adequate training and whether significant differences exist between the faculties' results. The aim of the study is to determine whether there are significant differences between musculoskeletal learning outcomes between graduates from different health science faculties
METHODS: A multi-centre, cross-sectional study was performed in which medical interns completed the Freedman-Bernstein test after graduation and prior to commencing their formal two-month block in orthopaedics. Data was then analysed to determine whether significant differences existed between the test scores of the eight health science faculties' graduates
RESULTS: A total of 259 completed tests were analysed. The mean score was 46% (range 4-88%, 95% CI 44-48%), and 244 of the 259 interns failed the test (94% failure rate). The lowest and highest mean scores, by health science faculty, was 34% (95% CI 28-40%) and 60% (95% CI 55-64%) respectively. An ANOVA test indicated statistically significant differences between test scores of the different health science faculties (p<0.001
CONCLUSION: We have demonstrated competency scores consistent with previous literature from South Africa and shown that there are statistically significant differences between the health science faculties based on Freedman-Bernstein test scores. This evidence suggests differing levels of musculoskeletal knowledge attained at health science faculties in South Africa, and no improvement in undergraduate education in the last decade
Level of evidence: Level 4

Keywords: musculoskeletal, competence, Freedman-Bernstein, undergraduate training


 

 

Introduction

The prevalence and cost of treating musculoskeletal disease have increased dramatically over the last several decades.1,2 Musculoskeletal pathology also accounts for a substantial proportion of the global burden of disease.3 Patients attending primary healthcare facilities in Cape Town showed a 36% prevalence of musculoskeletal conditions.4

With this increasing burden of disease, basic competence in the management of musculoskeletal conditions should be considered an essential prerequisite for graduating South African medical students. Deficiencies have been highlighted in the ability of junior doctors to manage this musculoskeletal burden. Naidoo showed the poor quality of orthopaedic referrals to a tertiary hospital in KwaZulu-Natal. Basic principles of acute orthopaedic intervention were shown to be lacking, including splinting, analgesia and the handling of compound fractures.5 A study of community service doctors in rural KwaZulu-Natal showed skills and knowledge in orthopaedics to be inadequate, and management of major joint dislocations to be poor.6 Undergraduate orthopaedic training was also shown to be deficient in South Africa when a study showed junior doctors requested further basic orthopaedic training to fill the voids in their skill set. Half of the topics chosen by South African doctors were orthopaedic essential practical skills such as wrist and ankle fracture reduction techniques, and management of open fractures.3

Globally, deficiencies in musculoskeletal training and knowledge have also been highlighted. In 1998 Freedman and Bernstein designed and validated a musculoskeletal competency test.7 This 25 mark short-answer question paper was developed in the United States of America to test how well medical school graduates understood basic musculoskeletal problems and has a pass mark of 73% or 70%, validated by the chairs of both orthopaedic and internal medicine residency programmes respectively.7 Multiple global studies have subsequently shown failure rates of 74-94% with this test.8-11 Dachs et al. showed that deficiencies in undergraduate musculoskeletal training were evident when 72 of the 79 interns (91%) from South African health science faculties failed the Freedman-Bernstein test.12 This deficiency was perceived to be due to insufficient training time allocation at undergraduate level, with Dachs et al. showing only 2% of training time (four to six weeks) dedicated to musculoskeletal medicine in South African health science faculties,13 a sentiment shared globally in multiple studies.7,14,15

South African universities differ with regard to curricular approach (systems approach/problem-based learning/mixed), programme length, entrance criteria, resources, setting (rural/urban) and teaching methods.16-18 The common goal is to create competent doctors to serve our population. The aim of this study was to determine whether the different faculties' training programmes resulted in different levels of musculoskeletal knowledge.

 

Methods

A multi-centre cross-sectional study was performed by administering the Freedman-Bernstein musculoskeletal competence test to qualified South African intern doctors prior to starting their formal orthopaedic rotation, with the aim of testing knowledge acquired as an undergraduate. Interns were enrolled at Frere Hospital over a three-year period from January 2017 to January 2020 by the author and co-author after their orthopaedic block orientation session. Similarly, interns were also enrolled in January 2020 from other training hospitals in Gauteng, KwaZulu-Natal and the Eastern Cape. This was done to increase sample size and overcome selection bias. Intern doctors are allocated to hospitals by the national intern allocation process and we obtained representation from all health science faculties at each site. Invigilators were recruited from other centres by approaching the intern curator and orthopaedic head of department and arranging with a consultant or registrar to act as an invigilator. After a verbal informed consent, all tests were administered under invigilated examination conditions with no time limit. The tests were anonymous and voluntary, asking only for health science faculty name and year of qualification. Only graduates from South Africa's eight health science faculties were enrolled. The population consisted of first- and second-year interns. Once the tests were completed, and invigilators had couriered their tests to East London, they were marked by the author with the validated scoring system and answer key published in Freedman and Bernstein's original article (Table I), and results were recorded on Excel data sheet tables. The pass mark of 70% was chosen as suggested by the chairs of internal medicine for our study.7 Statistical analysis was done using IBM SPSS Statistics version 25 to estimate means and their binomial exact 95% confidence intervals. A one-way analysis of variance (ANOVA) was then performed to determine whether statistically significant differences existed between the health science faculty groups' mean scores. Once significance was confirmed then multiple comparisons between each faculty and the rest were done, with use of post hoc tests and the Bonferroni adjustment, to check for significance of differences between specific faculties (p<0.05).

 

Results

Combined test analysis

A total of 262 intern doctors completed tests; three of the doctors had graduated from international medical schools and were excluded. Of the 259 included tests, 203 were from Frere Hospital and 56 were from the other centres. The mean score for the 259 interns was 46% (95% CI 44-48%). Only 15 out of 259 interns (6%) achieved a score of 70% or more and therefore showed basic competence in the test, thus demonstrating a 94% failure rate.

Comparison between health science faculties

The health science faculties were anonymised by numbering them 1 to 8 in order of ascending sample sizes. Sample sizes ranged from 12 interns to 67 interns. The mean percentage of each health science faculty's graduate group was then calculated (Table II, Figure 1). The lowest performing health science faculty group mean was 34% (95% CI 28-40%) and the highest 60% (95% CI 55 -64%). In terms of failure rates, half of the health science faculties recorded 100% failure rates. Of the health science faculties that recorded passing scores by graduates, the pass rate ranged from 2% to 22% (Figure 2). The ANOVA revealed a statistically significant difference between health science graduate groups (p<0.001). The post hoc tests and Bonferroni correction do an additional 28 comparisons of the health science faculties against one another to check for significance. University 4 had a mean score significantly higher than each of the other seven faculties and health science faculty 3 had a significantly lower mean than each of the top three performing faculties (Figure 3).

 

 

 

 

Discussion

The results of our study show that there has not been a demonstrable improvement in musculoskeletal competence among graduates from our health science faculties from previous literature. Our mean score of 46% (95% CI 44-48%) and failure rate of 94% are consistent with the work of Dachs et al., ten years ago, of 45% (95% CI 42-48%) and 91% respectively.12 This shows a concerning trend that, in the face of a rising musculoskeletal burden of disease, our junior doctors' competency scores have not increased and the recommended curricular reform and attention to musculoskeletal training has not been implemented. The 94% failure rate in our study is relatively high when compared to Freedman and Bernstein's original rate of 82% and to global studies on graduates in Nepal and the United Kingdom, where failure rates of 94% and 79% respectively were demonstrated.7,8,10 Furthermore, only half of our health science faculties managed to produce graduates who could pass the Freedman-Bernstein test.

The primary aim of our study was to determine whether significant differences existed between the musculoskeletal knowledge of health science faculty graduates. We have shown that statistically significant differences exist, with a difference between highest and lowest averages reaching 25%. Of the interns who passed the test, 13 out of 15 (87%) came from the top two health science faculties in our study.

The recently published Academy of Science of South Africa (ASSAf) report suggests that core competencies need to be developed by regulatory bodies, and that competency-based education should be subscribed to by health science faculties.19 Botha, in her dissertation on curricular mapping of undergraduate medical programmes, showed that inter-institution differences were recognised and there is a need for collaboration and transparency between health science faculties to improve the curriculum.17 Marais and Dunn also noted that with the concurrent service demands and training time constraints, we need to review the way we educate to compensate for the expanding field of orthopaedics.20 We have found in the literature that there is a drive to align, standardise and improve undergraduate training programmes.

We therefore performed a comparison of the training programmes with the highest and lowest mean scores to investigate possible reasons for these significant differences. We conducted telephonic interviews with the two lowest scoring health science faculties (mean scores 34% and 37%) and the two highest scoring faculties (60% and 49%) to obtain details on their undergraduate orthopaedic training programmes and look for similarities and differences which could possibly account for differences in their test scores. All four of these faculties generally followed a similar structure of exposing students to theoretical training in the preclinical years, followed by clinical rotations in orthopaedics at training hospitals. The faculty with the highest mean value and pass rate was interestingly a five-year programme. There were small differences in total musculoskeletal training time in the clinical years between the four faculties, all consisting of between four to six weeks, either running concurrently or divided up between the last three years of study. The preclinical musculoskeletal exposure time differed between faculties. The top two performing faculties, which both had traditional teaching models, had longer formal preclinical rotations; one consisted of a six-week consecutive block, the other a full year of regular dissection-based anatomy training and didactic lectures, more in keeping with the recommendations of Woolf et al.21When looking at how the clinical teaching programme was structured, all four programmes generally followed a similar framework and timetable of teaching. All four programmes consisted of varying amounts of exposure to the essential elements of orthopaedic departments such as morning meetings, clinics, theatre, ward rounds, patient examination, after-hours duty, and all had small group tutorials scheduled. With similar teaching structures, one must consider that other factors may account for test score differences, such as perhaps enforcement of the programme or the quality of teaching received, as suggested in the discussion of Weiss et al.'s article.22 An interesting distinguishing feature of the top performing faculty was the frequency with which they assessed their students, which was significantly more than any other faculty. Their students complete orthopaedic blocks in each of their final three years (two weeks each) and need to pass examinations in the form of objective structured clinical examinations (OSCEs), multiple-choice questions (MCQs) and clinical exams, occurring at the end of each block and again at the end of the year. They are therefore examined eight times on musculoskeletal course work before graduation. This requires revision of course material multiple times and may account for their higher test scores. With examinations occurring frequently and at different stages in their training, students who are struggling can be easily identified. These students are then either approached by faculty members or they can identify themselves as needing more help. Those students are then given additional individual tutorials to assist them with their learning, a method which has proven to increase musculoskeletal competence in the literature.3,23 The ASSAF report supports this method by emphasising the need for adequate assessment tools to be incorporated into the syllabus that will guide the development of core competencies and are essential to enable best training practices to be developed.19

With regard to time dedicated to musculoskeletal training, the literature shows that increasing training time does not always result in improved competency scores.22 Woolf et al. suggested that the ideal musculoskeletal training programme should consist of 12 weeks of clinical time and sufficient non-clinical background given before that.21 With all departments jostling for time in resource-constrained undergraduate programmes, it may not be possible to extend clinical training to beyond six weeks. Our health science faculties' results are poor and major reform is needed. Based on analysis of the top performing faculties in our study, we feel their teaching strategies may improve competence levels and help standardise results if adopted by other institutions, and should be the focus of further study.

South Africa's universities differ with regard to setting, resources, demographics, training time, entrance criteria for students and curricular approach.16-18 Some subscribe to a problem-based learning approach, others a more traditional systems approach or a hybrid version of the two.18 These differences are what make each faculty unique, and they have their advantages and disadvantages. These inherent differences may, however, result in differences in the mean test score of the faculties. Botha highlighted these considerable differences between faculties and indicated the need for collaboration between health science faculties in order to formulate a plan to standardise the outcome of training, the common goal being to produce competent doctors with core competencies who can best serve their community.17 We feel the significant differences in the outcome of musculoskeletal training shown in our study should prompt efforts to collectively improve training regardless of these inter-institutional differences.

There are limitations to our study. Even though we tried to get a true representation of South African health science faculty graduates by our multi-centred approach and large sample size, there exists the possibility that our sample is not a true representation of the graduate population of the faculty. Bias may have been introduced by the fact that interns select their hospitals of choice and therefore the sample is not truly random. We recruited interns over three years and the possibility exists that changes to the curriculum may have occurred during this time, which could affect our results. Nearly half of our intern sample was obtained from new interns in January 2020; we did not feel a sub-group analysis was necessary to show similarity between these groups. Our tests were administered under invigilated conditions to eliminate cheating, but the result obtained could be biased by interns having been previously exposed to the test questions. Our sample was a mixture of first and second year interns which may have caused bias by the fact that first years were more recently trained and second years may have gained knowledge in other blocks; however, we do not think this would have altered our result. The Freedman-Bernstein test, although validated to test basic musculoskeletal competence, does not assess clinical skills and confidence, which are other aspects of competence that also need due consideration, perhaps in further studies.

 

Conclusion

Our study has confirmed that undergraduate training in South Africa continues to produce graduates without the necessary basic musculoskeletal knowledge to deal with the rising burden of disease. We have furthermore provided evidence that statistically significant differences exist between the undergraduate training programmes. We feel this evidence strengthens the argument for inter-institutional collaboration to reform and standardise musculoskeletal curricula.

Acknowledgements

We would like to thank Dr Pieter Maré, Dr Jason du Plessis, Dr Jason van Heerden and Dr Jaco Viljoen who enthusiastically identified interns at their institutions and helped in printing, obtaining consent, administering the tests and then couriering the test papers to us. Statistical analysis by Mrs Marike Cockeran, MSc Statistics, North-West University, Potchefstroom, South Africa.

Ethics statement

The authors declare that this submission is in accordance with the principles laid down by the Responsible Research Publication Position Statements as developed at the 2nd World Conference on Research Integrity in Singapore, 2010. Study approval was obtained from the Walter Sisulu University Faculty of Health Sciences Postgraduate Education, Training, Research and Ethics Unit: protocol number 072/2019. Informed consent was obtained from all individual participants included in the study. All procedures were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008.

Declaration

The authors declare authorship of this article and that they have followed sound scientific research practice. This research is original and does not transgress plagiarism policies.

Author contributions

KPC contributed to the conceptualisation, design, data collection, data analysis and manuscript preparation.

NWG contributed to conceptualisation, design, data collection, manuscript preparation and supervised the study.

ORCID

Coetzee KP https://orcid.org/0000-0003-0261-4447

Gibson NW https://orcid.org/0000-0003-1546-8485

 

References

1. Global Alliance for Musculoskeletal Health [Interntet]. Available from: http://bjdonline.org/ [Accessed 21 January 2020).         [ Links ]

2. Day CS, Ho P. Progress of medical school musculoskeletal education in the 21st century. J Am Acad Orthop Surg. 2016;24:762-68. https://doi.org/10.5435/JAAOS-D-15-00577.         [ Links ]

3. Kalraiya A, Buddhdev P. The TROJAN project: creating a customized international orthopedic training program for junior doctors. Orthop Rev (Pavia). 2015;7:6-9. https://doi.org/10.4081/or.2015.5750.         [ Links ]

4. Parker R, Jelsma J. The prevalence and functional impact of musculoskeletal conditions amongst clients of a primary health care facility in an under-resourced area of Cape Town. BMC Musculoskelet Disord. 2010;11:2.         [ Links ]

5. Naidoo ND. An evaluation of the quality of orthopaedic trauma referrals to a regional hospital. SA Orthop J. 2009;8(1):66-69.         [ Links ]

6. Nkabinde TC, Ross A, Reid S, Nkwanyana NM. Internship training adequately prepares South African medical graduates for community service - with exceptions. SAMJ. 2013;103:930-34. https://doi.org/10.7196/SAMJ.6702.         [ Links ]

7. Freedman KB, Bernstein J. The adequacy of medical school education in musculoskeletal medicine. J Bone Jt Surg. 1998;80:1421-27. https://doi.org/10.2106/00004623-199810000-00003.         [ Links ]

8. Al-Nammari SS, Pengas I, Asopa V, Jawad A, Rafferty M, Ramachandran M. The inadequacy of musculoskeletal knowledge in graduating medical students in the United Kingdom. J Bone Jt Surg. 2013;107:1-10. https://doi.org/10.2106/JBJS.E01083.         [ Links ]

9. Al-Nammari SS, James BK, Ramachandran M. The inadequacy of musculoskeletal knowledge after foundation training in the United Kingdom. J Bone Joint Surg Br. 2009;91-B:1413-18. https://doi.org/10.1302/0301-620x.91b11.22445.         [ Links ]

10. Tamrakar R, Basnyat S, Gyawali B, et al. Adequacy of undergraduate orthopedic training at a Nepalese Medical Academy. Med J Shree Birendra Hosp. 2017;15:13-18. https://doi.org/10.3126/mjsbh.v15i2.16112.         [ Links ]

11. Day CS, Yeh AC, Franko O, Ramirez M, Krupat E. Musculoskeletal medicine: An assessment of the attitudes and knowledge of medical students at Harvard medical school. Acad Med. 2007;82:452-57. https://doi.org/10.1097/ACM.0b013e31803ea860.         [ Links ]

12. Dachs R, Roche S, Vrettos B, Macintyre K, Walters J. Assessment of undergraduate orthopaedic training at medical schools in South Africa. SA Orthop J. 2010;9(4):33-37.         [ Links ]

13. Dachs R, Roche S, Vrettos B, et al. Assessing musculoskeletal training in South Africa. SA Orthop J. 2014;13(3):57-63.         [ Links ]

14. Bhatnagar A, Vadhel M, Sethuraman V. Inadequacy of musculoskeletal knowledge among undergraduate medical students. J Orthop Trauma Rehabil. 2015;19:34-38. https://doi.org/10.1016/j.jotr.2014.03.003.         [ Links ]

15. Williams JR. The teaching of trauma and orthopaedic surgery to the undergraduate in the United Kingdom. J Bone Joint Surg Br. 2000;82:627-28.         [ Links ]

16. Burch VC, Sikakana C, Gunston GD, Whittle SR, Murdoch-Eaton D. Pre-university education curriculum reform and the generic learning skills of medical school entrants: lessons learned from South Africa. Educ Heal. 2018;29:1-8. https://doi.org/10.4103/efh.EfH.         [ Links ]

17. Botha GC. The status of curriculum mapping of undergraduate medical programmes in South Africa [dissertation] Stellenbosch University; 2016.         [ Links ]

18. Seggie JL. MBChB curriculum modernisation in South Africa -growing doctors for Africa. African J Heal Prof Educ. 2010;2:8-14.         [ Links ]

19. Academy of Science of South Africa. Consensus study report: Reconceptualising health professions education in South Africa. 2018.         [ Links ]

20. Marais L, Dunn R. Teaching and training in orthopaedics. SA Orthop J. 2017;16(4):15-19. https://doi.org/10.17159/2309-8309/2017/v16n4a1.         [ Links ]

21. Woolf AD, Walsh NE, Akesson K. Global core recommendations for a musculoskeletal undergraduate curriculum. Ann Rheum Dis. 2004;63:517-24. https://doi.org/10.1136/ard.2003.016071.         [ Links ]

22. Weiss K, Curry E, Matzkin E. Assessment of medical school musculoskeletal education. Am J Orthop. 2015;44:E64-67.         [ Links ]

23. Vermaak D, Potgieter J. The evaluation of a new general trauma course with an orthopaedic focus. SA Orthop J. 2012;11(2):68-71.         [ Links ]

 

 

Correspondence:
Dr Kaylem Paul Coetzee
Department of Orthopaedic Surgery, Frere Hospital
Amalinda Main Rd, Amalinda, East London, 5247
tel: (043) 709 2000; email: coetzee.kaylem6@gmail.com

Received: January 2020
Accepted: March 2020
Published: August 2020

 

 

Editor: Prof. Leonard C Marais, University of KwaZulu-Natal, Durban, South Africa
Funding: The study was funded by a Discovery research grant.
Conflict of interest: Both authors declare no conflicts of interest with regard to this study.

Creative Commons License Todo o conteúdo deste periódico, exceto onde está identificado, está licenciado sob uma Licença Creative Commons