RESEARCH

 

The Simulation Assessment Tool Limiting Assessment Bias (SATLAB): A student perspective from a South African university

 

 

A Makkink^I; I Adam^II

^IPhD; Department of Emergency Medical Care, Faculty of Health Sciences, University of Johannesburg, South Africa
^IIBHS; Department of Emergency Medical Care, Faculty of Health Sciences, University of Johannesburg, South Africa

Correspondence

 

 

---

ABSTRACT

BACKGROUND. Clinical simulation in healthcare education for teaching, learning and assessment is growing. Debate continues regarding the reliability and validity in simulation-based teaching, learning and assessment and the intersect of the three domains. The Simulation Assessment Tool Limiting Assessment Bias (SATLAB) seeks to bridge the gap between the classroom, the clinical environment and assessment. SATLAB uses competency outcome descriptors specific to an outcome or intervention, ranked on a six-point scale of competence, to define expected student performance for each component of a clinical simulation.
OBJECTIVE. To explore student perceptions of the SATLAB as a tool within simulation-based teaching, learning and assessment.
METHOD. The study followed a quantitative, cross-sectional design that used a purpose-designed online questionnaire hosted on the university's learning management system. Data were collected using purposive sampling from second- to fourth-year students. Quantitative data were analysed using SPSS. A total of 43 responses were received, representing a response rate of 45%.
RESULTS. Overall feedback from student respondents was positive. Respondents indicated that SATLAB addressed several perceived challenges in simulation-based teaching, learning and assessment and helped bridge the gap between the classroom, the clinical environment and the assessment domain.
CONCLUSION. The findings of this study demonstrate that SATLAB is positively perceived, with student buy-in among those exposed to its use. On this basis, SATLAB may be considered a student-centred method of teaching, learning and assessment within clinical simulation.

Keywords: Emergency medical care, clinical simulation, simulation assessment

---

 

 

The use of clinical simulation in healthcare education has grown exponentially, such that simulators, simulation centres and simulation programmes have become the norm. Despite this growth, the published literature on the use of clinical simulation use for prehospital (paramedic) training and assessment remains limited.^[1] The Simulation Assessment Tool Limiting Assessment Bias (SATLAB) was developed to bridge the gap between the classroom, the clinical environment and assessment, thereby improving student engagement in clinical simulation and enhancing the reliability and validity of the assessment.^[2,3] SATLAB uses competency outcome descriptors (COD) specific to an outcome or intervention, ranked on a six-point scale of competence, to define expected student performance for each component of a clinical simulation. Each COD is closely aligned to what would be expected in a clinical case and, consequently, in the clinical simulation mirroring that case. The number of CODs is directly linked to the number of critical components required for a student to demonstrate competency and may vary depending on several factors such, as the case, its complexity and the purpose of the assessment.^[2] Students receive a comprehensive training session on SATLAB in their first year of study and are encouraged to use the CODs during clinical simulations in class, and compare actions with the relevant CODs. As one of the end-users, students are an important factor in the success of any assessment method. To date, there is limited information on student perceptions of the SATLAB. The aim of this study was to explore student perceptions of the SATLAB as a tool within simulation-based teaching, learning and assessment.

 

Methods

The study followed a quantitative, cross-sectional design using a purpose-designed online questionnaire to gather data. The questionnaire was developed using a desktop study and included Likert-type, forced binary and open-ended questions. It was validated by three academic staff members familiar with SATLAB, who piloted and evaluated each question and indicated whether it was suitable for the study. Validators were also able to provide comments on each question. The completed questionnaire was uploaded to the Blackboard learning management system for each of the clinical simulation modules in which SATLAB was used. Participants included second- to fourth-year students enrolled in the four-year Bachelor of Health Sciences degree in Emergency Medicine Care (EMC) and second‑year students registered for a two-year Diploma in EMC. The questionnaire activated in April 2023, and an invitation email was sent via Blackboard to all relevant students together with an information document and consent information. Reminder emails were sent out two, four and six weeks after the initial communication. After eight weeks, the survey was closed in June 2023. Students were able to submit only one set of responses and, once submitted, were unable to access or change their responses. Responses could not be linked to participants, thereby ensuring anonymity. Data were exported from Blackboard to a.csv in Excel and subsequently imported into SPSS (version 28, IBM Corporation, New York, USA) for analysis. No responses were excluded from the dataset and analysis. Descriptive statistics were used to summarise categorical variables.

The study was approved by University of Johannesburg's Faculty of Health Sciences Research Ethics Committee (ref. no. REC-1770-2022).

 

Results

Demographic data

A total of 43 students completed the survey, representing a response rate of 45%. The median age range was 23 years (IQR 4.5; range 20 - 43), with a mean age of 25 years. Respondents were enrolled in the bachelor's degree in EMC and were registered in the second year (28%, n=12), third year (14%, n=6) and fourth year (35%, n=15), as well as in the second year of the EMC Diploma (23%, n=10). Some respondents did not answer all questions; therefore, where relevant, the total responses does not equal 43. The total number of submitted responses is stated, where applicable.

Importance of simulation-based assessment

Most respondents (80%, n=32/40) understood the importance of simulation-based assessment (SBA) and believed that all students should be trained in the SATLAB (80%, n=32/40), which had the potential to decrease stress levels related to SBA (69%, n=27/39).

SATLAB in teaching and learning in the classroom

Most respondents understood that the competency outcome descriptors originate from the clinical environment and how to apply them within the classroom (79%, n=30/38). Respondents also indicated that the language used was easy to understand without too much jargon (81%, n=25/31). Most respondents indicated that debriefing feedback during classroom simulations could be directly linked to the relevant COD (81%, n=26/32) and that the Best Practice CODs were written in a way that made them practically applicable and useful for directing practice within the clinical domain (75%, n=24/32). Two-thirds of respondents (66%, n=19/29) indicated that they used the CODs during class simulations and compared these with the facilitators' comments. Most respondents (86%, n=30/35) also indicated that it would be valuable to score their classmates during class simulations and to compare their results with those of their peers. Notably, six students indicated that they did not have access to the CODs and 42% (n=16/38) indicated that they had access to and used the CODs to guide their learning.

SATLAB in assessment of simulation

Most respondents (64%, n=23/36) indicated that the results from a SATLAB assessment were broken down into individual CODs, allowing them to identify specific areas for improvement. Most respondents (81%, n=26/32) also indicated that the assessment CODs were phrased in a way that allowed them to compare actual performance to what was expected in the simulation. Most students (77%,n=26/34) believed that the mark achieved in a simulation should reflect their competency within the clinical domain. Most respondents (62%, n=21/34) also indicated that SATLAB was applied consistently across academic years. Almost half (49%, n=17/35) of the respondents believed that CODs were only used for assessments with the aim of generating a mark. Most respondents (54%, n=19/35) indicated that they trusted the reliability of SATLAB and that the mark they achieved was generally a good indication of how they thought they had performed. A further 29% (n=10/35) were neutral, while 17% (n=6/35) indicated that they did not necessarily trust the reliability of SATLAB. Most respondents (54%, n=19/35) indicated that the feedback received when querying a mark was appropriate for them to identify their error and address them; 34% (n=12/35) were neutral while only 12% (n=4/35) did not believe that this to be the case. Less than a third (28%, n=12/35) of respondents agreed that they only looked at feedback when they had performed poorly or failed, while 47% (n=20) disagreed with this statement.

The six-point evaluation scale

Most respondents (91%, n=32/35) indicated that the six-point evaluation scale was easy to understand. Most respondents (89%, n=31/35) indicated that the six-point evaluation scale had a logical flow that allowed them to determine how performance was classified. Respondents were divided on whether the negative marking system was fair: 35% (n=12/34) believed it was fair, 32% (n=11/34) were neutral or unsure and 33% (n=11/34) disagreed that it was fair. Despite this, most respondents (86%, n=30/35) indicated that they understood why harm caused to the patient would produce a negative mark.

 

Discussion

The SATLAB is used to assess simulations within the South African paramedic setting, but to date, limited data are available on student perceptions of the tool. Students' trust in SATLAB's reliability, their perception that simulations were a beneficial teaching and learning experience, and that feedback and debriefing were important, align with findings from other studies exploring the benefits of clinical simulation.^[4-6] Our data suggest that SATLAB addresses several limitations of clinical simulation assessment in paramedic simulation, including the apparent disconnect between classroom and clinical domain, concerns about reliability and assessor bias, and the need for improved feedback and debriefing, and addressing the lack of development of improved simulation tools and reflective practice.^[4,6-8]The observed benefits of high levels of student understanding of the assessment tool (SATLAB), perceptions of its fairness, and clear links to both the clinical domain and classroom teaching, learning and assessment are echoed in the literature.^[5,7]

Study limitations

Limitations of the study include that it was limited to one institution and that the sample size was relatively small, which imited the power to test associations and the generalisability of the findings. The use of different facilitators for each academic year may have resulted in inconsistent application of the principles of SATLAB, resulting in differing student perceptions. Additionally, several students left some questions unanswered and this could not be explored further to understand the underlying reasons.

 

Conclusion

The findings of this study demonstrate that SATLAB is positively perceived, with buy-in from the students exposed to its use. On this basis, SATLAB may be considered a student-centred method of teaching, learning and assessment within clinical simulation and further research is needed to better understand SATLAB from other perspectives.

Data availability statement. The data for this study is available from the authors on reasonable request.

Declaration. None.

Acknowledgements. We acknowledge the students who participated in the study and shared their perspectives.

Author contributions. AM and IA conceptualised the study, performed data collection, analysed the data and drafted the manuscript.

Conflicts of interest. None.

Funding. None.

 

References

1. Wheeler B, Dippenaar E. The use of simulation as a teaching modality for paramedic education: a scoping review. Br Paramed J 2020;5(3):31-43. https://doi.org/10.29045/14784726.2020.12.5.3.31.         [ Links ]

2. Makkink AW, Vincent-Lambert C. The Development of SATLAB: A Tool Designed to Limit Bias and Improve Feedback in Simulation-Based Assessment. South African Journal of Prehospital Emergency Care 2020;1(1):26-34. https://doi.org/10.24213/1-1-3024.         [ Links ]

3. Vincent-Lambert C, Bogossian F. A guide for the assessment of clinical competence using simulation. Johannesburg, South Africa: U21 Health Sciences Group; 2017.         [ Links ]

4. Slabber H, Henrico K. Simulated clinical scenarios: The experiences of Emergency Care Practitioner students. South African Journal of Pre-Hospital Emergency Care 2022 2024;3(1):10-18. https://doi.org/10.24213/3-1-4964.         [ Links ]

5. Knox S, Brand C, Sweeney C. Perceptions of paramedic educators on assessments used in the first year of a paramedic programme: a qualitative exploration. BMC Med Educ 2023;23(1). https://doi.org/10.1186/s12909-023-04930-w.         [ Links ]

6. Vincent-Lambert C, Douglas CN. Views of emergency medical care students on the value of simulation for achievement of clinical competence. Afr J Health Prof Educ 2019;11(4):118. https://doi.org/10.7196/AJHPE.2019.v11i4.1041.         [ Links ]

7. Sobuwa S. Perceptions of staff and students of the role of clinical simulation on students' ability to perform academically. Afr J Health Prof Educ 2023;15(3):16-20. https://doi.org/10.7196/ajhpe.2023.v15i3.1677.         [ Links ]

8. Sandy PT, Meyer JT, Oduniyi OS, Mavhandu-Mudzusi AH. Paramedic students' confidence and satisfaction with clinical simulations of an emergency medical care programme in South Africa: A cross-sectional study. Health SA Gesondheid 2021;26. https://doi.org/10.4102/hsag.v26i0.1522.         [ Links ]

 

 

Correspondence:
A Makkink
amakkink@uj.ac.za

Received 1 August 2024
Accepted 22 May 2025