SciELO - Scientific Electronic Library Online

 
vol.60 issue3Renal replacement therapy and vascular access: the South African situationCOVID-19 in perspective author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Article

Indicators

Related links

  • On index processCited by Google
  • On index processSimilars in Google

Share


South African Journal of Surgery

On-line version ISSN 2078-5151
Print version ISSN 0038-2361

S. Afr. j. surg. vol.60 n.3 Cape Town Sep. 2022

http://dx.doi.org/10.17159/2078-5151/sajs3767 

GENERAL SURGERY

 

Striking a balance between usability and quality control in electronic health records

 

 

C-H KruseI, II; W BekkerII, III; JL BruceII, III; DL ClarkeII, IV

IDepartment of Ophthalmology, Grey's Hospital, South Africa
IIDepartment of Surgery, University of KwaZulu-Natal, South Africa
IIIDepartment of Surgery, Grey's Hospital, South Africa
IVDepartment of Surgery, University of the Witwatersrand, South Africa

Correspondence

 

 


ABSTRACT

BACKGROUND: This project is the first formal usability review of the hybrid electronic medical registry (HEMR) since its implementation in 2012
METHODS: A synchronous usability evaluation by novice operators was followed by a survey of veteran users. The usability evaluation was done by moderated think-aloud interview while completing tasks for a mock patient. The veteran survey was paper-based and focused on satisfaction of the system
RESULTS: A total of 141 comments on system errors were identified by the novice doctors. These consisted of 123 unique problems, of which three were hardware faults and were thus excluded. The identified issues were categorised into errors of control (27%), minimalist (21%), error (17%), match (13%), flexibility, visibility and consistency (9% each), and history (4%). Every unique usability violation was evaluated by the three experts who agreed that 82 of the 141 errors (58%) were valid and applicable. The other 59 items were rejected, not only because of the inability to reproduce some errors or programme shortcomings, but also because a series of "hurdles" were purposely included in the software to decrease cognitive dissonance and reduce error by the users. The survey of veteran users showed high levels of contentment with the system with regards to efficiency, satisfaction and preference
CONCLUSION: Despite many usability complaints by novices, almost half of them were rejected. Although usability in electronic health systems is important, it can often be sacrificed for more imperative aims such as safety, error filtering and clinical decision support

Keywords: hybrid electronic medical registry, usability, evaluation


 

 

Introduction

With the increased use and capacity of smart technology in everyday life, electronic medical records systems are no longer passive repositories, but present an opportunity to integrate human factors engineering concepts into the ergonomics of clinical practice.1,2 This should be done in such a way as to improve clinical outcomes of patients and to support human clinical decision making.3,4 Such systems are known as clinical decision support systems (CDSS). There has been widespread interest in the development of such systems in sub-Saharan Africa as they may well augment scarse clinical resources and may act as a force multiplier in relatively low resource environments.5-7 Unfortunately, poor design often results in systems that are cumbersome and which hamper clinical workflow and decision making, rather than supporting and enabling them.8,9

The Pietermaritzburg Metropolitan Trauma Service (PMTS) developed a bespoke system just under a decade ago and integrated this system into the departmental workflow at our institution. This system is called the hybrid electronic medical record system (HEMR) and is approaching its tenth anniversary and has successfully functioned during this time.10 It has proven invaluable in supporting departmental morbidity and mortality conferences and in underpinning a burgeoning surgical research programme. The system also set out to be a CDSS that helped staff recognise physiology patterns requiring urgent therapeutic intervention. Abnormal physiological parameters prompt a warning from the system. This warning is designed to stimulate staff to reassess the patient and to clinically intervene as appropriate. At the time of its introduction, extensive end user satisfaction analysis was performed. Several generations of house staff have used the system since its inauguration. Attempts to improve the system are ongoing.

The administrative staff interact with end users on a daily basis, undertaking quality control and trouble shooting. This process has resulted in incremental modifications to the user interface. This continuous ad hoc process must be complemented by formal reassessment and client satisfaction reviews to identify areas for potential quality improvement. This project is the first formal usability testing to be performed since the project was initiated in 2012. It is hoped that this review will identify areas for improvement as well as clarify the principles of ergonomic and human factors engineering which underpin the design and integration of this system.

 

Methods

A two-pronged approach to evaluating the HEMR system was employed: usability testing by novice operators was followed by a survey of veteran users. No identifying data were captured or stored for either the usability evaluation or the user survey. This research complied with the tenets of the Declaration of Helsinki and was approved by the Institutional Review Board.

Novice usability assessment

Synchronous (moderated) usability testing11 was performed on the HEMR at Grey's Hospital, a tertiary hospital in Pietermaritzburg, South Africa. The system had been implemented in 2012 for various clinical departments including general, trauma and paediatric surgery. A group of three surgical specialists contributed to preparing, administering and interpreting the evaluations. All three had extensive experience with the HEMR system and were versed in usability principles and testing methods.

The moderated usability test was the testing method of choice for this study since HEMR had been in use for almost a decade. The risk was high that veteran users of the system had by this time become blasé about potential faults and had identified shortcuts and tricks to circumvent most problems. Expert-driven usability inspections, such as heuristic evaluation, pluralistic usability walkthrough and formal usability inspections, were deemed less ideal for this evaluation.11 The evaluation protocol used in this study consisted of a moderated think-aloud where novices speak their views, with or without prompting, as they work though the various exercises.

The HEMR system was designed solely for use by doctors, therefore users new to the system were asked to enlist as evaluators of the record system. These doctors had all undergone formal training to use HEMR, but all had less than two months experience with it and therefore, although subject-matter experts, were considered novices to the system. The senior doctors/specialists prepared a scenario to be used by all participants. The scenario consisted of a realistic mock patient with incidents for admission, two surgical interventions, multiple morbidities/mortality and a discharge.

Each participant was given information pamphlets including a brief overview of the procedure, the list of general usability principles (Figure 1) and asked to complete a consent form. A specialist served as observer for the interview. The observer transcribed the comments of each participant, made an audio recording of the session if consented, prompted and answered questions.

 

 

Spreadsheets were created to organise and store the collected data. Columns were created for the violations as well as the best single-fit usability principle relating to each violation. Each of the three specialists individually analysed the data to ensure accurate mapping of the principles. Disagreements were resolved by discussion or majority vote if a resolution was not forthcoming. The team responsible for database programming and upkeep were tasked with evaluating each usability issue for applicability: whether the comment was reasonable and valid, and whether it was correctable.

Veteran user survey

All experienced users of the HEMR system were asked to complete a satisfaction survey. Doctors were considered experienced users of the system if they had been using it for more than one calendar year. Each participant was approached personally, and the paper-based survey filled in and handed directly to the interviewer. The questionnaire (Figure 2) contained four questions on efficiency, three on satisfaction and three on preference. The data collected were collated and descriptively analysed.

 

Results

Usability assessment

Seven of the eight novice doctors approached, agreed to take part in the usability assessment. Previous experience of the participants with the HEMR system ranged from one day to six weeks. A total of 141 comments relating to system errors were identified. These consisted of 123 unique problems, of which three were hardware faults and did thus not form part of this evaluation. Figure 1 shows how the number of new errors sharply declines as the study continues. By the seventh participant, most errors had already been discovered, and new violations were more difficult to find.

The most common fault category was control, with 38 errors (27%), which points to difficulties in managing data. Minimalist and error made up 29 (21%) and 24 (17%) respectively, showing violations in ease of use and how readily an unforced error can be made in the system. Match of HEMR flow to clinical practice had 18 (13%) errors, while flexibility, visibility and consistency all had nine (6%). The best performing metric was history, with only four (3%) violations, a measure of how effectively storage and retrieval of past medical notes could be done.

Expert panel review and responses

Each unique usability violation in Table I was evaluated by the three experts who agreed that 82 (58%) of the 141 errors were valid and applicable.

All of these relevant 82 usability issues were either immediately corrected or slated for future revision as in Table II.

The other 59 items were rejected, not only because of the inability to reproduce the error or programme shortcomings, but also because a series of "hurdles" were purposely included in the programme to decrease cognitive dissonance and reduce error. Some examples are included in Table III.

Rejections because of intrinsic software shortcomings included the inability for two doctors to make changes to the same patient's notes simultaneously.

Survey

Ten doctors who are veteran users of HEMR completed the survey. One hundred per cent gave 'satisfied' or 'highly satisfied' as answers to the questions on satisfaction with the current system. Ninety-seven per cent of the answers on productivity were also positive. Eighty per cent of answers in the predilection segment showed a preference for HEMR above handwritten or other electronic systems.

 

Discussion

A modern surgical department is dynamic in terms of turnover of patients and throughput of staff. House staff and surgical trainees rotate through the department and need to be trained in the use of this system on a regular basis. This necessitates ongoing training programmes and regular quality control.

This usability assessment has highlighted a number of issues around the integration of a CDSS and electronic medical record (EMR) system into a busy surgical department. There are a number of competing imperatives in the design of a CDSS and EMR system. The first is end user ease and comfort. A system which is cumbersome and difficult to use will generate resistance in the end users and this will result in decreased compliance. This was recognised from the outset and the system was carefully integrated to ensure that the end users were not presented with additional work but, in fact, had pre-existing tasks replaced or simplified by the system. For example, completion of an electronic admission interface was not an additional burden but replaced the clerking, which was initially done by hand and written out.

A complete and accurate clinical record is essential, but this cannot be achieved at the cost of end user comfort. Having said this, end user comfort must not be allowed to override the need to collect key information. This tension is highlighted in the differing responses from the novice and veteran user, which are tabulated in Table III. Novice users often felt that there were unnecessary duplications, whereas the veteran users understood these to be deliberate steps designed to reduce error and ensure compliance.

As a CDSS, the system is also tasked with supporting clinical decisions and assisting doctors to recognise abnormal physiology and to intervene therapeutically as appropriate, this third imperative of the system design requires that automation and ease of use do not replace user cognition but rather augment and prompt it. The divergent answers of the veteran and novice users highlight this tension. Cognitive dissonance is a well-described situation where clinicians can suppress external sensory inputs which do not meet their own preconceived mental construct.34 This results in the misinterpretation of an external environment and can lead to errors in management and unacceptable outcomes. Attempts to reduce this dissonance must take preference over usability.

Ultimately, improvements in usability emerge from of ongoing engagement with end users. This can improve the experience of end users interfacing with the HEMR and it is expected that this will result in increased compliance and completeness of data entry.

 

Conclusion

Despite many usability complaints by novices, many of which have subsequently been corrected, veteran users are highly satisfied with the HEMR system. Although usability in electronic health systems is important, it must be balanced against more imperative aims such as reduction of cognitive dissonance and error reduction.

Key points

Veteran users had high levels of satisfaction, but novice users identified 141 usability issues in the first formal usability review of the HEMR in South Africa since it was implemented in 2012.

Every unique usability violation was evaluated by the three experts who agreed that 82 of the 141 errors (58%) were valid and applicable.

The other 59 items were rejected partly because a series of "hurdles" had purposely been included in the software to decrease cognitive dissonance and reduce error by the users.

Although usability in electronic health systems is important, it can often be sacrificed for more imperative aims such as safety, error filtering and clinical decision support.

Conflict of interest

The authors declare no conflict of interest.

Funding source

None.

Ethical approval

Ethical approval was obtained from the University of KwaZulu-Natal Biomedical Research Ethics Committee (BREC/00002077/2020).

ORCID

C-H Kruse https://orcid.org/0000-0002-8805-8383

W Bekker https://orcid.org/0000-0003-0695-5994

JL Bruce https://orcid.org/0000-0001-8666-4104

DL Clarke https://orcid.org/0000-0002-8467-1455

 

REFERENCES

1. Pamplin J, Nemeth CP, Serio-Melvin ML, et al. Improving clinician decisions and communication in critical care using novel information technology. Mil Med. 2020;185(1-2):e254-61. https://doi.org/10.1093/milmed/usz151.         [ Links ]

2. Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors. 2015;57(5):805-34. https://doi.org/10.1177/0018720815576827.         [ Links ]

3. Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008;15(5):585-600. https://doi.org/10.1197/jamia.M2667.         [ Links ]

4. Laing GL, Bruce JL, Clarke DL. Tick-box admission forms improve the quality of documentation of surgical emergencies, but have limited impact on clinical behaviour. S Afr Med J. 2014;104(6):435-8. https://doi.org/10.7196/SAMJ.7673.         [ Links ]

5. Kavuma M. The usability of electronic medical record systems implemented in sub-Saharan Africa: a literature review of the evidence. JMIR Hum Factors. 2019;6(1):e9317. https://doi.org/10.2196/humanfactors.9317.         [ Links ]

6. Tilahun B, Fritz F. Comprehensive evaluation of electronic medical record system use and user satisfaction at five low-resource setting hospitals in Ethiopia. JMIR Med Inform. 2015;3(2):e22. https://doi.org/10.2196/medinform.4106.         [ Links ]

7. Jawhari B, Ludwick D, Keenan L, Zakus D, Hayward R. Benefits and challenges of EMR implementations in low resource settings: a state-of-the-art review. BMC Med Inform Decis Mak. 2016;16(1):116. https://doi.org/10.1186/s12911-016-0354-8.         [ Links ]

8. Jaspers M, Peute L, Lauteslager A, Bakker P. Pre-post evaluation of physicians' satisfaction with a redesigned electronic medical record system. Stud Health Technol Inform. 2008;136:303-8.         [ Links ]

9. Roman LC, Ancker JS, Johnson SB, Senathirajah Y. Navigation in the electronic health record: a review of the safety and usability literature. J Biomed Inform. 2017;67:69-79. https://doi.org/10.1016/jobi.2017.01.005.         [ Links ]

10. Laing GL, Bruce JL, Skinner DL, et al. Development, implementation and evaluation of a hybrid electronic medical record system specifically designed for a developing world surgical service. World J Surg. 2014;38(6):1388-97. https://doi.org/10.1007/s00268-013-2438-2.         [ Links ]

11. Hollingsed T, Novick DG. Usability inspection methods after 15 years of research and practice. Proc. 25th Annu. ACM Int. Conf. Des. Commun., New York, NY, USA: Association for Computing Machinery; 2007. p. 24955. https://doi.org/10.1145/1297144.1297200.         [ Links ]

 

 

Correspondence:
C-H Kruse
Email: RuralEye@gmail.com

Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License