RESEARCH

 

Training and support to improve ICD coding quality: A controlled before-and-after impact evaluation

 

 

R Dyers^{I, II}; G Ward^{III, IV}; S du Plooy^V; S Fourie^VI; J Evans^VII; H Mahomed^{VIII, IX}

^IMB ChB, MSc, MMed, FCPHM (SA); Division of Health Systems and Public Health, Department of Global Health, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
^IIMB ChB, MSc, MMed, FCPHM (SA); Western Cape Government: Health, Cape Town, South Africa
^IIIMB ChB; Western Cape Government: Health, Cape Town, South Africa
^IVMB ChB; School of Public Health and Family Medicine, Faculty of Health Sciences, University of Cape Town, South Africa
^VWestern Cape Government: Health, Cape Town, South Africa
^VIMB ChB, MBA; Western Cape Government: Health, Cape Town, South Africa
^VIIPhD; Western Cape Government: Health, Cape Town, South Africa
^VIIIMB ChB, MMed, PhD; Division of Health Systems and Public Health, Department of Global Health, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
^IXMB ChB, MMed, PhD; Western Cape Government: Health, Cape Town, South Africa

Correspondence

 

 

---

ABSTRACT

BACKGROUND. The proposed National Health Insurance policy for South Africa (SA) requires hospitals to maintain high-quality International Statistical Classification of Diseases (ICD) codes for patient records. While considerable strides had been made to improve ICD coding coverage by digitising the discharge process in the Western Cape Province, further intervention was required to improve data quality. The aim of this controlled before-and-after study was to evaluate the impact of a clinician training and support initiative to improve ICD coding quality.
OBJECTIVE. To compare ICD coding quality between two central hospitals in the Western Cape before and after the implementation of a training and support initiative for clinicians at one of the sites.
METHODS. The difference in differences in data quality between the intervention site and the control site was calculated. Multiple logistic regression was also used to determine the odds of data quality improvement after the intervention and to adjust for potential differences between the groups.
RESULTS. The intervention had a positive impact of 38.0% on ICD coding completeness over and above changes that occurred at the control site. Relative to the baseline, patient records at the intervention site had a 6.6 (95% confidence interval 3.5 - 16.2) adjusted odds ratio of having a complete set of ICD codes for an admission episode after the introduction of the training and support package. The findings on impact on ICD coding accuracy were not significant.
CONCLUSION. There is sufficient pragmatic evidence that a training and support package will have a considerable positive impact on ICD coding completeness in the SA setting.

---

 

 

The health sector in South Africa (SA) uses the World Health Organization (WHO)'s International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10) codes for epidemiological surveillance as well as patient billing. The proposed National Health Insurance (NHI) policy states that diagnosis-related groups (DRGs) will be the mechanism through which provincial and national hospitals purchase services from the national health authority.^[1] The formulation of DRGs, which can roughly be summarised as average cost for similar health conditions, is dependent on accurate and complete ICD coding. Comorbidity and complications increase the costs of managing health conditions at hospital level. The omission of ICD codes from patient records would therefore result in under-costing DRGs and under-resourcing of hospitals. Also, morbidity profiles of hospitals would be incomplete, rendering hospital admission data a poor proxy for the burden of disease for communities in the hospital's drainage area.

While private hospitals have dedicated coders to produce comprehensive sets of ICD codes for patient encounters, clinicians at public hospitals are required to code diagnoses of all inpatients themselves. However, review of the first 18 months of the NHI pilot described the implementation of the ICD system in the public sector as unsatisfactory and in need of strengthening.^[1] In response to this challenge, the Western Cape Government: Health (WCGH) department commissioned a software application for discharge summaries, the electronic Continuity of Care Record (eCCR), to assist clinicians with ICD coding by integrating ICD code browsers, notes and basic coding rules.

However, as another SA study pointed out, the mere introduction of an electronic system may not produce the desired results without engaging and supporting the intended users of the system.^[2] A review of the eCCR pilot showed that while ICD coding coverage was far better than in previous years, the data quality was still inadequate for billing and surveillance purposes.^[3] While 74% of the patient discharge records' primary ICD codes were accurate, only 45% of records had complete sets of the required codes during a pilot at a central hospital in 2013. This study followed on the recommendations from that study for additional training, oversight of junior clinicians and co-ordination of competing processes.

Increasing demands on clinicians make it difficult for them to commit to costly, time-consuming accredited ICD coding courses, although such programmes have been shown to have a positive impact on data quality.^[4,5] Independently of this research, the WCGH introduced a package of support interventions at one of two central hospitals where the eCCR was implemented. The package included orientation to the eCCR, on-site training in the fundamentals of ICD coding, senior review of discharge summaries prepared by junior staff, access to an in-house-developed online ICD coding training course, and on-site support of a case manager designated to support eCCR users in ICD coding.

There is little literature on the impact of training and support on ICD coding quality. Previous research used inter-observer reliability as the standard for quality, but did not appraise codes against the original patient record.^[6,7] In general, research into ICD coding targets dedicated coders and focuses on efficiency and productivity.^[7]

 

Objective

A retrospective evaluation of the impact of the ICD coding support package by comparing data quality before and after the introduction of the package at the intervention site and a control site, each at tertiary level hospitals in the Western Cape Province of SA. The study formed part of a larger evaluation of the eCCR and ICD coding in the Western Cape.

 

Methods

Study design

This was a quasi-experimental study in which the quality of ICD-10 data in the eCCR was assessed before and after the implementation of training and support at the intervention site. ICD-10 data quality was also assessed at a control site to determine the change in data quality over and above changes that may have occurred naturally without the intervention.

Study setting and population

The study was conducted in the internal medicine departments of two central hospitals in the Western Cape. Patient records and data from the eCCR were reviewed for patients who were discharged over periods of 2 months: baseline 1 August - 30 September 2014, and post-intervention 1 November - 31 December 2014. During these periods, it was required that all patients admitted to general internal medicine wards at both hospitals receive discharge summaries prepared using the eCCR.

Sample size

The two-sided Fisher's exact test statistic was used to calculate the sample size for this study. The significance level of the test was targeted at p<0.05. It was hypothesised that ICD quality in the intervention group might improve by 15%, while the control group might change by 5%. Group sample sizes of 160 each were required to achieve 80% power to detect a difference in differences (DID) in group proportions of 0.10 from baseline to post-intervention. Each group sample size was increased by 10% to account for the possibility that original patient records might be missing, bringing the group sample size to 176.

After the eCCR database had been cleaned, 352 records were randomly selected from the intervention and control sites in proportion to the total discharges at baseline and post-intervention.

Before and after sample sizes were weighted according to the total number of patients discharged with the eCCR in each of the study periods so that folders had an equal chance of being randomly selected at each site.

Data collection

Data were extracted from the eCCR database, original patient records and the human resource management information system.

The ICD codes from the eCCR were checked by one investigator (RD) against original patient records at both the intervention and the control sites to maximise the consistency with which the outcome variables were generated. Data quality checks were performed on a 10% random sample of the data to the satisfaction of a co-author (GW). Similar to a method described by Chute et al.,^[8] and as used in the pilot eCCR study,^[3] the primary ICD code for each patient record was reviewed and classified as one of the following:

• Match. The primary diagnosis in the patient record was coded to the highest level of detail available in the ICD-10 Master Industry Table, SA version - June 2013 (SA MIT).

• Partial match. The primary diagnosis in the patient record was within the scope of the medical concept of the chosen primary ICD code descriptor, but not to the highest level of detail available in the ICD-10 SA MIT.

• No match. The primary diagnosis in the patient record was not within the scope of the medical concept(s) of the chosen primary ICD code.

The accuracy of primary ICD codes alone was reported in this study because this is the dominant cost driver in the formulation and selection of DRGs.

The narratives in the discharge summary from the original patient record were used to determine the relevant clinical concepts of the admission episode. This assumed that clinicians summarised the most relevant clinical information in the patient episode. The eCCR discharge summaries were checked for any clinical information that should have been coded as primary or secondary diagnoses. The ordering of the codes was not used to determine coding quality, except where it influenced the primary ICD code. The technical terms relating to ICD coding in the context of this study are defined in Table 1.

Data from the patient records and eCCR were entered onto predesigned data collection forms and then entered directly into a piloted, preformatted Excel 2013 spreadsheet (Microsoft, USA) by the principal investigator (RD). A 10% sample of randomly selected folders was checked by a co-investigator, an expert in ICD coding (GW), to ensure consistency in the application of the rules used by the investigator to derive the outcome data.

Inclusion criteria

Records of inpatients who were discharged, using the eCCR, from the general internal medicine departments at two central hospitals between 1 August 2014 and 30 September 2014 for the baseline period, and between 1 November 2014 and 31 December 2014 for the post-intervention period, were included.

Exclusion criteria

Records of patients who died in hospital prior to discharge, and records of patients for whom the original paper or scanned electronic patient record could not be found after three requests on separate dates, were excluded.

Measurement tools

The International Statistical Classification of Diseases and Related Health Problems, 10th revision (SA version, January 2014), derived from and licensed to SA by the WHO, was used as a reference for checking the accuracy and completeness of ICD codes.^[9] The instructional notes from the Centers for Disease Control, USA, as well as additional notes specific to SA, were used to assist in the appraisal of ICD coding quality. These resources were integrated into the eCCR and were therefore available to clinicians at the intervention site during the study period. Patient data were collected from folders and clinician characteristics from human resources records. The investigators were not blind to the study site or pre-/ post-intervention period when assessing the outcomes in this study.

Statistical analysis

The record of a patient admission was the unit of analysis. If a primary ICD code was classified as a match, as described above, it was regarded as accurate. If all the relevant clinical concepts were represented by at least partially matching ICD codes, a record was regarded as complete. The term ICD coding quality was used to refer collectively to primary ICD code accuracy and coding completeness in order to reduce repetitive statements concerning these two outcome variables.

Data were imported from Excel into Stata version 13.1 (StataCorp, USA) for analysis. Categorical variables were described with proportions and 95% confidence intervals (CIs). Means and 95% CIs and medians and interquartile ranges were calculated for continuous and count variables, respectively. The before and after groups and the intervention and control groups were treated as four independent groups in the analysis. To test for statistically significant differences in patient characteristics between the groups, the χ² statistic was used for categorical data, one-way analysis of variance for normally distributed continuous data, and the Kruskal-Wallis test for non-parametric data.

The impact of the intervention support package was determined by calculating the difference between ICD coding quality pre- and post-intervention at the control site, and then subtracting this answer from the difference between ICD coding quality pre- and postintervention at the intervention site, i.e. the DID.^[10,11] Other than the inspection for CI overlap, significance testing could not be performed on the DID calculation as the outcome measurements were on the overall performance of independent groups as four distinct units rather than the individual patient records. As further recommended by Rohrer et al.,^[10]the odds of the outcome variables in the postintervention group were determined using firstly logistic regression which produced crude odds ratios (ORs), and secondly multiple logistic regression which produced adjusted ORs to account for group differences and patient and clinician characteristics.^[10]

The associations between ICD coding quality and characteristics of both the patient and the discharging clinician have been demonstrated in previous research.^[3] Based on those findings and the assumption that these factors would modify likely ICD code quality, we adjusted the regression model using the patient's age, gender, comorbidity and length of stay in hospital, the clinician's rank, time period relative to the intervention, and study site. The 95% CIs and p-values for the ORs were also reported, p<0.05 being regarded as statistically significant. Clinicians prepared varying numbers of summaries. This introduced a cluster design effect that was adjusted for in the analysis.

Ethics approval

The study was approved by the Health Research Ethics Committee at Stellenbosch University (ref. no. S13/08/137) and was conducted according to accepted and applicable national and international ethical guidelines and principles, including those of the international Declaration of Helsinki, October 2008. Ethics approval included a waiver of patient consent for the patient record review. Permission was obtained from the Provincial Health Research Committee to proceed with the research and to access data from routine systems (ref. no. 2013/RP/140). Patient identifiers were removed prior to analysis and reporting.

 

Results

Included records

None of the 352 records requested from the intervention and control sites had missing folders. None of the patients had died prior to discharge, and therefore all records were included in the analysis. There were no missing data.

Patient and clinician characteristics and associations with ICD coding quality

Descriptive characteristics of patients and clinicians are shown in Tables 2 - 3. Although there appeared to be a greater proportion of females at the intervention site than at the control site, this was not statistically significant (p=0.52). There were no statistically significant differences between the groups for the patient characteristics of age (p=0.31), length of stay (p=0.41) and comorbidity (p=0.30). There were, however, statistically significant differences between the groups in terms of the rank of the discharging clinicians (p<0.01). These differences are also apparent in Tables 2 - 3. None of the associations between patient characteristics and ICD code accuracy were statistically significant in the crude and adjusted analyses. While the association with the clinician rank of 'specialist' appeared significant (Table 4), this is likely to be a spurious finding as only one specialist's discharge summaries were sampled for this study (Tables 2 - 3). The odds of a record being encoded completely decreased by 33% for every additional comorbid condition in the patient (Table 5).

 

 

 

 

Impact of training and support on ICD coding quality DID results

The number of records with accurate primary ICD codes improved slightly from 71.1% (95% CI 61.1 - 80.4) at baseline to 79.1% (95% CI 69.4 - 86.4) after the intervention at the intervention site, while the accuracy of records at the control site remained essentially unchanged. The DID in primary ICD code accuracy between the intervention and control sites was only 6.6% (Fig. 1).

 

 

While the percentage of records with complete codes at the control site improved only slightly from 22.5% (95% CI 14.5 - 34.2) to 25.5% (95% CI 17.9 - 35.0), the intervention site percentages improved considerably from 27.1% (95% CI 18.5 - 37.7) to 68.1% (95% CI 57.8 - 77.0), which translates into a DID of 38.0% (Fig. 2).

 

 

Multiple regression results

Relative to the baseline period, patient records at the intervention site had a 6.6 (95% CI 3.5 - 16.2) adjusted OR of having a complete set of ICD codes for an admission episode after the introduction of the training and support package for ICD coding. This includes adjustment for patient characteristics, clinician characteristics, time period relative to the intervention and study site. However, for the same scenario, the adjusted OR of 1.9 (95% CI 0.97 - 3.6) for accuracy of the primary ICD codes was not statistically significant.

 

Discussion

The results of this study describe the impact of a training and support intervention on the completeness and accuracy of discharge ICD codes generated in an electronic discharge summary application for clinicians. Potential confounding factors that could influence the impact of the intervention were also taken into account.

Despite use of the same criteria and measurement tools, the ICD code quality at both sites during the baseline period of this study was notably lower than that found in research conducted 1 year previously, in which accuracy and completeness were reported as 74% and 45%, respectively.^[3] This may be because the novelty of using an electronic application for discharges had worn off, or because the researchers in the previous study had under-estimated the Hawthorne effect. Surprisingly, the relationships described between ICD coding quality and patient and clinician characteristics in the previous research were not evident in this study. Besides strength of association being very weak and non-significant, there was little change between the ORs in the crude and adjusted analyses of these characteristics (Tables 4 - 5). The significant association with the rank of specialist should be interpreted cautiously, as there was only one specialist at the intervention site whose discharge summaries were sampled for this study. The significant associations move from strongly positive to strongly negative between the crude and adjusted analysis, suggesting instability in this finding.

Indeed, exposure to the training and support intervention had the strongest association with ICD coding completeness. However, the intervention package did not make much difference to the accuracy of primary ICD codes. Given the inherent limitations of the ICD system described by Chute et al.,^[12]accuracy in the region of 75% may be as good as it gets for a clinical setting where discharges are mostly prepared by the most junior clinicians who, besides not having been trained to expert level in ICD coding, are still learning to diagnose and manage complex cases in a tertiary hospital. Other research has suggested that it may be impossible ever to achieve 100% accuracy and completeness in ICD coding owing to the design of the ICD system.^[6,12,13] None of the disease classification systems are able to capture all clinical concepts that are of interest to clinicians. ^[8,13] Differences between descriptors of the ICD coding system and everyday clinical terminology also contribute to inaccurate and incomplete coding.^[12,14]

As has been the case in previous research, this study showed that increasing comorbidity had a negative association with the quality of ICD codes, possibly owing to the challenge of finding correct terminology for the ICD descriptors in the look-up browser for each additional clinical concept that required encoding.^[3] The addition of SA synonyms for the American terminology used in the ICD code descriptors and help notes to the eCCR since the 2013 study did not seem to have a significant effect on the data quality, though there were reports of improved user experience in the qualitative component of the larger eCCR evaluation study (Dyers et al., unpublished data).

While these results are encouraging in terms of a system-strengthening intervention to improve ICD coding quality, this may still not be of an acceptable standard for the purposes of revenue retrieval and compliance with financial prescripts. Twenty percent of inaccurately coded patient records may negatively impact on DRG costing, resulting in underfunding of services purchased by hospitals from the national health authority as proposed in the NHI policy. Despite the notable impact of the intervention of 38% on the completeness of ICD coding, there is still room for improvement by 32% to ensure that all the required clinical concepts are encoded. For the purpose of initial DRG formulation, this may require the use of expert encoders. Repeated training interventions may also progressively improve coding quality.

Study limitations

There may have been patient, clinician and service confounding variables that were not adjusted for. Although there was a risk of measurement bias in this study due to the investigators not being blinded to the retrospective 'assignment' of patients to the intervention and control groups, efforts were made to apply the same coding rules to all groups consistently by a single observer, i.e. the principal investigator (RD), who had no particular interest in the performance of the intervention package.

The order of ICD codes was not considered for this study. The observed improvement in performance may therefore still not have been according to international coding standards. The use of only one clinical discipline at two central hospitals limits the generalisability of these results. However, this retrospective quasi-experimental evaluation forms part of a province-wide quality improvement cycle from which local policy-makers can draw lessons and be mindful of the caveats to the findings.

While there were imbalances in clinician numbers and characteristics between the groups over the study periods due to clinician rotations and varying team numbers, addressing these by intervening in the work environment or randomising patient and clinician assignment to balance the number of discharges per clinician would have created an artificial scenario and produced results that could never be achieved in the real working world. This research made use of a pragmatic approach to assessing the impact of a system-strengthening intervention where the complexity of the actual healthcare delivery setting was deliberately retained. This resulted in meaningful findings for translation into policy. However, it is acknowledged that the two central hospitals have different histories, university links and cultures, the potential role of which in the findings cannot be completely excluded.

Recommendations

It may not be affordable for managers to introduce the entire training and support package in all clinical departments in all hospitals, the most expensive component of the package being the case manager. However, policy-makers should consider scaling up the less costly components, such as the orientation programme, senior review of discharge summaries prepared by junior staff and access to the online ICD course.

In addition, it may be worthwhile to explore the affordability and cost-effectiveness of incrementally introducing onsite support by designated case managers to clinical areas that treat complex patients and where in-hospital costs are high, e.g. secondary, tertiary and high-care units for obstetrics, paediatrics, general surgery and internal medicine. This may have short-term cost benefits in these areas that could also spill over into other clinical areas in the medium term as clinicians rotate through the various disciplines in their training. As the more stable members of clinical teams, i.e. the senior clinicians, become more comfortable with ICD coding, the improvement in data quality may be sustained and possibly improved in the long term.

Hospital managers are advised to pursue the use of 'checklists, alerts, and predictive tools; embedded clinical guidelines that promote standardized, evidence-based practices; electronic prescribing and test-ordering that reduces errors and redundancy; and discrete data fields that foster use of performance dashboards and compliance reports'.^[15] This should form part of ongoing quality improvement processes for hospital data in general and not just for ICD coding, so that there is coherence and efficiency in the generation of all health service data.

Additional research and innovative monitoring mechanisms that include larger samples of patient records and health facilities over longer periods of time are recommended to get a more reliable picture of ICD coding quality.

 

Conclusion

Despite the inherent limitations of this non-randomised study design, this research provides sufficient pragmatic evidence that training and support had a substantial positive impact on ICD coding quality in an SA hospital setting. Additional research is required to explore the long-term impact, sustainability and cost-effectiveness of this intervention package to support clinicians in generating good-quality data for hospital inpatients.

Acknowledgements. We thank the following people at WCGH for providing us with data for this research: Ian de Vega, Lesley Shand, Nadine Ross, and Noel Weeder and team. Our thanks also go to Frans Vorster, Peter Raubenheimer, Tracey Naledi and Krish Vallabhjee for providing technical support, and to Tonya Esterhuizen for statistical support.

Author contributions. All the authors were involved with the design of the study. RD, GW and SdP were involved with the data collection. SdP developed the eCCR prototype software tool. RD wrote the first draft of the manuscript with the assistance of HM, and SF and JE commented on subsequent drafts.

Funding. None.

Conflicts of interest. None.

 

References

1. Matsoso MP, Fryatt R. National Health Insurance: The first 18 months. S Afr Med J 2013;103(3):156-158. http://dx.doi.org/10.7196/SAMJ.6601        [ Links ]

2. Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: Hard lessons still to be learnt. BMJ 2003;326(7394):860-863. http://dx.doi.org/10.1136/bmj.326.7394.860        [ Links ]

3. Dyers RE, Evans J, Ward GA, Mahomed H. Are central hospitals ready for National Health Insurance? ICD coding quality from an electronic patient discharge tool for clinicians. S Afr Med J 2016;106(2):181-185. http://dx.doi.org/10.7196/SAMJ.2016.v106i2.10079        [ Links ]

4. Lorenzoni L, da Cas R, Aparo UL. The quality of abstracting medical information from the medical record: The impact of training programmes. Int J Qual Health Care 1999;11(3):209-213. http://dx.doi.org/10.1093/intqhc/11.3.209        [ Links ]

5. Groom A. Congratulations! You've passed the coding course. Paper presented at the International Federation of Health Records Organizations Congress, Melbourne, Victoria, 2 - 6 October 2000. Re-printed in ICD Coding Newsletter (November 2000). Melbourne: Victorian ICD Coding Committee and Victorian Department of Human Services, 2000.         [ Links ]

6. Watzlaf VJ, Garvin JH, Moeini S, Anania-Firouzan P. The effectiveness of ICD-10-CM in capturing public health diseases. Perspect Health Inf Manag 2007;4:6.         [ Links ]

7. Stanfill MH, Hsieh KL, Beal K, Fenton SH. Preparing for ICD-10-CM/PCS implementation: Impact on productivity and quality. Perspect Health Inf Manag 2014;11(Summer):1f.         [ Links ]

8. Chute CG, Cohn SP, Campbell KE, Oliver DE, Campbell JR, for the Computer-Based Patient Record Institute's Work Group on Codes & Structures. The content coverage of clinical classifications. J Am Med Inform Assoc 1996;3(3):224-233. http://dx.doi.org/10.1136/jamia.1996.96310636        [ Links ]

9. World Health Organization. International Classification of Diseases. Geneva: WHO, 2012. http://www.who.int/classifications/icd/en/ (accessed 2 June 2012).         [ Links ]

10. Rohrer JE. Nonrandomized impact evaluation studies: Errors and tips. J Prim Care Community Health 2010;1(2):70-72. http://dx.doi.org/10.1177/2150131910374718        [ Links ]

11. Rohrer JE. Quasi-experimental evaluation without regression analysis. J Public Health Manag Pract 2009;15(2):109-111. http://dx.doi.org/10.1097/01.PHH.0000346006.59275.1e        [ Links ]

12. Chute CG, Huff SM, Ferguson JA, Walker JM, Halamka JD. There are important reasons for delaying implementation of the new ICD-10 coding system. Health Aff (Millwood) 2012;31(4):836-842. http://dx.doi.org/10.1377/hlthaff.2011.1258        [ Links ]

13. Chute CG. Clinical classification and terminology: Some history and current observations. J Am Med Inform Assoc 2000;7(3):298-303. http://dx.doi.org/10.1136/jamia.2000.0070298        [ Links ]

14. Jiang G, Pathak J, Chute CG. Formalizing ICD coding rules using Formal Concept Analysis. J Biomed Inform 2009;42(3):504-517. http://dx.doi.org/10.1016/j.jbi.2009.02.005        [ Links ]

15. Silow-Carroll S, Edwards JN, Rodin D. Using electronic health records to improve quality and efficiency: The experiences of leading hospitals. Issue Brief (Commonw Fund) 2012;17(Jul):1-40.         [ Links ]

 

 

Correspondence:
R Dyers
robindyers@me.com

Accepted 15 March 2017