SciELO - Scientific Electronic Library Online

 
vol.106 issue3Factors associated with female high-risk drinking in a rural and urban South African site author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Article

Indicators

Related links

  • On index processCited by Google
  • On index processSimilars in Google

Share


SAMJ: South African Medical Journal

On-line version ISSN 2078-5135
Print version ISSN 0256-9574

SAMJ, S. Afr. med. j. vol.106 n.3 Pretoria Mar. 2016

http://dx.doi.org/10.7196/samj.2016.v106i3.9969 

RESEARCH

 

Providers' perceptions of the implementation of a performance measurement system for substance abuse treatment: A process evaluation of the Service Quality Measures initiative

 

 

B MyersI, II; P P WilliamsI; K JohnsonIII; R GovenderIV, V; R ManderscheidVI; J R KochVII

IPhD; Alcohol, Tobacco and Other Drug Research Unit, South African Medical Research Council, Cape Town, South Africa
IIPhD; Department of Psychiatry and Mental Health, Faculty of Health Sciences, University of Cape Town, South Africa
IIIMA; Alcohol, Tobacco and Other Drug Research Unit, South African Medical Research Council, Cape Town, South Africa
IVPhD; Department of Sociology, Faculty of Humanities, University of Cape Town, South Africa
VPhD; Violence, Injury and Peace Research Unit, South African Medical Research Council, Cape Town, South Africa
VIPhD; National Association of County Behavioral Health and Developmental Disability Directors, Washington DC, USA
VIIPhD; Department of Psychology, Virginia Commonwealth University, Richmond, Virginia, USA

Correspondence

 

 


ABSTRACT

BACKGROUND: In South Africa, concerns exist about the quality of substance abuse treatment. We developed a performance measurement system, known as the Service Quality Measures (SQM) initiative, to monitor the quality of treatment and assess efforts to improve quality of care. In 2014, the SQM system was implemented at six treatment sites to evaluate how implementation protocols could be improved in preparation for wider roll-out.
OBJECTIVE: To describe providers' perceptions of the feasibility and acceptability of implementing the SQM system, including barriers to and facilitators of implementation.
METHODS: We conducted 15 in-depth interviews (IDIs) with treatment providers from six treatment sites (two sites in KwaZulu-Natal and four in the Western Cape). Providers were asked about their experiences in implementing the system, the perceived feasibility of the system, and barriers to implementation. All IDIs were audio-recorded and transcribed verbatim. A framework approach was used to analyse the data.
RESULTS:
Providers reported that the SQM system was feasible to implement and acceptable to patients and providers. Issues identified through the IDIs included a perceived lack of clarity about sequencing of key elements in the implementation of the SQM system, questions on integration of the system into clinical care pathways, difficulties in tracking patients through the system, and concerns about maximising patient participation in the process.
CONCLUSION: Findings suggest that the SQM system is feasible to implement and acceptable to providers, but that some refinements to the implementation protocols are needed to maximise patient participation and the likelihood of sustained implementation.


 

 

In South Africa (SA), a large unmet need exists for substance abuse treatment.[1] Negative beliefs about the quality of services are a major deterrent to substance abuse treatment initiation,[2] with studies demonstrating that people with untreated substance use disorders generally perceive available treatment services to be of limited effectiveness.[3-5]

Evidence of the quality and effectiveness of substance abuse treatment can help to counter these negative perceptions; however, these data are lacking for SA services.[6] To address this gap, a consortium of stakeholders developed a performance measurement system for SA's substance abuse treatment services. When implemented routinely, such systems are able to produce data that can be used to monitor the quality of services, identify targets for quality improvement, and assess the impact of interventions designed to improve service quality.[7-9]

This performance measurement system, known as the Service Quality Measures (SQM) initiative, emerged from a lengthy development process. This process involved the identification of key domains on which service quality should be assessed, selection of indicators for each domain, and methods for measuring each indicator; the development of a patient survey for measuring some of these indicators; and two pilot tests of this survey to ensure a parsimonious, psychometrically valid measure. This developmental process has been described in detail elsewhere.[10-12]

The finalised system comprises three tools: the SA Community Epidemiology Network on Drug Use (SACENDU)'s admission form that collects sociodemographic and substance use history information and is completed on enrolment into treatment, the SA Addiction Treatment Services Assessment (SAATSA) that accumulates patient-reported data on perceived outcomes and quality of services, and a discharge form that gathers data on the type of services received and the patient's response to treatment. To support implementation we developed a toolkit for providers, specifying in detail the protocols to be followed when implementing this system. In developing these protocols, we were guided by providers' requests for simplicity and congruence with existing administrative practices as much as possible.[10]

In 2014, we conducted a pilot implementation of the system at six treatment facilities to identify ways in which the protocols could be improved prior to implementing the system on a larger scale. This article describes providers' perceptions of the feasibility and acceptability of implementing the SQM system, including barriers and facilitators to implementation.

 

Methods

Study design and sites

We conducted qualitative in-depth interviews (IDIs) with providers from six treatment facilities to understand their experiences of implementing the SQM system. Two of these sites (one inpatient and one outpatient) were located in KwaZulu-Natal and the other four (two inpatient and two outpatient) were in the Western Cape Province. These facilities were puposefully selected as sites for this pilot implementation, as they serve diverse population groups with dissimilar patterns of drug use and have different types of treatment infrastructure.

Sample

Managers of the selected sites were asked to identify at least two staff members to participate in the IDIs. To be eligible, they had to have direct experience of implementing the SQM system. The final sample comprised 15 IDI participants, who represented a diverse range of roles and included administrators, clinical psychologists, social workers, counsellors and managers (Table 1).

 

 

Data collection

All IDIs were conducted during September and October 2014. Before the interview, each participant was asked to provide written informed consent. IDIs were guided by a set of open-ended questions that were developed by the SQM national steering committee. These questions explored the participants' experiences of implementing the SQM system, their perceptions of the feasibility of implementing this system, acceptability of the system, and obstacles to and facilitators of implementation.

IDIs were conducted by two experienced qualitative interviewers who were not associated with the implementation process, one of whom assisted with note-taking. All IDIs were conducted in English, and were audio-recorded and transcribed verbatim. The average duration of an IDI was 30 minutes. The study was approved by the Centers for Disease Control and Prevention (CDC) and the South African Medical Research Council's ethics committee.

Analysis

Qualitative data analysis was conducted using the framework approach,[13] which comprises five stages (familiarisation, identifying a thematic framework, indexing or coding, charting or mapping, and interpretation of the data). Coding occurred iteratively; as new themes emerged in the analysis all relevant information was retrieved and examined for further coding designations.

All transcripts were coded independently by two project staff who met after every two transcripts to compare notes. Coding discrepancies were resolved by discussion. Intercoder reliability checks were conducted, with a kappa score of 0.77 being obtained. We used NVivo 10.0, a qualitative software program, to aid data analysis.

 

Results

Experiences of implementation

Participants generally reported positive experiences of the implementation process. They described the system as 'easy to implement' and the implementation process as 'understandable'. As one participant stated: 'Implementing the SQM toolkit was a valuable experience and quite user-friendly.'

Feasibility of implementation and acceptability of the system for providers

With the exception of one participant, who felt that there was a mismatch between the particular structure of their treatment services and the SQM implementation process, almost all participants reported that the SQM system was feasible to implement in their programme as part of routine services. As one participant reflected, 'It's feasible, it seemed like a hell of a thing at the beginning, but it's just routine. We have so many things to be filled out, forms, and if you just train people properly that's fine.'

Participants viewed the SQM initiative as acceptable because they thought implementing this system would help them to improve their services. As one person stated, 'I think it's a good thing because it gives us that feedback, you give us the analysis of data and it can help, and we always want to improve our services as much as we can'. Similarly, another stated, 'I think it is actually nice to have something like this because it tells you about the service that you are rendering and can help improve services'. The acceptability of the system is also reflected in the finding that several of the facilities wanted to continue to implement the system beyond the pilot period.

Obstacles to implementation

Although most participants reported that their patients were positive about participating in this system and appreciated being asked 'to express their views on treatment', they also mentioned some barriers to patient participation. In this pilot project, patients were introduced to the SQM system on their initial contact with the treatment facility. Participants reflected that this was not the optimal time to introduce the initiative as some patients felt 'bombarded' and 'overwhelmed' by the screening and administrative procedures and therefore 'just refused outright' to participate in the SQM process. As one participant explained, 'We do that 'introduction1 on intake usually and we sometimes sit with psychotic or paranoid patients, or extremely anxious patients and we can't really do that in the beginning with them'.

Another obstacle to patient participation was the requisite level of literacy necessary to complete the SAATSA. Participants noted that although most patients found it 'easy' to complete the form, some required assistance, which might place considerable time demands on treatment staff. As one participant reflected, 'There were other patients who needed help and assistance and I had to explain every single thing'.

Participants also mentioned that although the implementation protocols were a 'well thought out process', there was a lack of clarity about when each element of the SQM system should be completed. Participants described their initial uncertainty about integrating the system into their usual clinical care processes: 'We had a big question mark as to where exactly and in which week to begin implementation. And we actually had to modify the treatment programme. So that was one of the barriers we had to overcome.' Similarly, a participant from another facility spoke of the challenges of knowing when to complete forms: 'The challenge initially was I think making sure that staff understood exactly what they had to 'do1 and when they had to do it. The challenges of when exactly you apply the initial steps and the timing in all of that.'

Several participants mentioned difficulties in keeping track of when patients needed to complete their SAATSA forms. This was partly due to the lack of a system for reminding providers that patients were due to complete the form, which resulted in counsellors forgetting to ask patients to complete the SAATSA. Irrespective of facility size, difficulties in managing the system intensified when facilities were busy: 'Remember we have got a lot of clients that come in and a lot of clients are completing phases, so it is hard to keep track of clients especially after they have completed the whole programme. So, they have gone and you've forgotten to get them to fill it in, there's no way we're going to get them back.'

Participants also reported difficulties in remembering to complete the discharge form. Some treatment sites kept patient folders open for many months after they last had contact with a patient, which delayed the completion of the discharge form. In response to this difficulty, some of the implementing sites developed a tracking system that flagged when patients were due to complete the SAATSA: 'We have a specific monitoring tool in place ... it flags any clients that are due for their SAATSA. And then when it comes time for them to be discharged then the counsellor knows automatically when to do the discharge for the SQM project.'

In a few instances, participants thought that the length of time it took to receive SQM forms from the SQM project team hindered the implementation process. One participant reported that their administrative staff were frustrated when it took longer than expected to receive the forms. Other participants mentioned not receiving enough SAATSA questionnaires in languages other than English. These implementation support issues potentially reduced the number of patients who completed the SAATSA questionnaires. Participants suggested that better communication with administrative staff with regard to their need for additional forms could resolve this issue.

Facilitators of implementation

All participants reported that the training was 'very good' and mentioned how it helped to facilitate implementation of the system. Some participants were pleased that all staff in the facility received training in the system. According to these participants, this reduced ambiguities and enhanced staff support for the implementation process: 'I think originally it was going to be some of us would go and teach the others and then extra space was created for other people. I think that was a very wise decision because for us to secondhand train other people when we have just learnt it is not good and different things, different questions come up you know. People hear different things. The fact that everybody attended the training, you know, they are very much on board.'

Similarly, participants reported that the technical support provided by the SQM project team aided implementation. They approved of the SQM toolkit and reported using it as an important reference guide during the implementation process. They described the technical support as accessible and readily available. As one participant explained, 'What I think was very helpful, whenever I had a question if I didn't know something, there was that open communication. So I could pick up the phone and say we don't know, we're confused, or please clarify and there were no issues with that.' A few participants suggested that additional site visits for fidelity and quality assurance purposes may further enhance the implementation process.

 

Discussion

This article examined the feasibility and acceptability of implementing the SQM system in SA substance abuse treatment facilities. Providers from the full spectrum of available treatment services generally viewed this performance measurement system positively, suggesting that it was acceptable to providers. More specifically, they seemed to appreciate how implementing the SQM system would help them to attain their service delivery goals and could benefit their patients. This finding is encouraging, as the acceptability of a healthcare system innovation is highly predictive of its adoption.[14]

While the vast majority of participants reported that the SQM system was feasible to implement as part of standard practice in their resource-constrained settings, they did identify barriers to implementation and areas where the implementation protocols could be refined to facilitate ease of use. They highlighted how the current practice of introducing patients to the SQM system on their first contact with the treatment facility deterred patient participation. They explained how patients were often overwhelmed by administrative processes on entry into treatment and/or the effects of withdrawing from substances and therefore were unwilling to participate in additional, voluntary activities. Based on this feedback and to maximise patient participation, the SQM project team has moved the time at which patients are introduced to the initiative to a later point in the clinical care pathway.

Related to this, providers described how they were initially uncertain how to integrate this system into their usual clinical and administrative processes. Although each provider received a toolkit outlining implementation guidelines and had an opportunity to participate in training, the guidelines did not address the many practical implementation questions that arose once the facilities started implementing the system. As the SQM team was available to answer these questions and provide facilities with support for implementation, this probably did not jeopardise the feasibility of implementation. However, in facilities where there were many unresolved queries about the implementation process, it took longer for the SQM system to become fully implemented, which affected the number of patients who participated in the initiative. To address this gap and ensure ease of use, we plan to expand our training materials and implementation guidelines to include more practical information about when and how to implement the SQM system.

Providers also reported difficulties in keeping track of when each element of the SQM system was due for completion. Currently, the SQM is a pen-and-paper system that relies on treatment providers to recall when patients are due to complete the SAATSA. Remembering when to complete forms was more difficult when treatment facilities were very busy; during these periods some patients were not given an opportunity to complete the form. This raises concerns about the extent to which the eligible patient population is covered by the SQM system. Some treatment sites had developed their own electronic patient management system with built-in reminders for when each SQM form is due for completion; these appear to resolve difficulties with remembering when to complete forms.

We have considered transitioning to an electronic platform to address this challenge, as such a platform will provide for electronic reminders and have the added benefit of reducing the workload of treatment providers. Although the SAATSA is completed by the patients, in cases where they have low literacy levels treatment centre staff help them by reading the questions and recording their answers, which is very time consuming. This is a concern, as findings from earlier work suggest

that excessive burden to providers will threaten the sustainability of any performance measurement system.[10] An electronic system that allows for audio-computer-assisted personal interviewing could help with this difficulty by enabling even functionally illiterate patients to complete the SAATSA. Apart from reducing the workload of treatment providers, such a system would have the added benefit of reducing social desirability bias.[15] It would also obviate providers' reliance on the SQM team for paper forms and frustration when receipt of forms is delayed. Despite the benefits of an electronic system, we are challenged by the fact that many treatment sites do not have a sufficient number of computers to implement an electronic version of the SQM system.'101 While finding the resources to move the SQM system onto an electronic platform is an important goal for this initiative; in the short term we are trying to identify ways of assisting treatment providers to remember to implement the system. These include helping treatment sites to develop their own systems for tracking patients through the care pathway and also providing treatment sites with more regular on-site support.

Findings from this process evaluation should be considered in the light of a limitation synonymous with qualitative research. As this study was limited to treatment providers from two of the nine provinces in SA, findings may not be generalisable to other provinces. However, as our sample included treatment providers across the entire spectrum of facilities available in the country, we are confident that the concerns raised by our participants will be broadly applicable to all substance abuse treatment providers.

 

Conclusion

Limitations notwithstanding, this process evaluation yielded valuable information that will be used to strengthen the SQM initiative. Firstly, it highlighted that substance abuse treatment providers perceived the SQM system as acceptable and feasible to implement. Secondly, several areas were found where the implementation protocols should be refined prior to widespread implementation of this performance measurement system. These refinements include changing the timing of the implementation to enhance the likelihood of patient participation, introducing processes to remind providers to implement each element of the SQM system, and extending the provider toolkit and implementation guidelines to address practical questions about how to integrate the system into the clinical care pathway. Finally, as there was consensus that the training and ongoing support provided for implementation helped mitigate some of the barriers to implementation experienced by providers, training and support should form a critical part of the roll-out of this performance measurement system.

Acknowledgement. This article received support from the President's Emergency Plan for AIDS Relief (PEPFAR) through the Centers for Disease Control and Prevention, under the terms of Cooperative Agreement Number 5U2GPS001137-4, and by the Western Cape Department of Social Development.

Disclaimer. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention or the Western Cape Department of Social Development.

 

References

1. Herman AA, Stein DJ, Seedat S, Heeringa SG, Moomal H, Williams DR. The South African Stress and Health (SASH) study: 12-month and lifetime prevalence of common mental disorders. S Afr Med J 2009;99:339-344.         [ Links ]

2. Myers B, Louw J, Pasche S. Inequitable access to substance abuse treatment services in Cape Town, South Africa. Substance Abuse Treatment, Prevention, and Policy 2010;5:28. [http://dx.doi.org/10.1186/1747-597X-5-28]        [ Links ]

3. Myers B, Fakier N, Louw J. Stigma, treatment beliefs, and substance abuse treatment use in historically disadvantaged communities. Afr J Psychiatry 2009;12:218-222.         [ Links ]

4. Myers B, Kline LT, Doherty AI, Carney T, Wechsberg WM. Perceived need for substance use treatment among young women from disadvantaged communities in Cape Town, South Africa. BMC Psychiatry 2014;14:100. [http://dx.doi.org/10.1186/1471-244X-14-100]        [ Links ]

5. Parry CDH, Petersen P, Carney T, Needle R. Opportunities for enhancing and integrating HIV and drug services for drug using vulnerable populations in South Africa. Int J Drug Policy 2010;21:289-295.         [ Links ]

6. Myers B, Burnhams NH, Fakier N. Monitoring and evaluation of substance abuse services in South Africa: Implications for policy and practice. International Journal of Mental Health and Addiction 2010;8:557-565.         [ Links ]

7. Garnick DW, Lee MT, Chalk M, et al. Establishing the feasibility of performance measures for alcohol and other drugs. J Subst Abuse Treat 2000;23:375-385.         [ Links ]

8. Harris AHS, Kivlahan D, Bowe T, Finney JW, Humphreys KH. Developing and validating process measures of health care quality: An application to alcohol use disorder treatment. Med Care 2009;47:1244-1250. [http://dx.doi.org/10.1097/MLR.0b013e3181b58882]        [ Links ]

9. Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Conditions: Quality Chasm Series. Washington: National Academies Press, 2005.         [ Links ]

10. Myers B, Petersen Z, Kader R, et al. Identifying perceived barriers to monitoring service quality among substance abuse treatment providers in South Africa. BMC Psychiatry 2014;14:31. [http://dx.doi.org/10.1186/1471-244X-14-31]        [ Links ]

11. Myers B, Petersen Z, Kader R, Parry CDH. Moving beyond access. Towards a quality-oriented substance abuse treatment system in South Africa. S Afr Med J 2012;102:667-668.         [ Links ]

12. Myers B, Govender R, Koch JR, Manderscheid R, Johnson K, Parry CDH. Development and psychometric validation of a novel patient survey to assess perceived quality of substance abuse treatment in South Africa. Substance Abuse Treatment, Prevention and Policy 2015;10:44. [http://dx.doi.org/10.1186/s13011-015-0040-3].         [ Links ]

13. Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess R, eds. Analyzing Qualitative Data. London: Routledge, 1994.         [ Links ]

14. Zhang X, Yu P, Yan J, Spil ITA. Using diffusion of innovation theory to understand the factors impacting patient acceptance and the use of consumer e-health innovations: A case study in a primary care clinic. BMC Health Services 2015;15:71. [http://dx.doi.org/10.1186/s12913-015-0726-2]        [ Links ]

15. Azevedo Simões A, Bastos FA, Moreira RA, Lynch KG, Metzger DS. Acceptability of audio computer-assisted self-interview (ACASI) among substance abusers seeking treatment in Rio de Janeiro, Brazil. Drug and Alcohol Dependence 2006;82:S103-S107.         [ Links ]

 

 

Correspondence:
B Myers
bmyers@mrc.ac.za

Accepted 22 January 2016.

Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License