On-line version ISSN 2071-2928
Afr. j. prim. health care fam. med. (Online) vol.7 n.1 Cape Town 2015
Adaptation et validation de l'Outil d'Evaluation des Soins primaires des Etats-Unis (PCAT version élargie) en vue de son utilisation en Afrique du Sud
Graham Bresick; Abdul-Rauf Sayed; Cynthia le Grange; Susheela Bhagwan; Nayna Manga
Faculty of Health Sciences, University of Cape Town, South Africa
BACKGROUND: Measuring primary care is important for health sector reform. The Primary Care Assessment Tool (PCAT) measures performance of elements essential for cost-effective care. Following minor adaptations prior to use in Cape Town in 2011, a few findings indicated a need to improve the content and cross-cultural validity for wider use in South Africa (SA).
AIM: This study aimed to validate the United States of America-developed PCAT before being used in a baseline measure of primary care performance prior to major reform.
SETTING: Public sector primary care clinics, users, practitioners and managers in urban and rural districts in the Western Cape Province
METHODS: Face value evaluation of item phrasing and a combination of Delphi and Nominal Group Technique (NGT) methods with an expert panel and user focus group were used to obtain consensus on content relevant to SA. Original and new domains and items with > = 70% agreement were included in the South African version - ZA PCAT.
RESULTS: All original PCAT domains achieved consensus on inclusion. One new domain, the primary healthcare (PHC) team, was added. Three of 95 original items achieved < 70% agreement, that is consensus to exclude as not relevant to SA; 19 new items were added. A few items needed minor rephrasing with local healthcare jargon. The demographic section was adapted to local socio-economic conditions. The adult PCAT was translated into isiXhosa and Afrikaans
CONCLUSION: The PCAT is a valid measure of primary care performance in SA. The PHC team domain is an important addition, given its emphasis in PHC re-engineering. A combination of Delphi and NGT methods succeeded in obtaining consensus on a multi-domain, multi-item instrument in a resource-constrained environment.
CONTEXTE: Il est important de mesurer les soins primaires pour faire des réformes dans le secteur de la santé. Le PCAT mesure la performance sur des éléments essentiels pour fournir des soins rentables. Après avoir fait de petits changements avant de l'utiliser au Cap en 2011, on a constaté la nécessité d'améliorer le contenu et la validité cross-culturelle pour une utilisation plus large en Afrique du Sud.
OBJECTIF: Cette étude avait pour but de valider le PCAT développé aux Etats-Unis avant de l'utiliser comme mesure de référence de la performance des soins primaires avant de faire des réformes majeures.
CADRE: Les cliniques de soins primaires du secteur public, les utilisateurs, les praticiens et les gestionnaires dans les districts urbains et ruraux de la Province du Western Cape.
MÉTHODE: Evaluation de la valeur nominale de la formulation de l'article et une combinaison des méthodes du Groupe technique nominal et de Delphi avec un panel d'experts et d'un groupe d'utilisateurs pour obtenir un consensus sur le contenu applicable à l'Afrique du Sud. Des domaines et articles nouveaux et originaux avec un consensus de > = 70% ont été inclus dans le ZA PCAT.
RÉSULTATS: Il a été décidé à l'unanimité d'inclure tous les domaines originaux du PCAT. On a ajouté un nouveau domaine, l'équipe de PHC. Trois des 95 articles originaux ont réuni un consensus de < 70%, c.-à-d. de les exclure car ils ne sont pas applicables à l'Afrique du Sud; 19 nouveaux articles ont été ajoutés. Quelques articles ont dû être légèrement reformulés et remplacés par le jargon local des soins de santé. La section démographique a été adaptée aux conditions socioéconomiques locales. Le PCAT pour adultes a été traduit en isiXhosa et Afrikaans.
CONCLUSION: Le PCAT est une mesure valide de la performance des soins primaires en Afrique du Sud. Le domaine de l'équipe de PHC est une addition importante étant donné son importance dans la reconfiguration du PHC. Une combinaison des méthodes Delphi et NGT a réussi à atteindre un consensus sur un instrument multi-domaine et multipoint dans un environnement aux ressources limitées.
Measuring primary care performance
Primary health care (PHC), considered the backbone of a country's health system,1 is a complex, multifaceted range of activities, a unique integration of knowledge, values and skills drawn from clinical, public health, behavioural and anthropological sciences. In addition to diagnosis, treatment and rehabilitation, the range of skills and activities include person-centred communication, prevention and health promotion applied in a comprehensive family- and community-orientated approach to care. A primary care measurement strategy therefore needs a range of dimensions for it to be a valid measure. Primary care dimensions (e.g. comprehensive care) are themselves multifaceted. Each dimension requires a variety of indicators (items) to describe and measure it (content validity); and item phrasing has to be congruent with the indicator and dimension being measured (face validity).
The Primary Care Assessment Tool (PCAT, http://www.jhsph.edu/pcpc/pca_tools.html) is a multi-dimension, multi-item instrument developed and tested for reliability and validity by Starfield and Shi, Primary Care Policy Center, Johns Hopkins School of Public Health.2 The PCAT measures primary care organisation and performance on four core dimensions (access, continuity, coordination, comprehensiveness)and three derivative dimensions (community orientation, family-centredness, cultural competence)known as domains defined in the PCAT manual (http://www.jhsph.edu/pcpc/pca_tools.html). They have been shown to be essential for cost-effective primary care in developing and developed contexts.3,4,5
When these essential features are available to primary care users and implemented in their care, the outcomes include improved health and satisfaction, reduced cost, and reduced inequity.5,6 The domains are also in line with the Alma Ata Declaration,7 universally accepted definitions of primary care,4,8,9 and the principles of family medicine,10,11 including principles relevant to sub-Saharan Africa.12 By surveying the three main primary care stakeholders (users, providers/ practitioners and clinic managers) the PCAT measures the extent to which users' experience of primary care approximates what is essential for cost-effective care.
The extent of primary care clinic adherence in each domain is determined by complex summing of participant responses to a range of items (questions) pertinent to that domain. The PCAT is also able to determine the size of differences between the three stakeholders' domain scores when the respective user, provider and manager instruments are used. Whilst the dimensions are considered universal, their generalisability may be limited in different cultural and socio-economic settings. Given the focus on health systems strengthening worldwide, it is not surprising that the PCAT is increasingly being subjected to cross-cultural validation to extend its generalisability to countries wanting to align with cost-effective care.13,14,15,16,17 There is also evidence supporting the PCAT's ability to measure the impact of changes to PHC systems.17,18,19,20
This article describes the process used to adapt and validate the adult expanded version (PCAT AE) for use in South Africa (SA). Whilst there are other measures of PHC that have been found to be valid in contexts other than that in which they originate,21 the literature suggests that the PCAT is the most widely used and adapted. Other PHC measures include the Primary Care Assessment Survey (PCAS), the Components of Primary Care Index, the EUROPEP Interpersonal Processes of Care, the Primary Care Evaluation tool (PCET) used mainly in Europe, and the General Practice Assessment Survey (GPAS), all comprising multi-item dimensions.
In 2011, following an earlier visit by PCAT author, the late Professor Barbara Starfield, a study team in the Division of Family Medicine at the University of Cape Town conducted a pilot audit of primary care in one Cape Town health substructure (two sub-districts) in collaboration with the service provider, Cape Town Metro District Health Services (MDHS). The adult expanded (AE) version of the original United States of America (USA) PCAT was scrutinised by the study team under the supervision of the author to determine whether changes were necessary prior to local use. Adhering to the author's guidelines for changes to the PCAT domains (B-K)b only minor adaptations were made.a These involved rephrasing some items to improve local comprehension using colloquial phrasing and local healthcare jargon.
Given the high prevalence of tuberculosis (TB) in SA and in the Western Cape in particular, the item screening for lead exposure was replaced with TB screening to improve the content validity of G domain (comprehensiveness -services available). Changes were made to the introduction (A), health insurance (L) and demographic (N) sections to fit the South African context. The adapted version was piloted on 10 patients before the English version was finalised and translated into isiXhosa and Afrikaans to cover the three major languages spoken in Cape Town. Translation included bilingual translators, back translation and piloting with Xhosa- and Afrikaans-speaking users. The translations were further scrutinised by bilingual fieldworkers during their two-day training to administer the PCAT AE. Minor changes were made to the wording of a few items and they were piloted on patients by the fieldworkers as part of their training before the translations were finalised.
The ZA PCAT (AE) 2011was registered with the Johns Hopkins' Policy Center before starting data collection. All eight clinics in the substructure were sampled. Data from 461 user (patient) PCAT AE questionnaires were entered and analysed. Coding of responses and the analysis was conducted according to the PCAT manual. PCAT Likert scale responses to domain items are coded on a scale of 1 to 4 and 9, with 1 indicating 'Definitely not', 2 indicating 'Probably not', 3 indicating 'Probably', 4 indicating 'Definitely', and 9 indicating 'Not sure/do not remember'. In the analysis 9 is coded as 2 except for comprehensiveness (services provided), where 9 is coded as 0. The score for each domain is calculated by summing all item responses in that domain (with reverse coding where required by the manual) divided by the number of domain items to produce a mean score.
Table 1 shows the results (mean, standard deviation (SD) and range) by domain for the 461 users' data. Users scored community orientation lowest (mean 2.4) and cultural competence highest (mean 3.5).
The distribution of the dimension (domain) scores confirmed what might be expected locally, providing evidence for the PCAT's construct validity in the South African context. This was further supported by service managers and providers generally accepting the results as a reflection of primary care performance in the substructure. The 2013 study team nevertheless wished to strengthen the content and cross-cultural validity of the PCAT before wider use in SA, and to improve its alignment with provincial and national health plans. The 2011 findings suggested that a few domain items may not have been well understood by primary care users. Although none of the items approximated > 50% 'Do not know' responses (requiring adjusted coding and analysis), some were much higher than others. In addition, the users' PCAT score for ongoing care (domain D) was considerably higher than the provider and manager scores and higher than expected by study team members who had years of experience working in the clinics studied.
A European Union grant was obtained to strengthen the PCAT's validity for South African use and to extend the 2011 pilot measure of primary care performance to other health districts in the Western Cape Province prior to major health sector reform. Study objectives to strengthen the relevance of the PCAT content for SA included improving its face, content and cross-cultural validity by (1) reviewing the phrasing of domain items for user comprehension; (2) reviewing the domains and their content for alignment with the healthcare setting in SA, the Western Cape demographic and socioeconomic context, local and national healthcare policies; and (3) improving the 2011 translations.
This study is part of a bigger European Union-funded study in the School of Public Health and Family Medicine at the University of Cape Town which is aimed at improving users' experience of primary care.
Research design and method
Approval for the study was obtained from the Human Ethics Research Committee in the University of Cape Town's Faculty of Health Sciences and the Western Cape Provincial Department of Health Research Committee. There were no risks to participants in the study. All data provided were analysed and reported anonymously.
The study was conducted in two parts: (1) an expert panel and a primary care user focus group reviewed the wording of PCAT domain items for their face value; and (2) a combination of modified Delphi and nominal group technique (NGT) methods was used to determine consensus on domain and item relevance for SA. The ZA PCAT was piloted at a primary care clinic before being finalised and translated.
Part 1: Review of item wording for local use
Expert panel (1a)
This panel, consisting of 2011 PCAT study investigators (two family physician researchers, a family physician in charge of clinical governance at a clinic and a fieldworker) had face value evaluation meetings to review the wording of all PCAT items in domains B-K and to rephrase items where necessary to improve comprehension by primary care users in SA. Problematic wording identified in 2011 was given particular attention. Changes were agreed upon by simple consensus. Selected items were referred to the user focus group to assist with improving user comprehension.
User focus group (1b)
Two members of the PCAT study team purposively selected and invited six primary care users from amongst patients attending a clinic in the baseline measure study sampling frame to join a focus group. Six consenting adult patients who had attended the clinic at least four times and who were conversant in two of the three main languages spoken in the Western Cape (namely English and Afrikaans) were selected. A modified form of the NGT method was used to obtain group consensus on the phrasing of items selected in part 1a. The process involves each item in turn being presented to the group on a flipchart in one of the two languages. Each participant records her and/or his back translation on a blank sheet without discussion. All responses are recorded anonymously on a flip chart, after which each participant, without discussion, chooses and records the one response that she or he feels most accurately reflects the item presented. Responses (phrasing) achieving > = 70% agreement22 are accepted. The steps are repeated for each item. The investigators determine whether the phrasing chosen maintains the intention of the original item; if not, investigators rephrase the item to maintain the original intention, taking into account the information obtained from participants by that stage. The above steps are repeated to determine consensus on the item thus phrased.
Part 2: Obtaining consensus on domain and item relevance for South Africa
Delphi process (2a) (steps 1-4 in Appendix 1)
A second expert panel comprising 2 family physicians, 2 clinical nurse practitioners, 2 clinic managers, and 2 family physician educator/researchers were purposively selected by the study team to examine and determine consensus on the relevance of all PCAT domains and domain items (B-K) for the primary care context in SA. The task of each panellist was to determine from her/his own experience, (i) which items were relevant, and (ii) which items, if any, should be added. Background information given to the panellists included a summary of the PCAT, its purpose, and relevance to PHC and family medicine. The panellists were informed of the 2011 PCAT study; that lessons from 2011 were being applied; that rephrasing of some questions might be necessary; and that the process outlined was to validate the ZA PCAT for use in SA by using a combination of the Delphi and NGT methods (Appendix 1). Inviting, consenting and providing information to the panellists were done via email. Panellists were asked to scrutinise all the PCAT domains and their respective items and to score them for relevance on a scale of 0 to 3 (not at all relevant - very relevant). At the end of this stage the emailed data were collated and the percentage agreement calculated for the 9 domains (B-K) and their items. The results together with new items were tabulated.
Nominal Group Technique process (2b) (steps 5-7 in Appendix 1)
The expert panel in 2a was convened by two study investigators and part 2a results presented (items which achieved < 70% agreement; new items generated; and comments) by displaying them verbatim in poster form for panellists to view and to ensure that their items and comments had been recorded. Three questions were provided to guide the selection and scoring of new items generated in part 2a: (i) Is it important and relevant to primary care?; (ii) In which domain does it belong?; and (iii) Can clinic staff do anything about it? The method requires that the meaning of each new item be clarified and similar items merged (step 5) in a facilitated group discussion. Given that the above discussion may stimulate further thinking on domains and items to include, panellists are given a second opportunity on their own (i.e. NGT silent phase repeated) to generate items they wished to include.
All new items thus generated are in turn clarified to ensure they are understood by all participants and that they differ from existing items (step 5 repeated). These were added to a table along with new items generated in the Delphi process (2a) and original PCAT items that achieved < 70% agreement and scored for relevance as in 2a (i.e. items that achieved < 70% agreement in 2a were re-scored), and were recorded independently. Each panellist then verbally submitted her/ his scores in round-robin fashion (step 6). The scores were recorded and summed by the investigators, and the summed scores presented in a projected table (step 7). The percentage agreement for each item was calculated to determine which items achieved consensus (step 7), that is > = 70% agreement. Repetition of steps 3-7 above substituted for the usual successive rounds in the Delphi method, which was limited to one round in part 2a. The percentage agreement for each item is calculated, where 100% = the number of panellists x 3, that is the maximum score per item. Only domains and items on which consensus is achieved (i.e. > =70% agreement) are retained (step 8).
Item review and rephrasing (1a and 1b)
Domain and item relevance was not considered at this stage. The phrasing of domain items (questions) considered problematic by the 2011 study teamrequired minimal rephrasing. The team nevertheless wanted user assistance on five questions (D4, F1, G2, J2, K2) to improve user comprehension. These were presented to the six focus group participants for scrutiny as described in 1b of the methods section. Most participants had difficulty understanding the task, especially back translation and its purpose, and preferred to add and discuss questions on problems they encountered with the service. After unsuccessful attempts to explain and guide participants through the planned NGT process, this was abandoned. Instead an open discussion on the phrasing of the questions was conducted to gain as much information from patients as possible for item rephrasing.
The Delphi process (2a)
Results of the Delphi process (2a) revealed that the expert panel considered all the domains as very relevant; median = 92.6%; range = 48.2% - 100.0%; interquartile range 85.2% - 96.3%. Only 3 of the 95 domain items received < 70% agreement (C6: 48.2 %; C7: 48.2%; H9: 51.9%). No new domains but two new items were added (C4NGT, C5NGT) by the end of this stage.
The Nominal Group Technique process (2b)
In the NGT process (2b) a total of 19 new items were generated and scored, as described in methods sections 2a and 2b. The results are presented in Table 2. These items related to the first contact - access (C), ongoing care (D), coordination (E), comprehensiveness (services available [G] and services provided [H]), and culturally competent (K) domains. One new item, the PHC team, emerged during clarification of newly generated items. The panel felt that it was important to add the PHC team as a domain instead,given the emphasis on the PHC team in SA'sdistrict health policies and plans. A list of items to describe the composition of the PHC team in a comprehensive primary care service was generated. PHC team items were agreed on by simple consensus and not subjected to the NGT stepped process due to time constraints.
This was followed by a final round of scoring (individually and without discussion), which included all the original PCAT items which received < 100% agreement in 2a and all new items generated as described above. Whilst rescoring the few original items that achieved < 100% but > 70% in part 2a was not essential, the panel felt that a final round of scoring that included these items should be the final step. Table 2 shows the final percentage agreement for all original items which achieved < 100% agreement, all new items (coded as 'NGT') as well as the PHC team domain.
Following piloting of the ZA PCAT AE 2013 on 10 patients at a clinic in the baseline study sampling frame, minor changes were made to the phrasing of a few items. This completed the ZA PCAT validation process. All domains and items achieving consensus at the end of part 2b constitute the ZA PCAT AE 2013.b The rationale for retaining original PCAT items which did not achieve consensusb for use if needed is explained in the discussion below. The provider (PE) and manager (FE) instruments were accordingly aligned with the ZA PCAT AE by the study team. The additions to the English AE version were translated and back translated for the isiXhosa and Afrikaans versions and existing items in the 2011 translations were reviewed by bilingual study team members before being finalised.
Following lessons from the 2011 pilot study, we sought to validate the PCAT content and cross-cultural applicability for use in SA, that is improve its relevance to primary care in SA and its comprehension in three languages before extending the audit to other health districts. Minimal item rephrasing was necessary following that done in 2011. The fact that no domains were removed, only one new domain was added, only three items achieved < 70% agreement, and a high median percentage agreement on domain and item relevance was achieved, indicates the high content validity of the original PCAT for SA.
The silent phases in parts 2a and 2b ensured that participants were able to generate items and to agree/disagree independently of each other. The discussion during the clarification stage and the range of scores in part 2b suggest that participants were not unduly led by each other's views or scores. The addition of the PHC team domain aligns the PCAT with the emphasis on the PHC team in local and international policy documents and research.23,24,25,26,27 In a study identifying key principles of family medicine in sub-Saharan Africa, practising as members of a PHC team emerged as important and included nurse practitioners.12 Including the PHC team as a domain in the ZA PCAT is therefore likely to be supported by primary care physicians in Africa.
Nurse practitioner (CNP) was not generated as a PHC team item. This can be explained by the fact that PHC in SA is a nurse-led service.23 In most primary care clinics CNPs are the only clinical practitioners. Where there are doctors, CNPs practice as members of the clinical team alongside them.
The PHC team domain uses the same Likert scale for rating responses to items and is analysed in the same way as existing domains. PHC team scores are included in the total primary care score - a summing of all the domain scores. New items which do not use the PCAT Likert scale (patient waiting times, travel costs and patient satisfaction) are included under the relevant domains but are analysed separately. The study team elected to retain original items C6, C7, H9 in the printed versionb used for the data collection in order to allow for international comparison of results where the original PCAT is used. Analysis of the data will include or exclude these items as required. Cronbach alpha and factor analyses on the ZA PCAT 2013 will be included in the second article reporting the results of the PCAT survey of users, providers and managers.
The method used to improve the wording of domain items for better comprehension by users did not follow the standardised seven-step method described by Sousa and Rojjanasrirat28 in their review of methodological approaches to translation, adaptation and cross-cultural validation of research instruments; also used in the Korean adaptation of the PCAT.15 A key component of our cross-cultural method was the user focus group (part 1b). However, most of the participants had difficulty understanding the task, especially back translation and its purpose, which led to the planned NGT process being abandoned. Users' preference for raising and discussing problems they encountered with the service is highlighted in patient satisfaction surveys.
The research team nevertheless constituted an expert panel with years of experience practising and teaching in local primary care services. The same team conducted the 2011 study and included a research assistant with experience in developing and administering questionnaires in other research projects in local primary care services. The combined experience was applied in updating and improving the translations of the English PCAT into the two other main languages spoken in the Western Cape. Edits to the demographic section (N) to align it with Western Cape demographic and socio-economic features were also made. Even though the translation method did not follow all the steps suggested by Sousa and Rojjanasrirat,28 the bilingualism advised for cross-cultural validation was well represented on the panel and the research assistantwas trilingual.
The pilot conducted on 10 patients before finalising the ZA PCAT AE 2013 will have reduced the impact of our failure to achieve the primary objective of the focus group, by providing another opportunity to identify difficulties with phrasing and administering the PCAT to users. In addition, the 2011 translation conducted by the study team was also scrutinised by the bilingual fieldworkers. Their practical training on patients served as a second pilot of the isiXhosa and Afrikaans translations in 2011.
The focus group experience nevertheless provided important insights, including the depth of feeling amongst users about their frustrations with primary care; the challenges faced by users who are functionally illiterate (which complicated the task of the researchers in this context); the importance of a participatory method when involving user stakeholders and the need for researchers to be flexible. The communication challenge between informant and researcher reflects a common feature of the primary care consultation, namely the potential conflict between patient and practitioner agendas, where 'give and take' is required to achieve a therapeutic partnership.
Combining the Delphi and NGT methods in a two-stage process had the benefit of participants independently generating responses in at least one round before meeting for the NGT, thus reducing the impact of attrition of participants and delayed responses that can bedevil a Delphi process. The combination of the Delphi and NGT methods may be better considered a modification of the NGT.22 In verbal feedback at the end of part 2b the expert panel reported that, as busy clinicians and managers, they found the combination of Delphi and NGT methods useful in seeking consensus on the content and wording and preferred to meet once-off rather than have a number of iterations as required by the Delphi method. They also preferred the opportunity to interact with each other, especially in the prioritisation and clarification stages of the NGT.
In addition, the modification provided the opportunity for doctors, nurse practitioners and managers to contribute as equals in a key stakeholder group. Modelling such a method that can be used in clinics by managers and practitioners to obtain multi-stakeholder consensus on complex activities and interventions in a time-constrained and limited-resource context may also be useful. Involving these key stakeholders should also increase the likelihood of the ZA PCAT's use in ongoing monitoring, evaluation and quality improvement, also guiding the revitalisation of PHC and assisting health sector reform in SA. Whilst the study did not specifically address aligning the PCAT with provincial and national health policies and plans, it can be argued that including managers, practitioners and educators with years of experience in health services in SA guided the work with these in mind - evidenced by the addition of the PHC team.
As noted above, the findings concur with key principles of family medicine in sub-Saharan Africa and therefore point to the potential for the ZA PCAT to be similarly used in other parts of Africa, as well as to measure the impact of interventions aimed at strengthening PHC systems. This includes measuring the impact of postgraduate training in family medicine (primary care physicians) on person-centred comprehensive, community-based primary care as promoted in the Victoria Falls Statement.24 Given that primary care in SA is nurse-driven, the finding that the PCAT content is highly relevant to SA suggests that CNPs should also be trained to apply the essential elements of primary care. If the benefits of a PHC team are to be realised, training in the family medicine approach to primary care - currently the preserve of primary care physicians - should be extended to CNPs. Primary care physicians and CNPs should be trained together on the content, application and measurement of the essential dimensions of primary care. Primary care facility and district managers should be aware of the importance of these elements and trained in methods that improve access to them when allocating and managing resources. They should also be trained in the use of the PCAT to monitor the organisation and performance of the primary care services they manage.
Limitations of the cross-cultural validation method used, when compared with those suggested by a scholarly review,28 are discussed above. This will need to be considered if there are significant differences in scores between the three language groups after the baseline results are analysed. Other study limitations include the expert panel and patient informants being local only. The full Delphi method would have permitted wider representation on the expert panel and could have included participants in other sub-Saharan countries. The items in the PHC team domain do not describe team functioning or the quality of team-based care and therefore limit the potential value of this measure. Items that describe team function should be developed and added in future studies.
Is the ZA PCAT 2013 suitable for the Western Cape only? Given the diversity across the nine provinces in SA, translation into local languages and some rephrasing will be necessary, along with changes to the demographic section, depending on region or province. However, we think it unlikely that major domain and item changes will be necessary.
These limitations notwithstanding, the findings are in keeping with those of PCAT validation studies in other countries15,16,17,18,19,20 where the essential features of primary care measured by the PCAT were also found to apply. The Brazilian study18 kept closely to the original items, whereas other country studies removed a number of domain items, such as in the Chinese PCAT.17 Haggerty et al.21 examined the validity of a number of instruments that measure PHC from the user perspective in the Canadian context, including the PCAT, the PCAS, the Components of Primary Care Index, and the EUROPEP Interpersonal Processes of Care - all composed of multi-item dimensions. The study found that these instruments performed similarly in Canada as in their original contexts. The Brazilian PCAT study is of note for SA. It showed that the PCAT also applies in a developing context.18 The PCET, not included in the Haggerty study, is also of interest.29 It includes the four core primary care dimensions measured by the PCAT (access, continuity, comprehensiveness and coordination) as well as four health system functions (financing, creating resources, stewardship and delivery services) that are worthy of assessment in a primary care audit. These functions can be included alongside a PCAT audit without having to adapt the PCAT.
Other audit instruments used in SA currently include the National Core Standards (NCS) instrument26 and the chronic diseases audit tool used in the Western Cape. With respect to possible duplication, whilst the NCS includes items on patient-centred care, they are not used to assess relational continuity (PCAT ongoing care [D)] domain), a multifaceted core primary care dimension. The NCS does not seek to determine performance on the other key dimensions of the primary care process, but focuses instead on infrastructure, equipment and administrative resources required.
Similarly, the chronic diseases audit tool-currently a record audit of selected indicators of common chronic disease care-is not a measure of comprehensive PHC. Regarding patient satisfaction surveys, used as a means to determine patients' views on their health care, there may be some item overlap. However, the PCAT is not a patient satisfaction questionnaire. It is, rather, an evidence-based measure of performance on features of primary care demonstrated to be essential for cost-effective care. Likewise, patient health survey instruments such as the SF-3630 are limited to evaluation of health and not a measure of the range of dimensions necessary for cost-effective primary care.
This is the first of two articles reporting PCAT studies in Africa. It describes the content and cross-cultural validation of the PCAT for use in SA. The results suggest an important role for the PCAT in the Western Cape Province and nationally, given the South African National Health Policy's imperative of PHC re-engineering and broader health sector reform.23 Future audits of primary care performance should include private sector services. The findings have implications for the training of primary care doctors and nurse practitioners as well as clinic and district managers.
Further research should include strengthening the PHC team domain to enable assessment of team functioning and performance.
A second article will describe the baseline results of the ZA PCAT 2013 study in the Cape Town MDHS and the Cape Winelands, a rural district.
We thank the members of the expert panel and other key informants for their willingness to provide the information necessary to validate the PCAT; the district directors and clinic managers for ready access to the services; other members of the study team, including Prof. Derek Hellenberg, Mr Deon September and Ms Delena Fredericks; Prof. Gregory Hussey for assisting with the structure and editing of this article; and the European Union for funding this study. Deon September (data collection and entry, fieldworker supervision and general research assistance) and Delena Fredericks (administration and management) made a very significant contribution to this study.
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
G.B. (University of Cape Town) is the principal investigator and leader of the PCAT project and assumed primary responsibility for writing the article. A-R.S. (University of Cape Town) provided all the statistical advice and calculations for the study proposal, conducted the data analysis, generated the summary tables, figures and graphs and oversaw reporting of all the statistical elements in the method and results sections. C.l.G. (University of Cape Town) and S.B. (University of Cape Town) were centrally involved in the data collection and management, advising on the content and structure of the article and ensuring accurate reporting of the method and results sections. N.M. assisted with data collection and management and contributed to general project oversight, coordination and management.
2. Shi, L, Starfield, B, Xu, J. Validating the Adult Primary Care Assessment Tool. JFamPract. 2001;50(2):161. [ Links ]
4. Starfield B. Primary care: Balancing health needs, services, and technology. New York: Oxford University Press; 1998. [ Links ]
5. Starfield B, Shi L, MacinkoJ. The contribution of primary care to health systems and health. Milbank Quarterly. 2005;83:457-502. http://dx.doi.org/10.1111/j.1468-0009.2005.00409.x [ Links ]
7. World Health Organization. Declaration of Alma-Ata. 1978. Available from: http://www.who.int/publications/almaata_declaration_en.pdf [ Links ]
8. Molla S, Donaldson K, Yordy D, Lohr K, Vanselow NE. Primary care: America's health in a new era. Report of a study by a Committee of the Institute of Medicine, Division of Health Care Services. Washington, DC: National Academy Press;1996, 395. [ Links ]
9. Institute of Medicine. Defining Primary Care: An Interim Report. Washington, DC: National Academy Press; 1994. [ Links ]
10. Shahady EJ. Principles of family medicine: an overview. Essentials of family medicine. 2nd ed. Baltimore: Williams & Wilkins;1993, pp. 3-8. [ Links ]
11. McWhinney IR. A textbook of family medicine. New York: Oxford University Press; 1997. [ Links ]
12. Mash R, Moosa S. Exploring the key principles of Family Medicine in sub-Saharan Africa - international Delphi consensus process. SA Fam Pract. 2008;50(3):60-65. http://dx.doi.org/10.1080/20786204.2008.10873720 [ Links ]
13. Harzheim E, Starfield B, Rajmil L, Alvarez-Dardet C, Stein AT. Internal consistency and reliability of Primary Care Assessment Tool (PCATool-Brasil) for child health services. Cad SaudePublica. 2006;22(8):1649-1659. http://dx.doi.org/10.1590/S0102-311X2006000800013 [ Links ]
14. Berra S, Hauser L, Audisio Y, Mantaras J, Nicora V, de Oliveira MM, et al. Validity and reliability of the Argentine version of the PCAT-AE for the evaluation of primary health care. Rev Panam Salud Publica. 2013;33(1):30-39. http://dx.doi.org/10.1590/S1020-49892013000100005 [ Links ]
15. Jeon KY. Cross-cultural adaptation of the US consumer form of the short Primary Care Assessment Tool (PCAT): the Korean consumer form of the short PCAT (KC PCAT) and the Korean standard form of the short PCAT (KS PCAT). Qual Prim Care. 2011;19(2):85-103. [ Links ]
16. Haggerty JL, Beaulieu MD, Pineault R, Burge F, Levesque JF, Santor DA, et al. Comprehensiveness of care from the patient perspective: comparison of primary healthcare evaluation instruments. Healthc Policy. 2011;7(Spec Issue):154-166. http://dx.doi.org/10.12927/hcpol.2011.22708 [ Links ]
17. Yang H, Shi L, Lebrun LA, Zhou X, Liu J, Wang H. Development of the Chinese primary care assessment tool: data quality and measurement properties. Int J Qual Health Care. 2013;25(1):92-105. http://dx.doi.org/10.1093/intqhc/mzs072 [ Links ]
18. Macinko J, Almeida C, Klingelhoefer de Sa P. A rapid assessment methodology for the evaluation of primary care organization and performance in Brazil. Health PolicyPlann.2007;22:167-177. http://dx.doi.org/10.1093/heapol/czm008 [ Links ]
19. Haggerty JL, Pineault R, Beaulieu MD, Brunelle Y, Gauthier J, Goulet F, et al. Room for improvement: patients' experiences of primary care in Quebec before major reforms. Can Fam Physician. 2007;53(6):1057, 2001:e.1-6, 1056. [ Links ]
20. Wang W, Shi L, Yin A, Lai Y, Maitland E, Nicholas S. Development and validation of the Tibetan primary care assessment tool. Biomed Res Int. 2014;2014:308739. http://dx.doi.org/10.1155/2014/308739 [ Links ]
21. Haggerty JL, Burge F, Beaulieu MD, Pineault R, Beaulieu C, Levesque JF, et al. Validation of instruments to evaluate primary healthcare from the patient perspective: overview of the method. Healthc Policy. 2011;7(Spec Issue):31-46. http://dx.doi.org/10.12927/hcpol.2011.22691 [ Links ]
23. Western Cape Provincial Department of Health. Health Care 2030 - The Road to Wellness. Cape Town: Western Cape Provincial Department of Health; 2014. [ Links ]
25. Department of Health, South Africa. National Health Insurance in South Africa. Policy (Green) Paper. Pretoria: Department of Health, 2011. [ Links ]
26. Sixty Second World Health Assembly. Primary health care, including health system strengthening. WHA resolution 62.12; Geneva; 2009. Available from: http://www.who.int/hrh/resources/A62_12_EN.pdf?ua=1 [ Links ]
28. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: A clear and user-friendly guideline. J Eval Clin Pract. 2011;17(2):268-274. http://dx.doi.org/10.1111/j.1365-2753.2010.01434.x [ Links ]
29. World Health Organization, Regional Office for Europe. Primary Care Evaluation Tool (PCET). Available from: http://www.euro.who.int/en/health-topics/Health-systems/primary-health-care/publications/2010/primary-care-evaluation-tool-pcet [ Links ]
30. Brazier JE, Harper R, Jones NM, O'Cathain A, Thomas KJ, Usherwood T, et al. Validating the SF-36 health survey questionnaire: New outcome measure for primary care. BMJ. 1992;18;305(6846):160-164. [ Links ]
Room 2.20, Level 2, Falmouth
Building, Anzio Road
Received: 06 Oct. 2014
Accepted: 17 Dec. 2014
Published: 19 June 2015
a Edits to item phrasing were made in B2, B3, C3, C6, C8, C9, D7, D12, E1, E2, F1, F3, G2, G3, G10, G12, G13, G14, G15, G18, G20, G21, G22, G23, G25, H2, H8, H12, J1, J2, J3, J18, K2. Local vernacular was used and / or examples added in order to improve user understanding, e.g. 'prenatal' replaced with 'antenatal' (G20), and 'blood or sputum' added to E1 as examples of laboratory tests. In G9, screening for lead exposure was replaced with TB screening. Changes to the introduction and screening (A), insurance (L), and demographic (N) sections were not subject to the guidelines and were made independently by the team to suit the local and SA context. Cross-cultural improvements to demographic and socio-economic sections included adding items on home language, type of housing, sanitation, level of education and household income.
b South African Adult Primary Care Assessment Tool (ZA PCAT AE 2013), adapted from the original PCAT by the South African PCAT Study Team, Division of Family Medicine, School of Public Health and Division of Family Medicine, University of Cape Town, 2013. Original Adult Primary Care Assessment Tool - Expanded Version developed by Barbara Starfield, Primary Care Policy Center, Johns Hopkins University 1998 (for access to the document, please contact corresponding author).
APPENDIX 1: Information given to expert panellists (Phase 2).
Consensus method and outline of the process
You are being presented with the user (patient) version of the PCAT only. The same domains and similar questions (items) are asked of clinic practitioners and managers in the practitioner and manager PCAT versions, but from their respective perspectives. Within each domain, a number of questions (domain items) to users of primary care services determine whether that domain is present (accessible) and applied (utilised) in their care. We want to know which questions may not be relevant for determining quality primary care in SA given your understanding of patients' and communities' health needs; your experience managing common presenting problems in their context; and whether any additional questions should be added.
Part 1 (steps 1-3) will be conducted on your own at the end of which you email your responses to the study team.
Part 2 (steps 5-7) will be conducted with the panellists meeting as a group. We envisage that each part can be done in 90mins or less. By the end of step 7 an enhanced ZA PCAT will have been agreed on via the consensus process outlined.
The attached PCAT questionnaire has 9 primary care domains (essential elements) each with a number of items posed as questions to describe the presence and practice of each domain. You are being asked to:
- rate the relevance of domain items on a scale of 0-3.
- add any domains and / or items that you feel are important in the SA context that are missing in the current version.
(comment on italicised text if you wish and have the time)
The ratings of the expert panel will be combined to obtain an overall score for each item; depending on the scores, domains items will be retained, removed or added. Items that receive consistently low scores will be removed at the end of part 1. The remaining and/or additional items will go into round 2 and follow the same process above. Further rounds will follow if necessary until consensus is achieved by the panel on which items should remain, which should be removed and which if any items or domains should be added. Consensus is defined as 70% agreement and will constitute the revised ZA version of the PCAT to be used in the study.
Expert panel: Combined Delphi and Nominal Group Technique method:
PART 2a: Steps 1-4 are conducted independently via email prior to the expert panel convening:
1 . Introduction and aim of the exercise
The purpose and objectives of the consensus method have been explained above.
2 . Presentation of the questions and definition
The domain items (questions) for scoring are as per the attached ZA PCAT Adult Expanded (AE) version along with the definition of each domain. Score the relevance for each domain item in the electronic document.
3 . The 'silent' phase - scoring the relevance of each domain and items and generating new domains and items
3 .1 Before you score the domain items, please read the definition of that domain and keep it at hand to refer to as needed.
3 .2 Score the relevance of domain items (questions) from your knowledge and experience of what is required for good comprehensive primary care in SA on a scale of 0-3 where 0 is definitely not relevant, i.e. not at all relevant; and 3 is definitely relevant, i.e. totally relevant.
3 .3 At the end of each domain, add any item(s) not currently in the ZA PCAT that you feel should be included in that domain. You do not need to number them. Please indicate briefly next to any new items, why you would think these should be added.
3 .4 In addition, after scoring all the domains, add any other domain(s) (in the box at the end of the document) that you feel should be included and items that will determine the presence (access to) and utilisation (application) of that domain. Indicate briefly next to any new domain(s), why you think it should be added. You may add a domain even if you do not have items to describe it.
Submit the document with your scores and any additional items (i.e. responses to step 3) to the investigators (Graham Bresick<email@example.com> and Nayna Manga <firstname.lastname@example.org>) via email by 19 April 2013
4 . Prioritisation
All the panellists' responses will be captured, analysed and prioritised by the study team in preparation for Part 2 below.
PART 2b: Expert group convenes
Participants and investigators are introduced. A brief overview of the purpose, objectives and process of the consensus method is given and any questions for clarification dealt with.
5 . New item presentation, clarification and rationalisation Any new items and domains generated in step 3 are presented to the expert panel and their meaning clarified in the group to ensure they are understood by all and how they differ from existing items. New domains and items with similar meanings and intentions will be merged. The relevance of items is not discussed at this stage. This step ends with a second list of new domains and items.
6 . 'Silent' phase rating of new domains and items
Each participant scores each domain and item in the list generated in step 5 above on his/her own on a sheet of paper as in step 3.2. (Highlight new items in their respective domains). The scores are submitted without discussion and analysed (prioritised) by the investigators to determine which from list will be added.
Presentation of final list to the panel
All domains and items scored as relevant and not relevant in step 3.2 in part 1 and step 6 in part 2 are presented. Any 'borderline' items are noted and can be re-scored - again individually and without discussion.
All the domains and items thus scored as relevant for inclusion will constitute the final content of the ZA PCAT 2013 i.e. > = 70% of the maximum score = consensus
8 . Compilation of ZA PCAT v2
The PCAT study team compile ZA PCAT 2013 according to steps 1-7 in parts 1 and 2 above.
Participants are asked to record two things they found useful and two they did not find useful about the process. Part 2 ends with any general discussion the panel may wish to have.