EDITORIAL

 

AGREE to disagree: Critical appraisal and the publication of practice guidelines

 

 

Faced by an explosion in available evidence for multiple new treatments, busy clinicians value guidelines that are clear, reliable, unbiased and locally applicable. Finding them can be difficult, however. The science of guideline development has moved rapidly in the past decade, resulting in a more robust and systematic process. However, just as the language of evidence-based medicine can be subverted to sound convincing while hiding errors and biases, so too guidelines may look convincing but lack many of the elements needed to ensure quality of care. In particular, the pharmaceutical and health technology industries are intensely aware of the marketing potential offered by widely disseminated and ostensibly neutral documents that ultimately influence medical practice.

A few examples illustrate the potential challenges. The recently published JNC8 guidelines on hypertension management^[1] followed a very robust process,^[2] and yet ended with some of the authors taking a dissenting view,^[3] which at the least raised concerns about the credibility of the guideline. The European Society of Cardiology guideline on perioperative beta-blockade was the subject of potential conflict of interest concerns when the guideline chairperson was investigated for scientific misconduct related to the DECREASE family of trials.^[4] At the time it was estimated that 10 000 iatrogenic deaths could have been prevented annually by refraining from following this guideline.^[5] A 2009 review of cardiology guidelines^[6] found that of 7 000 recommendations only 11% were based on randomised controlled trial (RCT) evidence, and a 2011 study of infectious disease guidelines^[7] reported a similar finding, with only 14% of 4 000 recommendations having RCT backing. These observations were further corroborated in a recent systematic analysis in which the validity of guidelines published by interventional medical societies was investigated. It was established that most associated guidelines failed to grade the evidence, and when graded, lower-quality evidence was used. Furthermore, most guidelines failed to disclose conflicts of interests.^[8]

The US Institute of Medicine has defined clinical practice guidelines as 'statements that include recommendations intended to optimise patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options'.^[9] They are important in influencing public health policy, promoting distributive justice and advocating better healthcare delivery for those in need.^[10,11] The increased frequency with which clinical treatment guidelines/recommendations are being published in South Africa (SA) is of great significance in a resource-poor country, accentuating the need to ensure their validity. Consequently, heightened responsibility is called for in terms of applying due process throughout the development and publication of all guidelines.

 

Evidence collection and synthesis

The application of the evidence-based medicine philosophy, while not infallible, is central to the promotion of a fair and equitable distribution of resources. Sackett et al.^[12] define evidence-based medicine as 'the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.' By individual clinical expertise, they refer to the proficiency and judgement that individual clinicians acquire through clinical experience and clinical practice.^[12] While it is accepted that 'expert opinion' falls low down on the hierarchy of sources of clinical evidence, if principles of evidence-based medicine are to be applied in the true spirit of their intention, then the need for consensus opinion is fundamental.^[13] It is in this context that clinical guidelines and recommendations contribute profoundly towards the process of decision making and resource allocation.

 

Making recommendations

Priority setting is challenging in any health system. Irrespective of our sector of practice, financial resources are finite and there are competing interests in terms of resource allocation. The demand for health services exceeds available resources, and in order to meet the health needs of a community, allocation of health services and associated financial resources must be prioritised.^[14] As Chinnock et al. state,^[15] wherever healthcare is provided, it is essential to know which interventions work, which do not, and which are likely to be harmful. This is especially important in situations where health problems are severe and scarcity of resources makes it vital that they are not wasted.

In a 2011 position paper regarding the effective and efficient conservation and distribution of healthcare resources, the American College of Physicians^[16] noted the requirement for a transparent and publically acceptable process for making resource allocation decisions with a focus on medical efficacy, clinical effectiveness and need, with consideration for cost, based on the best available medical evidence. It is important to emphasise that rationing in this context is not about the denial of care, but rather about choosing clinically effective alternatives based on peer-reviewed medical evidence to provide clinically appropriate and cost-effective care that maximises value.^[16]

Recommendations therefore need to integrate evidence, concepts of best practice and resource constraints in order to set practice directions and provide tools to audit the care delivered. This process is seldom easy, and it is useful for practitioners to understand the strength of the recommendation in order to decide how best to apply it in their practice.

 

Clinical guidelines and the SAMJ

The SAMJ (and its HMPG 'siblings') has recognised the important role that guidelines play in setting standards of clinical practice as well as the imperative of following transparent due process in their development and publication. To this end, it has sought to establish a formalised peer-review process.

In fulfilling this obligation, we introduce the readership to the AGREE II instrument as an objective appraisal mechanism against which prospective clinical guidelines will be assessed before their publication. This internationally developed and validated quality assessment tool is available in the public domain (at www.agreetrust.org) and can be applied generically to assess the quality of guidelines across all disease areas and at any stage of the healthcare continuum, whether it be screening, diagnosis or treatment.^[17] It can also be used to inform guideline development processes and provides guidance on the type of information that should be reported in a guideline.

The purpose of the AGREE II tool is to provide a framework (from Brouwers et al.^[17]) to: (i) assess the quality of guidelines; (ii) provide a methodological strategy for the development of guidelines; and (iii) inform what information ought to be reported in guidelines, and how it should be reported.

AGREE II outlines six unique domains of guideline quality:^[17]

Domain 1. Scope and purpose 'is concerned with the overall aim of the guideline, the specific health questions, and the target patient population'.

Domain 2. Stakeholder involvement 'focuses on the extent to which the guideline was developed by the appropriate stakeholders and represents the views of its intended users'.

Domain 3. Rigour of development 'relates to the process used to gather and synthesise the evidence, the methods to formulate the recommendations, and to update them'. It ensures an explicit link between the recommendation and the supporting evidence and facilitates an assessment of whether health benefits, adverse effects and risks were considered in formulating the recommendations.

Domain 4. Clarity of presentation 'deals with the language, structure, and format of the guideline'.

Domain 5. Applicability 'pertains to the likely barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the guideline'.

Domain 6. Editorial independence 'is concerned with the formulation of recommendations not being unduly biased by competing interests'. Any potential conflict of interest must be reported and appropriately managed.

An editorial sub-committee under the leadership of Prof. Marc Blockman has been appointed by the SAMJ to undertake this responsibility. This represents an important step in ensuring the rigour and validity of future guidelines available in SA.

 

Conclusion

A quality guideline is one in which there is confidence that the potential biases of guideline development have been addressed adequately, one which is deemed to be externally valid via a peer-review process, and one which is locally applicable and feasible.^[18] The introduction of this tool, as a critical appraisal mechanism, is an important advance in the use of evidence-based medicine principles (in its true sense) in the development and publication of practice guidelines in SA. Through ensuring the application of sufficient rigour in the development process, there is a firm belief that a clinician's ability to make informed clinical decisions will be enhanced, ultimately leading to improved patient care through discouraging the use of ineffective and wasteful interventions. This, in turn, will result in more efficient resource utilisation, will elevate the level of trust in the guideline itself, and is likely to impact on the ability and willingness to implement such a guideline in practice.

Declarations of interest. MB is a member of the National Essential Medicines List Committee and is chair of the Tertiary/Quaternary Essential Medicines List Committee. He is also a member of the Western Cape Provincial Pharmacy and Therapeutics Committee and its executive committee, a member of the Medicines Control Council (MCC) and its central clinical committee, and chair of the MCC's pharmacovigilance committee. MB also consults for various private sector drug and therapeutics committees, including those for Liberty Health and Metropolitan Health Risk Management. KC is a member of the Western Cape Provincial Pharmacy and Therapeutics Committee and its executive committee. AG is a member of the National Essential Medicines List Committee, the KwaZulu-Natal Provincial Pharmacy and Therapeutics Committee and the World Health Organization Expert Panel on Drug Policies and Management. KJ is a senior pharmaceutical policy specialist in the Essential Medicines Programme at the National Department of Health. TK has received funding from the South African Medical Research Council for the South African Guideline Excellence (SAGE) Project to explore guideline development and implementation in the South African primary healthcare context. JM is a member of the Gauteng Provincial Pharmacy and Therapeutics Committee and consults on health economics and health technology assessment to Discovery Health. AP is a member of the National Essential Medicines List Committee, a member of the Tertiary/Quaternary Essential Medicines List Committee, and chair of the Adult Essential Medicines Committee. He is also chair of the Eastern Cape

Provincial Pharmacy and Therapeutics Committee. BT oversees the process of health technology assessment, including medicines evaluation, for medical schemes contracted to Medscheme. RW is responsible for medicine policy development at Liberty Health Holdings.

 

Roger Wiseman

Liberty Health Holdings, Cape Town, South Africa roger.wiseman@vmed.co.za

Karen Cohen

Division of Clinical Pharmacology, Department of Medicine, Faculty of Health Sciences, University of Cape Town, South Africa

Andy Gray

Division of Pharmacology, Discipline of Pharmaceutical Sciences, University of KwaZulu-Natal, Durban, South Africa

Khadija Jamaloodien

National Department of Health, South Africa

Tamara Kredo

South African Cochrane Centre, South African Medical Research Council, Cape Town, South Africa

Jacqui Miot

Department of Pharmacy and Pharmacology, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa

Andy Parrish

Department of Internal Medicine, Walter Sisulu University, Mthatha, and Cecilia Makiwane Hospital, East London, Eastern Cape, South Africa

Bettina Taylor

Medscheme Health Policy Unit, Cape Town, South Africa

Marc Blockman

Division of Clinical Pharmacology, Department of Medicine, Faculty of Health Sciences, University of Cape Town, South Africa

 

1. James PA, Oparil S, Carter BL, et al. Evidence-based guideline for the management of high blood pressure in adults: Report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA 2014;311(5):507-520. [http://dx.doi.org/10.1001/jama.2013.284427]        [ Links ]

2. Sox HC. Assessing the trustworthiness of the guideline for the management of high blood pressure in adults. JAMA 2014;311(5):472-474. [http://dx.doi.org/10.1001/jama.2013.284429]        [ Links ]

3. Wright JT, Fine LJ, Lackland DT, et al. Evidence supporting a systolic blood pressure goal of less than 150 mm Hg in patients aged 60 years or older: The minority view. Ann Intern Med 2014;160(7):499-503. [http://dx.doi.org/10.7326/M13-2981]        [ Links ]

4. Hawkes N. Use of β blockers during surgery may have caused 10 000 deaths in the UK. BMJ 2013;347:f4914. [http://dx.doi.org/10.1136/bmj.f4914]        [ Links ]

5. Bouri S, ShunShin MJ, Cole GD, et al. Heart 2014;100(6):456-464. [http://dx.doi.org/10.1136/heartjnl-2013-304262]        [ Links ]

6. Tricoci P, Allen JM, Kramer JM, et al. Scientific evidence underlying the ACC/AHA clinical practice guidelines JAMA 2009;301(8):831-841. [http://dx.doi.org/10.1001/jama.2009.205]        [ Links ]

7. Lee DH, Vielemeyer O. Analysis of overall level of evidence behind Infectious Diseases Society of America practice guidelines. Arch Intern Med 2011;171(1):18-22. [http://dx.doi.org/10.1001/archinternmed.2010.482]        [ Links ]

8. Feuerstein JD, Akbari M, Gifford AE, et al. Systematic analysis underlying the quality ofthe scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines. Mayo Clin Proc 2014;89(1):16-24. [http://dx.doi.org/10.1016/j.mayocp.2013.09.013]        [ Links ]

9. Institute of Medicine; Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, eds. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press, 2011. http://wwwiom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Standards.aspx (accessed 3 April 2014).         [ Links ]

10. Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P, for the Board of Trustees of the Guidelines International Network. Guidelines International Network: Toward international standards for clinical practice guidelines. Ann Intern Med 2012;156(7):525-531. [http://dx.doi.org/10.7326/0003-4819-156-7-201204030-00009]        [ Links ]

11. Woolf SH, Grol R, Hutchison A, et al. Potential benefits, limitations, and harms of clinical guidelines. BMJ 1999;318(7182):527-530. [http://dx.doi.org/10.1136/bmj.318.7182.527]        [ Links ]

12. Sackett DL, Rosenberg WMC, Gray JA, et al. Evidence based medicine: What it is and what it isn't. BMJ 1996;312(7023):71-72. [http://dx.doi.org/10.1136/bmj.312.7023.71]        [ Links ]

13. Evans D. Hierarchy of evidence: A framework for ranking evidence evaluating healthcare interventions. J Clin Nurs 2003;12(1):77-84. [http://dx.doi.org/10.1046/j.1365-2702.2003.00662.x]        [ Links ]

14. Gibson JL, Martin DK, Singer PA. Evidence, economics and ethics: Resource allocation in health services organizations. Healthc Q 2005;8(2):50-59. [http://dx.doi.org/10.12927/hcq..17099]        [ Links ]

15. Chinnock P, Siegfried N, Clarke M. Is evidence-based medicine relevant to the developing world? PLoS Med 2005;2(5):e107. [http://dx.doi.org/10.1371/journal.pmed.0020107]        [ Links ]

16. American College of Physicians. How Can Our Nation Conserve and Distribute Health Care Resources Effectively and Efficiently? Philadelphia: American College of Physicians, 2011. http://www.acponline.org/advocacy/current_policy_papers/assets/health_care_ resources.pdf (accessed 13 March 2014).         [ Links ]

17. Brouwers M, Kho ME, Browman GP, et al., on behalf of the AGREE Next Steps Consortium. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010;182:E839-E842. [http://dx.doi.org/10.1503/cmaj.090449]        [ Links ]

18. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. Qual Saf Health Care 2003;12(1):18-23.         [ Links ]

 

 

For the South African Medical Journal Editorial Sub-Committee for Guideline Review