IN PRACTICE
ISSUES IN MEDICINE

 

Meta-analytic magic, ivermectin, and socially responsible reporting

 

 

A G Parrish^I; M Blockman^{II, III}; K Cohen^{IV, V}; H Dawood^{VI, VII}; R de Waal^{VIII, IX}; A L Gray^{X, XI}; T Kredo^{XII, XIII}; T D Leong^XIV; J Nel^{XV, XVI}; H Rees^{XVII, XVIII}; G Reubenson^{XIX, XX}

^IMB ChB, DA (SA), FCP (SA), MMed (Med), MMedSci (Clin Epi); Chair, National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa; Department of Internal Medicine, Faculty of Health Sciences, Walter Sisulu University, Mthatha, South Africa; and Frere and Cecilia Makiwane hospitals, East London, South Africa
^IIMB ChB, PG Dip Int Res Ethics, MMed (Clin Pharm); National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^IIIMB ChB, PG Dip Int Res Ethics, MMed (Clin Pharm); Division of Clinical Pharmacology, Department of Medicine, Faculty of Health Sciences, University of Cape Town, South Africa
^IVMB ChB, MMed (Clin Pharm), MSc (Epi), MCFP (SA), Dip HIV Man (SA), Dip Obst (SA); National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^VMB ChB, MMed (Clin Pharm), MSc (Epi), MCFP (SA), Dip HIV Man (SA), Dip Obst (SA); Division of Clinical Pharmacology, Department of Medicine, Faculty of Health Sciences, University of Cape Town, South Africa
^VIMB ChB, FCP (SA), MSc (Epi); National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^VIIMB ChB, FCP (SA), MSc (Epi); Department of Medicine, Grey's Hospital, Pietermaritzburg, KwaZulu-Natal, South Africa; and CAPRISA (Centre for the AIDS Programme of Research in South Africa), University of KwaZulu-Natal, Durban, South Africa
^VIIIMB ChB, MPH; National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^IXMB ChB, MPH; Centre for Infectious Disease Epidemiology and Research, School of Public Health and Family Medicine, Faculty of Health Sciences, University of Cape Town, South Africa
^XBPharm, MSc (Pharm); National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^XIBPharm, MSc (Pharm); Division of Pharmacology, Discipline of Pharmaceutical Sciences, School of Health Sciences, University of KwaZulu-Natal, Durban, South Africa
^XIIMB ChB, MMed (Clin Pharm), PhD; National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^XIIIMB ChB, MMed (Clin Pharm), PhD; Division of Clinical Pharmacology, Department of Medicine, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa - GRADE Network South Africa; and Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa
^XIVBPharm, MSc (Med), MSc (Clin Epi); Essential Drugs Programme, Affordable Medicines Directorate, National Department of Health, Pretoria, South Africa
^XVMB ChB, FCP (SA), Cert ID (SA), DTM&H; National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^XVIMB ChB, FCP (SA), Cert ID (SA), DTM&H; Department of Internal Medicine, School of Clinical Medicine, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa
^XVIIGCOB, OBE, MB BCh, MA (Cantab), MRCGP, DRCOG, DCH; National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^XVIIIGCOB, OBE, MB BCh, MA (Cantab), MRCGP, DRCOG, DCH; Wits Reproductive Health and HIV Institute, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa
^XIXMB BCh, FC Paed (SA), DCH, DTM&;H National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa
^XXMB BCh, FC Paed (SA), DCH, DTM&H; Vice-Chair, National Essential Medicines List COVID-19 subcommittee, National Department of Health, Pretoria, South Africa; Department of Paediatrics and Child Health, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa; and Rahima Moosa Mother and Child Hospital, Johannesburg, South Africa

Correspondence

 

 

---

ABSTRACT

Some clinicians prescribe ivermectin for COVID-19 despite lack of support from any credible South African professional body. They argue that when faced by clinical urgency, weak signals of efficacy should trigger action if harm is unlikely. Several recent reviews found an apparent mortality benefit by including studies at high risk of bias and with active rather than placebo controls. If these studies are discounted, the pooled mortality effect is no longer statistically significant, and evidence of benefit is very weak. Relying on this evidence could cause clinical harm if used to justify vaccine hesitancy. Clinicians remain responsible for ensuring that guidance they follow is both legitimate and reliable. In the ivermectin debate, evidence-based medicine (EBM) principles have largely been ignored under the guise that in a pandemic the 'rules are different', probably to the detriment of vulnerable patients and certainly to the detriment of the profession's image. Medical schools and professional interest groups are responsible for transforming EBM from a taught but seldom-used tool into a process of lifelong learning, promoting a consistent call for evidence-based and unconflicted debate integral to clinical practice.

---

 

 

Evidence synthesis, and vulnerabilities in the translation of evidence into recommendations

Five recent reviews of ivermectin in the management of COVID-19 have achieved widespread social media and public prominence. Two were published in the American Journal of Therapeutics^[1,2]and two are preprints in journals in the Oxford University Press stable.^[3,4] The fifth is a recently released Cochrane review.^[5] In three of the five reviews, the summary reports mortality benefits using very positive language despite low-quality evidence rendering such a conclusion highly uncertain. For example, Kory et al.^[1]state: 'Meta-analyses based on 18 randomized controlled treatment trials of ivermectin in COVID-19 have found large, statistically significant reductions in mortality, time to clinical recovery, and time to viral clearance.'

Many readers may have neither the time nor the inclination to read the body of the reviews in sufficient detail to pick up the extreme frailty of such conclusions, and instead some have further disseminated these 'news-bite' affirmations of efficacy. Editors and reviewers have a responsibility to consider the implications of how information is presented, in this case to avoid global overuse of a medicine of undetermined efficacy. In some settings, patients have resorted to illegally imported or veterinary products when registered oral solid dosage forms were unavailable.

 

The importance of meticulous evidence-based medicine during the SARS-CoV-2 pandemic

The premise that authors generate unbiased systematic reviews and journals conduct peer review of adequate quality is foundational to the scientific medical enterprise, and evidence-based medicine (EBM) has taken this as a core principle. By moving away from opinion (eminence)-guided practice, EBM has advanced quality of care, but struggles to find a voice when weak positive evidence is being promoted and the alternative view is simply 'wait and see'. When opinion leaders use electronic media and medical journals to express a position 'backed by evidence', protestations of uncertainty may be overwhelmed by perceived authority and urgency.

In the past few decades, reaching conclusions about the efficacy and safety of medical interventions has moved from reliance on expert opinion to a more public and formalised collaborative process of searching for evidence, appraising its quality, and synthesising the data to inform decision-making. The conclusions reached are critically dependent on unbiased adherence to all steps, and on the quality of the underlying evidence. A critical final process entails transforming conclusions about strength and direction of evidence into clinically useful recommendations, often by groups independent of the review process. A key principle is that decisions can and should be made using the best available evidence, even when this is imperfect. Relying on anecdotal and weak observational evidence can be dangerously misleading, particularly in situations of limited biological plausibility.

Considerable time and effort are needed to perform a systematic review, and when done well, these are valuable resources for clinical and policy decision-making. Systematic reviews, like any human endeavour, have vulnerabilities. The more obvious of these can be detected using quality appraisal tools, but there are other more subtle ways in which bias can occur, rendering results less reliable. The rigour of the Cochrane process, and formal collaborative use of software such as RevMan,^[6] are specifically designed to address many of these issues. Registration of systematic review protocols, such as on the International Prospective Register of Systematic Reviews (https://www.crd.york.ac.uk/prospero/), PROSPERO, also aims to minimise bias while avoiding duplication of effort.

Examples of bias include undeclared intellectual conflicts of interest (where reviewers may not approach a research question entirely objectively), inconsistent rigour in 'risk of bias' assessment (where studies supporting a particular position may be assessed differently), inclusion of studies of low reliability, and issues with actual meta-analytical methods. The last is particularly problematic in an era where software allows almost instantaneous iterative data analysis so that it can be difficult to tell whether a submitted data analysis plan is truly based on a priori scientific considerations or post hoc adoption of a model found to yield preferred results. Other issues in meta-analytical technique, such as the handling of double-zero studies, weighting methodologies, and the handling of heterogeneity and potential small-study effects, engender vigorous debate, as in many other developing areas of statistics.

The drive to produce a plausible forest plot may lead to inclusion of studies at high risk of bias. Pooling weak evidence will still generate an effect estimate. When individual studies contribute very small numbers of events, sensitivity analyses omitting several studies at a time may leave the overall effect estimator unchanged, but this does not mean that the result is robust.

The Bryant et al.^[2] review raises several concerns. The authors themselves acknowledge that information for all outcomes except mortality was weak or very weak. For mild COVID-19, information on mortality was available in 5 of 11 trials, with a total of 13 deaths, and for severe COVID-19, in 5 trials, with a total of 539 patients, 200 of which were contributed by the recently withdrawn study by Elgazzar et al.^[7] The total number of patients in the ivermectin arm of the López-Medina et al.^[8] study is given as 275 in the forest plots and was reported as 200 in the published article, reflecting a decision to use 'as treated' numbers rather than those in the primary analysis. It is these study-level decisions, as well as the decisions about which studies to include, that lead to differences in reported review results.

 

Inclusion of the results of preprint studies in meta-analyses circumvents peer review

Peer review of articles prior to publication is a valuable tool for detection of error, ranging from simple mistakes in calculation and reporting to more substantive flaws in design and implementation. In extreme cases, peer review may detect fraud. Preprint servers have provided rapid access to new research in the current pandemic, but this accelerated access comes at the cost of reduced certainty about article quality. Published reviews using preprint reports bypass the peer review process and may lend undeserved legitimacy to these reports, as well as creating a critical weakness in the review itself.

The frailty of the conclusion of ivermectin efficacy has been further eroded by the recent withdrawal of the study by Elgazzar et al.,^'71which was not considered by Bryant et al.^[2]to be at high risk of bias. (Bryant et al. went so far as to state that use of preprint server studies did not constitute a risk of bias because they peer reviewed the studies themselves.) If the authors' own meta-analysis is repeated, first without the studies classified by the authors themselves as at high risk of bias, then also excluding the Elgazzar study and others with active controls (comparison with another medicine rather than a placebo), the pooled effect drifts back towards null (Fig. 1): there is no mortality benefit in either mild or severe disease subgroups, and no mortality benefit overall (relative risk (RR) 0.41; 95% confidence interval (CI) 0.13 - 1.22). This is a finding similar to that reached by Roman et al.^'41and reassuring in that the Elgazzar study was also correctly identified and omitted. The wisdom of even performing a meta-analysis with only six studies and so few events is also moot -most events (61%) occurred in the trial labelled as Fonesca et al. in the review (but now published by Galan et al.^[9]), with no mortality benefit (RR 1.06; 95% CI 0.58 - 1.94).

The review by Hill et al.^[3]placed the study by Niaee et al.,^[10]which enrolled patients with varying disease severity, in their mild COVID-19 group, generating a group of 'mild to moderate' disease. This analysis leaves the effect size significant for this group after removing the study by Elgazzar et al.,^[7]but based on a very small numbers of events; overall, the mortality benefit is not statistically significant. The Niaee study, which has been available in preprint form for more than 6 months and has not yet been published in a peer-reviewed publication, is itself concerning in that the table presenting participant characteristics contains potential errors. One example is that diastolic blood pressure is given as a median with an interquartile range, and overall this is given as 80 (80 - 80), which, if correct, would mean that half the patients had a diastolic blood pressure of exactly 80 mmHg. In total, 29% of patients in the study by Niaee et al.^[10]were SARS-CoV-2 polymerase chain reaction-negative (47% overall in the two control groups and 20% overall in the four ivermectin groups). It is assumed that these patients' entry into the study was based on positive findings on a chest computed tomography scan. It is this sort of detail that conscientious and competent peer review is designed to address. If a conservative approach is taken and the study by Niaee et al.^[10]is also excluded from the pooled analysis while these concerns are addressed, the potential overall mortality RR is a non-significant 0.67 (95% CI 0.25 - 1.79), drifting even closer to null. This movement of the estimate of treatment effect towards the null is an expected and frequent phenomenon as small weak studies are re-interrogated, and studies with a higher publication hurdle (because of negative results) are finally released.

The recently released high-quality Cochrane review^[5] reaches similar conclusions: only 4 studies (2 for mild and 2 for moderate/ severe disease) were considered appropriate for inclusion in the pooled mortality analyses, and no convincing evidence of efficacy was found. The meticulous attention to detail in this well-conducted review demonstrates the enormous effort required to perform a systematic review well. It can be argued that strict inclusion criteria, such as limiting the review to randomised controlled trials, have left out a large body of evidence. However, the aim is to prevent making mistakes by excluding unreliable information: as the proportion of unreliable information rises, so does the chance of reaching a mistaken conclusion. There is a place for observational data, particularly in the assessment of safety, but reliable estimates of efficacy must be based on reliable data. The emotional appeal of 'real-world evidence', and the demands of a pandemic, should not result in over-reliance on questionable or potentially biased data. Patients' lives rely on the rigour of the entire process, from clinical trial design and execution to the development of clinical practice guidelines.

Fig. 1 demonstrates the effect of changing trial inclusion criteria. A liberal policy of inclusion yields a larger effect size, and also generates a 'significant' benefit; as the high-risk studies, and the studies using an active comparator, are progressively excluded, the effect drifts back towards null; and the inclusion of a large negative study such as that by Vallejos et al.^[11] pushes this into a clearer lack of any signal of benefit, although the evidence is still incomplete.

 

Prescribing pressures and harm as opportunity cost

The ivermectin argument that 'even if not clearly efficacious, it can't do any harm' has opportunity costs: individuals may feel safer taking ivermectin, and indulge in more risky behaviour, its promotion on some social media sites places it as an alternative to vaccination, and its inclusion in therapeutic cocktails lends credibility to other useless or potentially harmful components. Users, convinced of unproven benefits, may be reluctant to enrol in appropriate, randomised, placebo-controlled trials.

Another clear harm of this approach is that it legitimises a laissez-faire approach to clinical care, where instead of prescribers relying on regulatory authorities and conscientious evidence-based practice, it is seen as reasonable to prescribe based on opinions expressed on social media platforms.

 

Clinical influences of group think and social media facilitation

The psychological process of group think^[12] further exacerbates this. The need for consensus leads to ignoring divergent opinions and explanations; this process is self-perpetuating in that the more strongly a group embraces a position, the more invested it becomes in demonstrating its validity, leading to 'cherry-picking' of supporting evidence. This selected evidence further strengthens beliefs, to the extent that countervailing explanations are stridently opposed, often accompanied by ad hominem attacks on those attempting to voice contrary views.

Social media facilitates group think, not only by allowing rapid dissemination of ever more extreme positions, but also by vastly increasing membership through international recruitment. In larger groups there will be higher absolute numbers at the extremes, leading to further polarisation.^[13]

 

Societal expectations for medical professionalism, and the clinician as scientist

Society entrusts clinicians to provide healthcare guidance. The implicit assumption is that the individual clinician-scientist will have carefully researched an issue and reached a reasoned and safe decision, which is then conveyed to a patient. When evidence is weak, complex and value laden, this is technically difficult, and reliance on unbiased guidance carefully synthesised by experts is entirely appropriate. It is precisely in times of uncertainty and perceived haste that regulatory authorities and careful reviews are most useful in preventing therapeutic mayhem. In South Africa, no credible professional body has yet supported the use of ivermectin, and the promoters of its use are doing so in the general press, on websites and through social media. It is arguably unethical, medicolegally fraught, and unprofessional to adopt a position so far removed from that entrusted to us by society.

Clinicians remain responsible for ensuring that guidance they follow is both legitimate and reliable. In the ivermectin debate, EBM principles have largely been ignored under the guise that during a pandemic the 'rules are different', probably to the detriment of vulnerable patients and certainly to the detriment of the profession's image. Medical schools and professional interest groups are responsible for transforming EBM from a taught but seldom used tool into a process of lifelong learning, promoting a consistent call for evidence-based and unconflicted debate integral to clinical practice.

Declaration. None.

Acknowledgements. None.

Author contributions. The National Essential Medicines List COVID-19 subcommittee conceptualised the manuscript. AGP and MB were the primary authors, and AGP incorporated co-author feedback. AGP and MB contributed to the final drafting of the article. All authors provided critical feedback and contributed to the final manuscript. AGP is the chair and GR vice-chair of the subcommittee.

Funding. None other than that TK is partly supported by the Research, Evidence and Development Initiative (READ-It). READ-It (project number 300342-104) is funded by UK aid from the UK government; however, the views expressed do not necessarily reflect the UK government's official policies.

Conflicts of interest. The authors are all members of the National Essential Medicines List COVID-19 subcommittee, which produced an updated rapid review^[14] on ivermectin, but have no financial conflicts of interest related to this medication.

 

References

1. Kory P, Meduri G, Varon J, Iglesias J, Marik PE. Review of the emerging evidence demonstrating the efficacy of ivermectin in the prophylaxis and treatment of COVID-19. Am J Ther 2021;28(3):e299-e318. https://doi.org/10.1097/MJT.0000000000001377        [ Links ]

2. Bryant A, Lawrie T, Dowswell T, et al. Ivermectin for prevention and treatment of COVID-19 infection: A systematic review, meta-analysis, and trial sequential analysis to inform clinical guidelines. Am J Ther 2021;28(4):e434-e460. https://doi.org/10.1097/MJT.0000000000001402        [ Links ]

3. Hill A, Garratt A, Levi J, Falconer J, et al. Meta-analysis of randomized trials of ivermectin to treat SARS-CoV-2 infection. Open Forum Infect Dis 2021;ofab358 (epub 6 July 2021). https://doi.org/10.1093/ofid/ofab358        [ Links ]

4. Roman YM, Burela PA, Pasupuleti V, et al. Ivermectin for the treatment of COVID-19: A systematic review and meta-analysis of randomized controlled trials. Clin Infect Dis 2021;ciab591 (epub 28 June 2021). https://doi.org/10.1093/cid/ciab591        [ Links ]

5. Popp M, Stegemann M, Metzendorf M-I, et al. Ivermectin for preventing and treating COVID-19. Cochrane Database Syst Rev 2021, Issue 7. Art. No.: CD015017. https://doi.org/10.1002/14651858.CD015017.pub2        [ Links ]

6. Review Manager (RevMan) (Computer program). Copenhagen: Nordic Cochrane Centre, Cochrane Collaboration, 2014. https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman/revman-5-download (accessed 14 August 2021).         [ Links ]

7. (Preprint withdrawn on 14 July 2021) Elgazzar A, Hany B, Youssef SA, Hafez M, Moussa H. Efficacy and safety of ivermectin for treatment and prophylaxis of COVID-19 pandemic. Research Square. https://www.researchsquare.com/article/rs-100956/v4 (accessed 5 January 2021).         [ Links ]

8. López-Medina E, López P, Hurtado IC, et al. Effect of ivermectin on time to resolution of symptoms among adults with mild COVID-19: A randomized clinical trial. JAMA 2021;325(14):1426-1435. https://doi.org/10.1001/jama.2021.3071        [ Links ]

9. Galan LEB, dos Santos NM, Asato MS, et al. Phase 2 randomized study on chloroquine, hydroxychloroquine or ivermectin in hospitalized patients with severe manifestations of SARS-CoV-2 infection. Path Glob Health 2021;115(4):235-242. https://doi.org/10.1080/20477724.2021.1890887        [ Links ]

10. Niaee MS, Gheibi N, Namdar P, et al. (Preprint). Ivermectin as an adjunct treatment for hospitalized adult COVID-19 patients: A randomized multi-center clinical trial. Research Square 2020 (epub 24 November 2020). https://www.researchsquare.com/article/rs-109670/v1 (accessed 19 July 2021).         [ Links ]

11. Vallejos J, Zoni R, Bangher M, et al. Ivermectin to prevent hospitalizations in patients with COVID-19 (IVERCOR-COVID19): A randomized, double-blind, placebo-controlled trial. BMC Infect Dis 2021;21:635. https://doi.org/10.1186/s12879-021-06348-5        [ Links ]

12. Bang D, Frith CD. Making better decisions in groups. Royal Soc Open Sci 2017;4:170193. https://doi.org/10.1098/rsos.170193        [ Links ]

13. Isenberg DJ. Group polarization: A critical review and meta-analysis. J Pers Soc Psychol 1986;50(6):1141-1151. https://doi.org/10.1037/0022-3514.50.6.1141        [ Links ]

14. National Essential Medicines List COVID-19 subcommittee. Rapid review of ivermectin for COVID-19. Update 30 July 2021. http://www.health.gov.za/covid-19-rapid-reviews/ (accessed 14 August 2021).         [ Links ]

 

 

Correspondence:
A G Parrish
andygp@mweb.co.za

Accepted 11 August 2021