CORRESPONDENCE

 

Naude et al. avoid answering the essential question: Mistake or mischief?

 

 

To the Editor: It is common cause that the Naude/Stellenbosch University/University of Cape Town meta-analysis^[1] played a decisive role in the multimillion rand prosecution of Prof. T Noakes by the Health Professions Council of South Africa (HPCSA), which has now been ongoing for more than 3 years. The complainant, Claire Julsing-Strydom, referred to the importance of a meta-analysis generally and/or the Naude et al.^[1] meta-analysis specifically on 12 separate occasions during her testimony. On 24 November 2015, under oath, she said of the Naude et al.^[1]publication: 'So before any media statements could be made we had to get that information and all these associations were waiting on that. It is not like the way you are saying it, it is not like everybody joined together to now make a statement against Prof. Noakes. We were all waiting for the evidence to be published.' Another prosecution witness, Prof. H Vorster, referred to the Naude et al.^[1]meta-analysis five times and quoted from it verbatim once. A third prosecution witness, Prof. A Dhansay, referenced the meta-analysis twice, using the term 'Cochrane' to ensure that it was afforded the appropriate esteem. Without the 'correct conclusion' from this meta-analysis, it is possible that the HPCSA trial against Noakes might never have happened. Therefore, the importance of the Naude et al.^[1]meta-analysis extends far beyond any role purely as a neutral scientific publication.

Had we realised the disproportionate consideration given to this ostensibly innocuous publication in the HPCSA trial, we would have examined it sooner. When we did examine it, we found the part used to claim that 'Banting had been debunked' to be replete with errors. We first presented these errors at the HPCSA/Noakes hearing on 24 October 2016, and then in the SAMJ.^[2] Other than a misplaced article in the Cape Times, including comments from the authors on 20 December 2016,^[3] the Naude et al.^[4] letter published in the March 2017 edition of the SAMJ is the first response to our allegations.

The authors cheaply suggest that we show a 'lack of understanding' of their protocol. We understand the Naude et al. ^[1] protocol only too well. Indeed, we appear to understand it rather better than do its authors:

• We understand that the authors set isolcaloric as a criterion, which would mitigate the satiety advantage of low(er)-carbohydrate (CHO) diets.

• We understand that the authors selected studies with an average CHO intake of 35% (35% fat, 30% protein) to represent 'low CHO' diets, which is substantially different from the 5% CHO (<50 g/day), moderate-protein and high-fat diet that is used for the therapeutic management of obesity and type 2 diabetes mellitus.

• We understand that they set an inclusion criterion of 25 - 35% fat in the so-called balanced diet. This criterion was reiterated in Tables 2 and 3^[1] and yet ignored by the authors, as they included two studies^[5,6] that failed their own criterion. These errors remain unaddressed by the authors in any of their responses.

• We understand that they set the key outcome measure as 'total weight change'. They then used end weight, with no reference to start weight, in two studies,^[7,8] which was in breach of their own protocol and absurd. These errors remain unaddressed by the authors.

In their response to the SAMJ, the authors have accepted only one of the numerous errors that we documented in their article - their admission that they included a duplicate study.^[9] This alone is grounds for retraction of the article.^[10]

The authors have not addressed any of the other numerous errors, material or otherwise, which we documented.^[2]

They have not addressed their sub-grouping, which resulted in the protein intake in the diet groups of their so-called high-protein and high-fat studies averaging 31.5 - 32.5% and 28.4%, respectively, likely a biologically insignificant difference. We described this sub-grouping as 'not necessary, not justified and not robust'.

The authors have not addressed their patently false claim that they used 52-week weight loss data for Wycherley et al.,^[11] when they used data from 12 weeks instead. Use of the 52-week data would have favoured the lower-CHO intervention.

The authors have not addressed using the wrong comparator diet for Krauss et al.,^[12] which unnecessarily introduced a second variable and was to the advantage of the so-called 'balanced' diet.

The authors have not explained from where they extracted the weight loss data of 2.65 kg for both the diet and control groups for Krauss et al.^[12] Those specific data could not be found in the original publication. Instead, the actual data reported by Krauss et al. ^[12] slightly favoured the lower-CHO diet intervention. The authors have not addressed reporting the weight loss as 7.95 kg in the control diet rather than the 7.9 kg actually reported by Farnsworth et al.^[13]

Instead of addressing these errors in detail, point by point, the authors have informed us of new errors that we missed. They wrote: 'We used data from intention-to-treat analyses (and only if not reported, we used data from per-protocol analyses).' This was not declared in their original protocol. Intention-to-treat (ITT) is mentioned in only two places in their 30-page article. It first appears in Table 8, which is the table assessing risk of bias. It then re-appears on p. 14 in the 'Results: Risk of bias' section. This is where ITT should appear - it is a measure of attrition bias (incomplete outcome data). The authors reported: 'Seven trials had low risk of attrition bias, with four reporting no attrition and three performing intention-to-treat analysis.^['1]

The authors appear now to be confessing that they used ITT numbers instead of completers for some studies and not for others. A re-examination of their meta-analysis (Fig. 3), confirms that in three cases the participant numbers used differ to those extracted in Table 5, which in turn differ from the numbers used in the original publications on five occasions. ITT numbers have been used for Frisch et al.,^[14] Klemsdal et al.^[15] and Sacks et al.,^[5] notwithstanding that Sacks should not have been included from the outset, having failed the authors' inclusion criteria.

These three studies favoured the low(er)-CHO diet, although the weight loss was reported the wrong way round by the authors for Sacks et al.;^[5] therefore, it did not appear this way. The Cochrane Handbook for Systematic Reviews of Interventions^[16] states that 'ITT analyses tend to bias the results towards no difference (section 16.2). This choice thus aided the authors' conclusion that there is little or no difference between the diets. Frisch et al.^[14] reported that 'Missing data were replaced with baseline data in the intention-to-treat analysis'. Klemsdal et al.^[15] used the last observation carried forward (LOCF) technique and Sacks et al.^[5] imputed an estimated weight gain of 0.3 kg/month for their ITT approach. But Cochrane warned of 'A common, but potentially dangerous, approach to dealing with missing outcome data is to impute outcomes and treat them as if they were real measurements ... Such procedures can lead both to serious bias and to confidence intervals that are too narrow . LOCF procedures can also lead to serious bias.'

In addition to this newly announced ITT methodology, which was not set out in their original protocol, the authors added another potential deflection, with an unsolicited reference to 'standard mean difference', which neither of us used. We used mean weight loss and standard deviations (SDs), as the authors did (apart from the De Luis et al. ^[7,8] studies, for which they used end weight). We converted standard error measurements to SDs where necessary, as the authors did, and we noted that we concurred with the authors' conversion on four occasions, but found them to have erred with this calculation on three other occasions. These errors remain unaddressed by the authors.

As regards the accusation of 'common mistake', the researchers will be well aware, as are we, that in meta-analysis significant means statistically significant. The size of the statistical significance is immaterial. What is significant, in the non-statistical use of the word, is that our repeat of the authors' meta-analysis, using their methodology, but without the errors, produced a different result - a result that would not have given those keen to prosecute Prof. Noakes the ammunition they were 'waiting for'.

It is understandable that the authors chose rather to raise new issues (ITT, standard mean difference and significance) and to ignore the numerous errors that we identified in our critique, presumably because they have no cogent answers. As a result, those material errors have remained unanswered since October 2016 and unless addressed, they render the article worthless, other than of retraction.

Given that only one error has been addressed and accepted (the duplication), we may never receive an answer to our research question: was this mistake or mischief? We may also never know if Prof. Noakes would have suffered for years in the way he has, had this article not made competence or conspiratorial errors.

Declarations of interest. ZH receives income from writing and from two small self-employment businesses: The Harcombe Diet Co. and Columbus Publishing. TN is the author of the books Lore of Running and Waterlogged and co-author of The Real Meal Revolution, Raising Superheroes, The Banting Pocket Guide and Challenging Beliefs. All royalties from the sales of The Real Meal Revolution, Raising Superheroes and The Banting Pocket Guide and related activities are donated to the Noakes Foundation, of which he is the chairman and which funds research on insulin resistance, diabetes and nutrition as directed by its Board of Directors. Money from the sale of other books is donated to the Tim and Marilyn Noakes Sports Science Research Trust, which funds the salary of a senior researcher at the University of Cape Town, South Africa. The research focuses on the study of skeletal muscle in African mammals, with some overlap to the study of type 2 diabetes in carnivorous mammals and of the effects of (scavenged) sugar consumption on free-living (wild) baboons.

 

Zoë Harcombe

Public Health Nutrition, Cardiff, UK

Tim Noakes

Department of Human Biology, Faculty of Health Sciences, University of Cape Town, South Africa. noakes@iafrica.com

 

References

1. Naude CE, Schoonees A, Senekal M, Young T, Garner P, Volmink J. Low carbohydrate versus isoenergetic balanced diets for reducing weight and cardiovascular risk: A systematic review and meta analysis. PLoS ONE 2014;9(7):e100652. https://doi.org/10.1371/journalpone.0100652        [ Links ]

2. Harcombe Z, Noakes TD. The universities of Stellenbosch/Cape Town low-carbohydrate diet review: Mistake or mischief? S Afr Med J 2016;106(12):1179-1182. https://doiorg/10.7196/SAMJ.2016.v106i12.12072        [ Links ]

3. Isaacs L. Noakes disputes diet study. Cape Times, 20 December 2016.         [ Links ]

4. Naude CE, Schoonees A, Senekal M, Garner P, Young T, Volmink J. Reliable systematic review of low-carbohydrate diets shows similar weight-loss effects compared with balanced diets and no cardiovascular risk benefits: Response to methodological criticisms. S Afr Med J 2017;107(3):170. https://doi.org/10.7196/SAMJ.2017.v107i3.12382        [ Links ]

5. Sacks FM, Bray GA, Carey VJ, et al. Comparison of weight-loss diets with different compositions of fat, protein, and carbohydrates. N Engl J Med 2009;360(9):859-873. https://doi.org/10.1056/NEJMoa0804748        [ Links ]

6. Keogh JB, Brinkworth GD, Clifton PM. Effects of weight loss on a low-carbohydrate diet on flow- mediated dilatation, adhesion molecules and adiponectin. Br J Nutr 2007;98(4):852-859. https://doi.org/10.1017/s0007114507747815        [ Links ]

7. De Luis DA, Aller R, Izaola O, et al. Evaluation of weight loss and adipocytokines levels after two hypocaloric diets with different macronutrient distribution in obese subjects with rs9939609 gene variant. Diabetes Metab Res Rev 2012;28(8):663-668. https://doi.org/10.1002/dmrr.2323        [ Links ]

8. De Luis DA, Sagrado MG, Conde R, Aller R, Izaola O. The effects of two different hypocaloric diets on glucagon-like peptide 1 in obese adults, relation with insulin response after weight loss. J Diabetes Complications 2009;23(4):239-243. https://doi.org/10.1016/j.jdiacomp.2007.12.006        [ Links ]

9. Luscombe ND, Clifton PM, Noakes M, Farnsworth E, Wittert G. Effect of a high-protein, energy- restricted diet on weight loss and energy expenditure after weight stabilization in hyperinsulinemic subjects. Int J Obes Relat Metab Disord 2003;27(5):582-590. https://doi.org/10.1038/sj.ijo.0802270        [ Links ]

10. Senn SJ. Overstating the evidence - double counting in meta-analysis and related problems. BMC Med Res Methodol 2009;9(1):10. https://doi.org/10.1186/1471-2288-9-10        [ Links ]

11. Wycherley TP, Brinkworth GD, Clifton PM, Noakes M. Comparison of the effects of 52 weeks' weight loss with either a high-protein or high-carbohydrate diet on body composition and cardiometabolic risk factors in overweight and obese males. Nutr Diabetes 2012;2(8):e40. https://doi.org/10.1038/nutd.2012.11        [ Links ]

12. Krauss RM, Blanche PJ, Rawlings RS, Fernstrom HS, Williams PT. Separate effects of reduced carbohydrate intake and weight loss on atherogenic dyslipidemia. Am J Clin Nutr 2006;83(5):1025-1031 (quiz 1205).         [ Links ]

13. Farnsworth E, Luscombe ND, Noakes M, Wittert G, Argyiou E, Clifton PM. Effect of a high-protein, energy-restricted diet on body composition, glycemic control, and lipid concentrations in overweight and obese hyperinsulinemic men and women. Am J Clin Nutr 2003;78(1):31-39.         [ Links ]

14. Frisch S, Zittermann A, Berthold HK, et al. A randomized controlled trial on the efficacy of carbohydrate-reduced or fat-reduced diets in patients attending a telemedically guided weight loss program. Cardiovasc Diabetol 2009;8:36. https://doi.org/10.1186/1475-2840-8-36        [ Links ]

15. Klemsdal TO, Holme I, Nerland H, Pedersen TR, Tonstad S. Effects of a low glycemic load diet versus a low-fat diet in subjects with and without the metabolic syndrome. Nutr Metab Cardiovasc Dis 2010;20(3):195-201. https://doi.org/10.1016/j.numecd.2009.03.010        [ Links ]

16. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. http://handbook.cochrane.org (accessed 27 March 2017).         [ Links ]