COMMENTARY

 

The Anatomy of a Bad Science: Reflections on Nattrass' 'commentary'

 

 

Jimi O. Adesina

College of Graduate Studies, University of South Africa, Pretoria, South Africa

Correspondence

Original Commentary: Why are black South African students less likely to consider studying biological sciences? – Nicoli Nattrass

 

---

Keywords: bad science, prejudice, methodology, quantitative analysis, ontological disconnect, transformation

---

 

 

In its May/June 2020 issue, the South African Journal of Science (S Afr J Sci) published a two-page 'commentary' by Nicoli Nattrass, a professor of economics and co-Director of the Institute for Communities and Wildlife in Africa at the University of Cape Town (UCT). The 'commentary' is titled 'Why are black South African students less likely to consider studying biological sciences?'¹ The piece reported 'the findings' of a survey of students at the University of Cape Town that was conducted in mid-2019. This was supposed to be an exploratory study, intended to offer insights into why Black South African students do not study biological sciences. In this article, I explore three dimensions of the research and the results reported in Nattrass' commentary.

First, I assess the methodological framing of the 'exploratory' survey and argue that the study design is gravely deficient. Second, I examine the descriptive and inferential statistics that Nattrass reports. I argue that there is significant dissonance between the results of the regression models reported and the conclusions that Nattrass draws. Third, I examine a set of highly problematic claims that shape the survey and offer insights into the presuppositions that explain the conclusions Nattrass draws from the study. These presuppositions involve a set of racially charged tropes about 'black South African' students. The summative conclusion is that what we have is bad science hanging on the horns of prejudice.

 

Methodological problems in the study design

The first issue of concern in engaging with the commentary concerns the study design. The study participants were recruited 'by approaching students during the lunch break.'¹ Nattrass referred to this as 'opportunistic survey.'¹ The choice of sampling technique is puzzling, instead of more robust probability sampling methods. The claim that the study is 'exploratory' is not credible. An exploratory study seeks to address, in tentative ways, issues that might not have been previously studied. Using the students at UCT may be permissible for such study-as a prelude to a more national survey-but this would not justify the use of non-probability sampling. The reason for this is simple. The study seeks to answer a 'why' question not a 'how' question. Even in its 'exploratory' nature, the (tentative, preliminary) answers that the researchers sought involve making claims beyond the study sample. Generalisability is at the heart of the question the researcher sought to answer.

The choice of non-probability sampling is all the more puzzling because, as staff of the university, the researchers had access to a reliable sampling frame from which they could develop the probability sampling. The sampling frame would involve a complete record of all the students at the university. As researchers at the university, they could easily access such sampling frame, from the Office of the Registrar. The frame would have provided the relevant socio-demographic distribution of the student population, their degree options, and other characteristics needed for generating a credible sample.

With the sampling frame, the researchers could have employed an appropriate stratified, random sampling procedure. Race-categories, fields of study (as a proxy for career trajectory), and socio-economic status of the study participants are all critical for the research questions that underpin this study. The sampling frame would have provided the basis for a robust stratification of the population from which they can draw the study sample. Further, the researchers would have indicated, beforehand, the margin of error used in arriving at the sample size. The error margin would have helped in interpreting even the descriptive statistics since we would know the confidence interval for the reported percentile distribution of the sample.

It is unclear if a scientific committee ever considered the study proposal at the university. If this was the case, the study should have been flagged. A research ethics committee would be concerned with issues of whether the research process could open the students to harm. Still, an approval that the study is scientifically robust should have come before the ethics clearance application.

The effects of the methodological deficiencies highlighted above are that the researchers stumbled blindly into the field. While 'black South African' students'(BSA) share of the total population of UCT students was 30 per cent, they are 54 per cent of the study sample. The sample size derived with convenience sampling was 211 students. Properly designed, with a 3 per cent margin of error, we would have expected a sample size of at least 1 030 students. The use of the sampling frame would have offered the researcher access to the email addresses of the potential respondents. They could have used this to invite them to complete an online questionnaire rather than the face-to-face interviews approach they employed. With this, they could have avoided methodological problems with the race or gender of the enumerators, for instance, that are widely known in social research.

Even for a study based on non-probability sampling, there is a curious homogenisation of the BSA students at UCT-that they are all from impoverished backgrounds. A background claim to the study is that 'obviously... persisting inequalities in the school system make it less likely that they [BSA students] will meet the entrance requirements for science courses.' This clearly shows a shocking lack of appreciation for the diversity of the UCT BSA students and their school backgrounds.

The issues raised above immediately undermine the author's capacity to make any credible inferences about the study population, much less the study universe. Similarly, the study cannot make any inferences about BSA students at UCT (beyond those in the study sample), much less BSA students in the country. The title and conclusion of the 'commentary' make claims that cannot be supported by the study design-even the ones that supposedly repudiate race as a critical variable with predictive power (see further discussion below).

A study on why BSA students 'are less likely' to consider studying biological sciences is not the same as 'no BSA student' considered studying biological sciences. If the population of BSA students in UCT is 30 per cent but the share of BSA among those studying Biological Sciences was 35 per cent, the share of BSA students studying Biological Sciences would still be low relative to the other categories of students but would be higher than their overall share of the university student population. The author makes no effort to engage with this scenario.

For a study concerned with explaining why BSA students are less likely to study the Biological Sciences, a researcher would want to consider a sample stratified, at least, along the lines of those studying the Natural Sciences and those who are not. Since an organising assumption is the economic status of the students at the university, one would also have expected a sample stratified by such status. Why would 'materialist values' enter the equation, and be a hypothesised reason for not choosing a career in conservation biology other than they choose well-paying professions because 'they are trying to escape poverty.' Financial aid could have been used as a proxy (a dummy variable) for the students' economic status.

Further, what share of the surveyed BSA students (114 out of 211) is in the Humanities or the Natural Sciences, for instance, relative to the category 'Other students' would matter for the question that the author claims she sought to answer? Would a student's degree focus have some bearing on what they think of a question such as whether humans evolved from apes? Suppose a large share of the BSA students in the sample is registered in Theology, and such students consider the idea that humans evolved from apes absurd. Would that reflect their race classification or their disciplinary orientation? Would a BSA student studying Medicine or Zoology hold the same position?

For a sample that the author admits has a higher share of BSA students than the population, it is interesting that there is no attempt to allocate weight to the sub-categories of the sample (BSA students and 'Other students') when reporting the findings. Even in the most rigorously designed probability sampling, sub-categories of the sample may be over-represented due to high non-response rates from other sub-categories of the sample. Researchers need not be held responsible for the non-response rate. What would be expected, however, is that the sample is properly weighted. In this case, the weight for BSA students would be 0.555 (30.0/54.0)-30 per cent being their share in the student population and 54 per cent, their share of the sample. Correspondingly, the weight for 'Other Students' would be 1.521. Even for the descriptive statistics, such weights matter.

 

Dissonance: The conclusion misreports the study result

While a poorly designed study may reflect limited methodological proficiency on the part of the researchers, misrepresenting the results of the survey is less easily dismissed. Nattrass acknowledges that the result of her 'exploratory regressions' is that attitudes are more important as predictors of the decision to study biological sciences, rather than 'the crude indicator of being a black South African' (cf. Table 2 and p.12 of the 'commentary'). Yet, Nattrass concludes with the exact opposite of this. The regression model shows, very dearly, that the predictive power BSA (a race-category) diminished as the attitudinal variables entered the regression model. By the time the fourth attitudinal variable was introduced into the model, the predictive power of BSA race-category had declined from a minus 17 per cent to a minus zero per cent. When the race category variable was dropped entirely from the model, the attitudinal variables retained their predictive power: no change was registered in their coefficients.

The pressing question to ask the author is this: if being a BSA student is less or not a predictor of whether one considered studying (zoology or) biological sciences, what is the purpose of the title of the paper? Given the low or no predictive power of race-category in the model, 'Are black South African students likely to consider studying biological sciences?' would have been a more appropriate title. In such a case, the answer would then have been in the negative: a definite 'No'.

The same diminished predictive power of the race-category of 'black South Africans' is evident in the regression model reported in Table 3 of the commentary: to explain the probability of supporting 'wildlife conservation but hav[ing] no interest in pursuing a career in it.' By the time the three attitudinal variables (including the so-called 'anti-conservation index') were added to the regression, the predictive power of the BSA race-category declined from 16 per cent to 3 per cent; even the 16 per cent was only significant at 5 percent probability. Interestingly, the regression model suggests that if you like having the starlings around at UCT, you are 28 per cent less likely to say that you support 'wildlife conservation but have no interest in pursuing a career in it.' Either way, attitudinal dispositions not race-category rule the day!

Race-based explanations by any means

Even with the low predictive power of race-category in the model, lurking under the cover are race-based explanations. This plays out in two ways. The first concerns the pattern of attitudinal disposition ascribed to the BSA variable. The variable may not be a good predictor, but the attitudes that are the predictors are racially ascribed. The black South African students in the sample are presented as less likely to agree that humans evolved from apes than the 'Other students.' The BSA students are less likely to like having the redwing 'starlings around at UCT.' They are more likely to agree to the statement that 'I support wildlife conservation but have no interest in having a career in it.' All these are statistically significant in the Fisher's Exact Test results that the author presents. Race explanation remains; only that they are disguised as attitudes.

Despite the diminished (or non-existing) explanatory power of race-category in the model, which Nattrass conceded, nonetheless, she insists that:

In short, the survey results suggest that black South African students are less likely to consider studying biological sciences than other students, and that this stance was linked primarily with career aspirations... and these were associated with materialist values and attitudes to local wildlife (p. 13).

No, they do not. Such 'result' is a thumb-suck. She suggests that BSA students are more likely to opt for degrees in accountancy and law because these are better paying. This is intended to underpin the claim of a materialist disposition among black South Africans. No evidence exists in the study to support this claim; no authority offered, and the study offers no such insight.

The correct, and ethically sound, interpretation of the regression statistics that Nattrass presents in Tables 2 and 3 is that regardless of the race categories of the respondents, the attitudinal variables are better predictors of whether a student considered studying conservation biology. But even with the descriptive statistics presented in Table 1, Nattrass ought to have known that Fisher's Exact Test results are for descriptive statistics. The results of the regression model that render the author's conclusions absurd are inferential statistics. Descriptive statistics merely describe the distribution of the sample (within a given confidence interval). Predictions and inferences are better, based on inferential statistics.

For a non-probability survey, Nattrass goes ahead to make generalisations that ignore the non-probability design of the study. A simple rule of quantitative research is this: never make claims about your research that it was never designed to carry. Nattrass' 'commentary' breached these simple rules of sample survey studies. She generalised from the sample not only to the study population (about BSA students at UCT) but to the study universe (about BSAs in South Africa beyond UCT).

Even so, these are all moot points. A poorly designed study will, in all probability, produce bad data. No amount of regression or other inferential statistics can fix that foundational problem. The aphorism, in statistical analysis, of 'garbage in, garbage out' would apply.

There are more minor issues to raise with the study. First, the study reduced 'biological sciences' to conservation biology. 'Biological sciences is the study of life and living organisms, their life cycles, adaptations and environment. There are many different areas of study under the umbrella of biological sciences, including biochemistry, microbiology and evolutionary biology.' ² How many BSA students at the university and other universities in South Africa are studying other biological sciences, other than conservation biology and wildlife?

The second issue concerns data aggregation. From the reporting of the regressions for the composite index used in Table 3 of the 'commentary', one gets the impression that the questions in the survey instrument were in the form of a Likert scale. However, the reporting on Table 1 involves a binary or categorical 'Agree' or 'Disagree' rather than the fuzzy set disposition of a Likert scale. Otherwise, it is difficult to see why Nattrass reports Fisher's Exact Test results. This is not a pedantic concern since what it does is to elide the more nuanced differences between someone who indicates a preference for 'disagree' as against 'agree', compared to others who expressed a preference for 'strongly disagree' against those who chose 'strongly agree.' The 'distance' between the former respondents is much less than the distance between the latter respondents. That precisely is the value of a Likert scale. Further, what happened to the respondents who selected 'Neutral' as their preferred answer? Keeping the data in the original Likert scale form in which it was collected would not prevent getting a robust descriptive statistics result; that is what Exact Tests modules are intended to do, beyond the Fisher's Exact Test. The Exact Tests modules are available in the major statistical packages: SAS, Stata or SPSS.

 

Presuppositions, Prejudice, and Ontological Disconnect

A scientific study cannot (and should not) be rejected purely on the ground that the result offends a segment of the population, even one with a population share of 80 per cent. A predictable response to such rejection (on the ground that it offends) is 'Don't shoot the messenger.' But what if the messenger is the message? A study may be rejected based on the prejudiced presuppositions that underpin it, especially the deployment of racially charged tropes. As Chinua Achebe notes concerning Joseph Conrad's The Heart of Darkness, 'travellers with closed minds can tell us little except about themselves.'³ The Nattrass paper is shot through with presuppositions that are products of prejudice rather than science.

Beyond the disadvantage imposed by relatively weak schools, Nattrass suggests that BSA students' choice of degree subjects is 'likely to be [for] other reasons too, notably materialist values and aspirations.' This underlining assumption is made without any evidence, and no authority cited. The presupposition-something previously enunciated by Nattrass⁴ -is that 'crass materialism' characterises black South Africans in the post-apartheid South Africa. By Black South Africans, Nattrass means '(Black) Africans' in contemporary South African population categorisation or the 'Bantu' in the apartheid classification. It is worth noting that Nattrass homogenises all BSA students, as coming from a poor socio-economic background. Choosing to follow a profession in law or accountancy, Nattrass suggests, is indicative of such materialist disposition.

As of January 2019, there were 27 223 attorneys in South Africa. Fifty-six per cent are 'White attorneys', and 44 per cent are 'Black attorneys (African, Coloured and Indian).'⁵ That is against a national population share of 7.9 per cent White, and 92.1 per cent Blacks (Black African: 80.7, Coloured: 8.8 per cent, Indian/ Asian: 2.6 per cent).⁶ The distribution of Chartered Accountants in South Africa is even more skewed. As of May 2020, 46 841 Chartered Accountants were on the register of the South African Institute of Chartered Accountants. Of these, 68.63 per cent were White (32 151), and 31.37 per cent Black (14 306). There were 6 670 Black Africans CAs or 14.23 per cent of the total number of CAs in South Africa; 1 904 or 4.06 per cent were Coloured; and 5 732 or 12, 23 per cent were Indian/Asians.⁷

Given the distributions in the legal and accounting professions, how is the decision of a Black (African) student to study law or accountancy considered 'materialistic'? Would a White student who decides to study law or accountancy have been labelled 'materialistic'?

I once heard a dean at a Faculty Board meeting say that as far as transformation is concerned, he is 'a minimum compliance person.' Is labelling Black (African) students going into the legal and accountancy profession materialistic driven by similar disposition? Is this the subversion, by other means, of a country's effort to overcome the prevailing legacy of its racist past? Consider a hypothetical situation where ALL 'black South African' students take to heart Nattrass' subliminal injunction not to be 'materialistic'-by not going into law and accountancy-would that not render permanent the apartheid footprint on the professions? As the saying goes, there are many ways to skin the cat called transformation.

In reporting the survey results, Nattrass' analysis takes a curious turn. She constructs what she calls 'an anti-conservation index' or a Fallist index! (More about this below). The 'anti-conservation index' is a composite measure drawn from three questions. The questions include whether national parks should be scrapped, to whether disciplines like conservation biology are colonial and should be scrapped, and whether the respondents like having the redwing starlings on the UCT campus.

Nattrass links the 'materialist index' to the World Values Survey. She claims that the twelve questions used in the World Values Survey as composites for the materialist index were included in the UCT survey. The distribution for the variables was not presented in the table that reports the descriptive statistics. Most significantly, the materialist index drawn from the World Values Survey has nothing to do with whether a respondent was studying accountancy or law. There is no evidence that the survey included a question of whether a student was studying accountancy or law. The 'anti-conservation index' and the 'materialist index' are attitudinal measures. These attitudes are held by BSA and OS respondents in the study. Yet, in the conclusion, Nattrass reverts to a student's choice of professions (law or accountancy?). 'Materialist values' are presented as 'a key determinant of not desiring a career in conservation' (p.13).¹ The conclusions drawn concerning 'materialist values of the BSA students is neither consistent nor derived from the study; it would seem that they derive from the author's predisposition rather than science.

In the regression analysis on Table 3, the predictive power of the 'materialist index' is only 5 per cent (Regression 3.3. and 3.4); same as the 'anti-conservation index'. This is against a 28 per cent predictive power of the variable about a preference for having the redwing starlings around the UCT campus. The predictive power of the BSA race-category dropped from 16 per cent to 3 per cent. The question that follows is a simple one: Why emphasise the 'anti-conservation', 'materialist' values rather than whether one likes having redwing starlings on the campus? Why resort to the claim that the career aspiration of BSA students hinders their preference for studying biological sciences, when the attitudinal variables, regardless of race-categories, have higher predictive powers?

Concerning the index based on the World Values Survey, citing Inglehart (1990) Held et al. (2009: 57) distinguished materialists from post-materialists thus: 'materialists [are mostly concerned] with physiological needs and stress physical and economic security... Post-materialists, by contrast, strive for self-actualisation, stress the aesthetic and the intellectual, and cherish belonging and esteem.'⁸ How does taking up a career in conservation translate into being post-materialist?

Conservation is big business in South Africa.⁹ In 2015 alone, the value of wildlife hunting value chain was R10.1 billion.⁹ The hunting component of the value chain was valued at R5.1 billion. Trophy hunting of kudu alone was valued at R1.2 million, at R13 000 per head of the animal killed for trophy.⁹ In the same year, a lion was sold for R230 000; the average price of a buffalo was R334 841 - eighty-four buffaloes were sold that year.⁹ There are no halos waiting to be placed, ipso facto, on the heads of people in the industry. There are decent people concerned with protecting animals, habitats, and fauna. But there are those who operate the canned hunting business; some breed lions to be slaughtered. You could argue that those involved in the business are more concerned with economic security needs than 'self-actualisation.' What more, trophy hunting involves the needless slaughter of wildlife for the hunter's self-amusement.

What would have happened if Nattrass took into consideration the fact that her data shows that 89.4 per cent of the BSA students in the survey disagree with the statement 'that many of South Africa's national parks should be scrapped and the land given to the poor' (against 94.7 per cent of 'Other students')? Other than a problematic hook on which the author seeks to hang prejudice, it is difficult to see how not wanting to pursue a career in conservation translates into being 'materialist', and then proceeding to hang this on the neck of the BSA students. The issue of absent self-reflexivity raised earlier applies. Does Achebe's aphorism offer some insight?

As a penultimate issue in this section, let us return to the so-called Fallist index. Regressions 3.3 and 3.4 in the 'commentary' involve what the author refers to as the ''anti-conservation' (or 'Fallist')' index. Nattrass' claim that 'disciplines like conservation biology are colonial and should be scrapped from UCT' are Fallist positions or opinions. She offers no evidence that this is the case or that there is such an opinion that was issued by the 'Fallist movement.' Indeed, is there a Fallist opinion, in the singular, that 'conservation biology' is colonial and should be scrapped from UCT? If there is something about the Fallist movement, it is the absence of a central authority that would purport to speak for everyone involved in the protests. Indeed, you could argue that while the Rhodes Must Fall phase of the movement was driven by the poor record of transformation (epistemic and cultural Euro-centricism) of their respective universities, the prohibitively high university fees served as the driving force behind the Fees Must Fall phase. One phase highlighted epistemic and cultural barriers: the other the economic barrier.

The impression that emerges from the 'commentary' is of a researcher with a deep-seated antipathy towards the Fallist movement. Such aversion may be legitimate as points of difference. It is problematic when it corrodes the scientific endeavour. The troubling part is that Nattrass seeks to render as irrational important conversations that the South African education system (not just the higher education sector) needs to have and act upon. It is legitimate to object to some of the methods employed in the campaigns that defined the protest movements without demonising the demands or rendering the demands themselves irrational. The Fallist index would seem more a product of prejudice than a legitimate effort in pursuing a scientific inquiry.

Further, one suspects a second layer of presupposition in the author's argument (possibly in the research instrument, as well): the false belief that 'Africans don't do conservation'; that conservation is alien to Africa. The larger argument is not so much about conservation, per se, but the modality of conservation. If you dispossess people of their lands and sources of livelihood to create a wildlife reserve for (European and American) tourists, those dispossessed have a right to question your idea of conservation. Thandika Mkandawire once referred to the 'eco-fascism' of those who demand nature reserves at the expense of the welfare of African people.¹⁰ If you have no ontological link to such land dispossession, you would see the conservation area but not its origin and persisting consequences. This is a classic case of ontologicaldisconnect-a disconnection from, and a lack of empathy for, the bearers of a collective memory of dispossession and who inhabit its aftermath. It is legitimate to argue that mass extinction of biological species, wanton depletion of wildlife, trophy hunting, and canned hunting are as colonial as one can imagine, and a marker of racial capitalism.

 

In lieu of a conclusion

The discussion above has sought to raise three distinct issues with Nattrass' commentary. First is the methodological deficiencies in the study design. Second is the dissonance between the results of the regression models in the commentary and the conclusions that Nattrass drew. The third is the extent of the corrosive effects of the author's presuppositions and prejudice on the premise and reporting of the study.

 

Acknowledgement

I thank Dr Ashley Sarimana, Dr Kehinde Omotoso, and Ms Motolani Adesina for the comments and editorial inputs that I received from them.

 

References

1. Nattrass N. Why are black South African students less likely to consider studying biological sciences? S Afr J Sci. 2020; 116(5/6):12-13. https://doi.org/10.17159/sajs.2020/7864        [ Links ]

2. Times Higher Education. Top universities where you can study biological sciences. London: THE [2020 June 6]. Available from: https://www.timeshighereducation.com/student/what-to-study/biological-sciences        [ Links ]

3. Achebe C. An Image of Africa. Research in African Literatures, 1978;9(1):1-15. https://www.jstor.org/stable/3818468        [ Links ]

4. Seekings J, Nattrass N. Policy, Politics and Poverty in South Africa. Basingstoke: Palgrave Macmillan, 2015.         [ Links ]

5. Law Society of South Africa. Statistics for the Attorneys Profession. Pretoria: LSSA [2020 June 07]. Available from: https://www.lssa.org.za/about-us/about-the-attorneys-profession/statistics-for-the-attorneys-profession/. The share of candidate attorneys is shifting the distribution (White candidate attorneys: 37 per cent; Black candidate attorneys: 63 per cent), but there are only 7 000 candidate attorneys in the country as of January 2019 (ibid).         [ Links ]

6. Statistics South Africa. Mid-year population estimates. Statistical Release PO302. Pretoria: StatsSA; 2019.         [ Links ]

7. South African Institute of Chartered Accountants. Membership Statistics. Johannesburg: SAICA [2020 June 08]. Available from: https://www.saica.co.za/Portals/0/Members/Race2020.pdf        [ Links ]

8. Held M, Müller J, Deutsch F, Grzechnik E, Welzel C. Value Structure and Dimensions. Empirical Evidence from the German World Values Survey. World Values Research. 2009; 2(3):55-76.         [ Links ]

9. Department of Environmental Affairs. Biodiversity Economy (Wildlife) Preservation: Unlocking the socio-economic potential of South Africa's biodiversity assets through sustainable use of wildlife resources. Presentation made to the Parliament of South Africa, Cape Town. 2018 February 27. Available from: http://pmg-assets.s3-website-eu-west-1.amazonaws.com/180227DEA.pdf        [ Links ]

10. Kate Meagher. Reflections of an engaged economist: An Interview with Thandika Mkandawire. Development and Change, 2019; 50(2):511-541. https://doi.org/10.1111/dech.12481        [ Links ]

 

 

Correspondence:
Jimi Adesina
Email: adesij@unisa.ac.za

Published: 10 July 2020

 

 

Note: A longer and extended version of this article was published in the CODESRIA Bulletin Online, Number 1, June 2020 (https://www.codesria.org/spip.php?article3058). The opinions expressed in this article are entirely mine. They do not in any way implicate the University of South Africa, the SARChI Chair in Social Policy, or the National Research Foundation.