COMMENTARY

 

We object to Bad Science: Poor research practices should be discouraged!

 

 

Palesa N. Mothapo^I; Ethel E. Phiri^II; Tando L. Maduna^III; Rhoda Malgas^IV; Rose Richards^V; Taime T. Sylvester^VI; Mlungele Nsikani^VII; Melissa K. Boonzaaier-Davids^VIII; Moleseng C. Moshobane^IX

^IDivision for Research Development, Stellenbosch University, Stellenbosch, South Africa
^IIDepartment of Agronomy, Stellenbosch University, Stellenbosch, South Africa
^IIIDepartment of Physiology, Stellenbosch University, Stellenbosch, South Africa
^IVDepartment of Conservation Ecology and Entomology, Stellenbosch University, Stellenbosch, South Africa
^VWriting Laboratory, Stellenbosch University, Stellenbosch, South Africa
^VIDST-NRF Centre of Excellence for Biomedical Tuberculosis Research, South African Medical Research Council Centre for Tuberculosis Research, Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
^VIICentre for Invasion Biology, Department of Botany and Zoology, Stellenbosch University, Stellenbosch, South Africa
^VIIIResearch and Exhibitions Department, Iziko Museums of South Africa, Cape Town, South Africa
^IXSouth African National Biodiversity Institute, Biological Invasions Directorate, Pretoria National Botanical Gardens, Pretoria, South Africa

Correspondence

 

 

---

Keywords: peer review, racism in research, research integrity, sample selection bias, science capital

---

 

 

On 8 June 2020, we, a diverse group of African emerging researchers, published a response to the commentary titled 'Why are black South African students less likely to consider studying biological sciences?' (1) published in the South Africa Journal of Science (SAJS). There are mounting arguments, in both print and social media, regarding the merits of the Nattrass (2020) commentary, particularly around its strong racial undertones as well as poor and unethical research practices. Nattrass' commentary has been intensely divisive, managing to engender stereotypes, anger, and disappointment. Conflicting arguments have emerged, which involve responses by other academics, politicians, and the public, but much of the furore has been strongly biased towards and along racial lines, with very little attention directed at the flawed nature of the research. Such questions as the one asked by Nattrass (1) in the title of the commentary are valid and should be explored. Such research, in fact any research, must involve scientific rigour, robust methodological approaches, sensitivity and adherence to ethical principles. With the right approach and the involvement of multi-sector collaborators, we can begin to innovatively and constructively address the potential societal challenges that may arise. Science should be respected and trusted, and should build a fundamental basis for societal benefits and decision-making processes. Issues of race, whether socially constructed or not (2) are sensitive, and should be treated as such. When dealing with sensitive subjects, it is important to be cognizant of one's inherent unconscious biases. To drive this, scientists, editors, leaders in academia and industry, government research institutions, NGOs and publishers have taken steps to promote ethical conduct in research by signing The Singapore Statement on Research Integrity (3). The Statement was founded on four principles: viz. honesty, accountability, professionalism, and stewardship, which inform the fourteen responsibilities of ethical research conduct. As researchers, it is important that we use these principles and responsibilities to guide our research, and to maintain our responsibilities to each other, to the people and to the environment. For this reason, we wish to express our concern that the Nattrass (2020) commentary and the research contained therein, has violated many of these governing principles. Moreover, in publishing this commentary with all its methodological flaws and ethical problems, the South African Journal of Science (SAJS) has also violated these principles and responsibilities.

 

1. Critical assessment of the research methodology employed

In our first response to this commentary, we questioned why this poorly conceived study was allowed to be published in SAJS. When exploring the methodology employed by Nattrass (2020), the Singapore principles have been considered. Accordingly, the commentary may have overlooked 3 fundamental professional responsibilities assigned to all researchers, which serve to enforce the use of integrity in research methodology. Regarding 'Research Methods (Responsibility #3): Researchers should employ appropriate research methods, base conclusions on critical analysis of the evidence and report findings and interpretations fully and objectively.'

The main finding that significantly fewer black students than 'other' students considered studying the biological sciences (Table 1, Nattrass 2020) is flawed, which is alarming as it forms the basis of the commentary, and may account for the interpretation of the regression models presented. Instead, closer examination of the actual data shows that when 'Other' students are placed in their respective individual races/categories, the percentage of black students could possibly be higher than each 'Other' individual race, i.e. 32.4% becomes higher than half (e.g. half white, half coloured) of 49.5% and becomes even higher when 'Other' students are divided into more of their respective individual categories. Thus, one is left to wonder about the extent and impact of variability that accounts for the data obtained from the 'Other students' groups. Indeed, the statistical analysis raises some questions as the author further affirms that 'there was a lot of variation that was left unexplored by the (regression) models'. Furthermore, the minimal nature of the data and biased sampling present a great limitation of making regressions difficult to fully explore even for the purpose of reporting preliminary findings.

It has long been discussed and demonstrated how targeted sampling is prone to determining the demographic selection and response behaviour of that select sample, leading to biased and discriminatory conclusions that might be drawn and ascribed to a population (4). The author affirms this by stating that this 'opportunistic survey... resulted in an over-sampling of black South Africans', and 'the results for the total sample are thus in no way 'representative' of UCT students.', which would in no way be representative of black South African students. This speaks to the issue of unconscious bias, which is discussed further in this rebuttal, and also highlights the sample selection bias that tends to steer data collection and interpretation towards false conclusions about different races, as previously shown (Ards et al. 1998). Various studies have demonstrated the impact of sample selection bias on the attribution of racial stereotypes and profiling, and increases the risk of assigning predetermined conclusions onto the wider population. Notably, psychology research has identified the existence of the misattribution of cultural belief as the basis of decision-making and behaviour in minorities (5). In this study, the problem of oversampling minorities whereby they form the largest percentage of participants was identified as one of the major drivers used as a tool to draw conclusions that were more favourable to the researcher/psychologist. This achieved the goal of overemphasising the role of beliefs, ethnicity and race on behaviours of minority groups who tend to be considered to be 'exotic' and more 'cultural' in their behaviour and views, whilst the behaviour of whites would be predominantly driven by personality in contrast. According to Causadias and colleagues (5) oversampling of the race of interest and preconceived bias could serve to dehumanise minorities by 'denying their individuality'. In order to truly understand the research question, the Nattrass study should have focused on inclusivity, that is, sample size across different ethnicities and career choices. The flawed study design fails to account for a number of other factors that may determine career choice for South Africans and first-generation students in general. Furthermore, the data collection methods were not robust and the narrative excluded practitioners in the biological and conservation sciences, educational psychologists, higher education, or social sciences (socio-anthropologists).

Furthermore, when conducting questionnaires and surveys, researchers have to present the test results for construct reliability and validity (6). These measures are there to ensure that the questions that are asked are assessing the concept that is tested, and that they are consistent in doing so. The reliability measurement is a measure of quality and accuracy, telling us that the questions were designed to give a clear view of the concept tested, measured with Cronbach's alpha test of internal consistency. The internal validity is a measure that indicates that the same results would be achieved and would be consistent when the instrument of measure is repeated under the same situations (6). There was no indication that this was done with the instrument designed by Nattrass and from the results and the mismatch between the conclusion and the questions asked, it is clear that there was no reliability and validity of measure in this case. Therefore, Prof Nattrass could possibly extrapolate these findings as broadly as observed (1).

With regard to 'Research Records (Responsibility #4): Researchers should keep clear, accurate records of all research in ways that will allow verification and replication of their work by others.'

The commentary published by Nattrass (2020) reports on preliminary findings with the prospect of contributing to the knowledge base with regard to the attitudes expressed by a particular race of students, generalised within the South African context (1). According to the author guidelines of the SAJC, submitting these findings as a commentary would exempt the author from peer-review. To the reader, this suggests that the author may have been seeking to hinder and avoid critical examination of their research as well as to prevent open and objective discourse about the validity of the findings with the wider research community. This strategy would then serve as a tool for achieving early publication of their findings. Indeed this has been reiterated by the author within public platforms where it is admitted that the findings had garnered some excitement with colleagues suggesting rapid release of the findings as reported in media releases.

General consensus reiterates that a commentary is submitted following the publication of a research article or presented as a short and concise review or opinion piece of published methods, policies, etc. (7), with further emphasis that a commentary serves as an 'extended note that sets forth an expert's take on the meaning of a study' (8). Thus, commentaries are expected to promote, while providing critical assessment of, published work (7) which has undergone peer-review possibly explaining (but not excusing) the exemption of the peer-review process with regard to the commentary. Therefore, by circumventing the peer-review process Nattrass (2020) was able to avoid open and objective discourse as the statements made and the research design would have been questioned, reviewed and objectively assessed.

 

2. Flaws with the peer-review process

The peer review process is a very important step in quality assurance of research findings, and ensuring that strong research standards and integrity are adhered to by researchers, and that information that is published is valid, true and adds value. According to Da Silva and Dobranski (9), there is a high level of gatekeeping by journals. Often times, only the Editor in Chief and /or Assistant Editors are involved -- a manuscript can either be accepted or rejected on the spot, without peer review, or can be sent-out to be scrutinised by one's peers (full peer review). Publishing the commentary suggests that anything can be published where an outlet presents itself, further threatening the validity, quality and trust of science and the scientific method. With the recent increase in research article retractions from top end journals (e.g. Mandeep et al., 2020), it has emerged that the major issues that resulted in the retraction are mainly unethical research practices, lack of repeatability and significant methodological flaws. This is evidence that all research must still be subjected to scientific and public scrutiny even after publication.

In our initial attempt to submit a rebuttal to this commentary, we were refused the opportunity by the editor, which led us to self-publish in social media. We experienced first-hand the gatekeeping that can happen. Further, in her comments in response to the Black Academic Caucus, Prof Nattrass (the author of the commentary in question) indicated that the commentary was not peer reviewed, which was confirmed by the editor of SAJS. This is shocking. Such non-scientific practices purporting scientific outcomes lay a foundation for distorting or corrupting the disciplines of science and their research methodologies. This calls for thorough review of different papers before publication, be it commentary, full research and reviews. Allowing this commentary to be published with its many flaws in the methodology and analyses, especially with content that is bound to be controversial, is deeply concerning. One would think that due to the nature of the 'study', the scientific process behind it and the analyses would be even more closely scrutinised before publication.

Editors are the point of decision making regarding whether a manuscript should be published or not. The commentary should have been stopped immediately at this point had the editor done the job. It is a general understanding that commentaries, research notes and perspectives are also peer-reviewed, and it is worrying to know that SAJS does not adhere to some of these international standards. There is no indication that ethical clearance was provided in the document that was published. Submissions involving research conducted on human or non-human vertebrates must meet the highest standards regarding both the ethical consideration given and reporting of the procedures followed. Full details are necessary so that a non-specialist reader can appreciate the need for the research undertaken. All reported research involving humans or other animals must be approved prior to commencement of the study by an institutional ethics committee. On publication, the specific ethics approval number must be provided.

Publishing bad science only serves to diminish public and peer trust and question the integrity of researchers.

 

3. Unconscious bias and its influence on research practices

Unconscious or implicit bias can be defined as existing the unconscious beliefs and prejudices that are ingrained in people, and influence the way that they perceive the world around them, as well as their behaviours (10). In academia, unconscious bias can be reflected in academic HR processes where black researcher progression is dependent on a standard designed to keep them low and unrecognised (11), maintaining the low status of Black academic participation in STEMI (13). Unconscious bias can influence the way we conduct research, driving specific narratives because of our negative perceptions. We can ask questions that drive our negative mindsets about specific topics, and how we perceive the world to be. This goes against the scientific method, which advocates against bias. In conducting this research, publishing it, as well as engaging with it in the public, Prof Nattrass has exposed her own bias. As a respected Professor and academic, the public and the scientific community have trust in her word and in her work. She did not find a problem with her research and proceeded to defend even when the issues were clearly highlighted to here-narrowing this only to her freedom of speech and opinion. To us, this is a clear indication that she is not even aware of her own biases and is blindsided by narrow-minded views of black people.

The manner in which she approached the data gathering process has shown that she was driven by a specific mindset and sought to find answers that strengthened her view on a clearly complex problem. She has allowed her own pre-conception and ideas to drive the way she structured her questions and the conclusions she has made. The use of language was also insensitive and crass, said with a lack of care and empathy, which has resulted in her sounding very offensive to the reader. Had the author kept her own biases in check she would have conducted the research in a responsible way, she would not need to defend her stance in the way that she did through her responses in the mediafurther perpetuating the stereotype and painting those who question her as angry and racist.

 

Issues with purposive/convenient sampling

In her method, Nattrass uses the convenient or purposive sampling approach. While the method itself can be effective, it requires a high level of neutrality. The researcher relies on chance, but can also direct the sampling to areas where enough willing participants can be found. Nattrass sampled students during a break and did not consider whether these students studied in science, but instead shows that she placed her own assumptions first when she decided to continue with the study in the current format. As a result, the outcomes of the study were highly flawed. In her research findings, Nattrass compares black students against 'other races', without providing the reader a basis for understanding who or what the other races were. Her current conclusions based on this highlight more of her bias, in that the 'other races' likely showed more variability than the category Black that she refers to, as a result, she cannot draw up any conclusion.

Unconscious bias has also been cited as the main driver around gatekeeping in providing access to and career progression in STEMI careers for black students and academics. For the latter, issues around skills and ability have been highlighted as the main issues that prevent career progression for black students, women and men in STEMI. The idea that black people could excel in these areas seems to be more challenging, and forces the bar to be raised far higher for black students and academics when compared to white people (McCoy et al. 2017). Therefore, not only does unconscious bias affect the ability to do proper research, it affects the position of black people in STEMI (Brown et al. 2016, and the stereotyping with subsequent research like this study, just serves to perpetuate the already negative image of black people in STEMI.

 

Lack of Science Capital, mentorship and support limit black students success in STEMI

According to Professor Louise Archer of University College London, Institute of Education 'The more science capital you have, the more you see yourself as a science person - and the more likely you are to aspire to do that as a career'. Science capital refers to the extent of science-based information a child experiences, is involved and exposed to in their family, schools and general attitude around it. Black students in Universities in South Africa often are the first generation university students. They started university without knowing special researchers, either than the common known scientists they learn about in books. They do not see their black peers progressing in their careers after acquiring the degrees and PhDs in STEMI. They do not equate these to success if people are not employed or starting companies, or being recognised for high achievements. In 2020, we still see the challenges of the racial divides in STEMI.

In a recent blog post, a University of Cape Town Masters student in conservation ecology detailed her experiences in this field. Apart from the expenses associated with studying here, which naturally would exclude poor black students, she has highlighted the high lack of sense of belonging. In the field, she had no social or science capital because she was a black woman and she also had to live through the negative perceptions shown by her educators towards the surrounding communities where she worked. These individuals who refused to develop programmes for young children who live in the surrounding areas where the protected and conservation areas were built, simply because they could rob them and take their valuable goods. These unconscious biases are more divisive and prevent real action and change from taking place, and limits the opportunities of young black students. Further, from this blog post, it was clear that even among black people, there is a negative perception about those who study biology related degrees. Them, seeming more white and betraying their blackness, or being classist since they are able to afford the expenses associated with studying the degree. Without the visibility of STEMI professionals in black communities, the challenges of increasing access and opportunities will remain high. Creating the kind of interest needed in the field, requires that black STEMI professionals are recognised and visible to those in their communities, and those who arrive at these institutions of higher learning (11). Black students cannot be encouraged if they see that Black researchers are not progressing; black PhDs are unemployed!

There is a currently prevalent stereotyping that black students do not study the biological sciences, or STEMI in general. As shown by studies in the USA, these stereotypes have a negative impact on academic performance and career choices in STEMI (11). This is known as stereotype threat, where 'the fear of doing something that would inadvertently confirm a stereotype' (12) causes black students to under-perform at all levels (11). Furthermore, in South Africa, we cannot focus on these factors or put significant pressure on a racial group that were not allowed access to these fields in the past. In fact, a majority of researchers in the biological sciences are predominantly male and white, and even though participation by females has increased, largely it is white females. Clearly, there is no science capital. Furthermore, there is a prevalence of clique formation in universities and it is easy for black students to have a poor sense of belonging in these groups. There could also be differences in the experience of black students in STEMI in historically black universities versus historically white universities. None of these contexts could have been extrapolated from the results that this commentary has drawn conclusions from.

 

Practice responsible science: The divided responses of scientists, politicians and the public who defend the commentary and Prof Nattrass

• We need open conversations and debates that are based on logic and facts

• There is a large focus on race, but more importantly the discussion should be how we can encourage more participation in STEMI for the racial groups that were not allowed access to these fields in the past

 

References

1. Nicoli N. Why are black South African students less likely to consider studying biological sciences? South African Journal of Science. 2020;116(5/6). https://doi.org/10.17159/sajs.2020/7864        [ Links ]

2. Bond MA. Diversity in Community: Rethinking Psychological Perspectives on Bridging Differences. In: Jansen B, editor. Rethinking Community through Transdisciplinary Research. Cham: Springer International Publishing; 2020. p. 69-83. https://doi.org/10.1007/978-3-030-31073-8_4        [ Links ]

3. Resnik DB, Shamoo AE. The singapore statement on research integrity. Account Res. 2011;18(2):71-5. https://doi.org/10.1080/08989621.2011.557296        [ Links ]

4. Cheung KL, Ten Klooster PM, Smit C, de Vries H, Pieterse ME. The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC public health. 2017; 17(1):276. https://doi.org/10.1186/s12889-017-4189-8        [ Links ]

5. Causadias JM, Vitriol JA, Atkin AL. Do We Overemphasize the Role of Culture in the Behavior of Racial/Ethnic Minorities? Evidence of a Cultural (Mis)Attribution Bias in American Psychology. American Psychologist. 2018;73(3):243-55. https://doi.org/10.1037/amp0000099        [ Links ]

6. Glen S. Reliability and Validity in Research: Definitions, Examples. StatisticsHowTocom: Elementary Statistics for the rest of us! 2020.         [ Links ]

7. Berterö C. Guidelines for writing a commentary. Int J Qual Stud Health Well-being. 2016;11:31390. https://doi.org/10.3402/qhw.v11.31390        [ Links ]

8. Jull A. How to write a commentary-an editor's perspective. Evidence Based Nursing. 2007;10(4):100-3. https://doi.org/10.1136/ebn.10.4.100        [ Links ]

9. Mandeep, R.M, Frank Ruschitzka, F., Patel, A.N. (2020) Retraction - Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. The Lancet, 395 (10240), 1820. https://doi.org/10.1016/S0140-6736(20)31324-6        [ Links ]

10.Dortch, D. and Patel, C. (2017). Black Undergraduate Women and Their Sense of Belonging in STEMI at Predominantly White Institutions, NASPA Journal About Women in higher Education, 10(2): 202-215. https://doi.org/.1080/19407882.2017.1331854        [ Links ]

11.Coleman, A., Acquaye, A., Cardona, X., Ingram, K., Kleczewski,T., Sawyers, E.S. (2018) STEMI is for ALL: Diverse Perspectives on Black and Latino STEMI Motivation' Publications & Research. 35. https://digitalcommons.imsa.edu/pres_pr/35        [ Links ]

12.McGee, E.O. and Martin, D.B. (2011). ''You Would Not Believe What I Have to Go Through to Prove My Intellectual Value!'' Stereotype Management Among Academically Successful Black Mathematics and Engineering Students. American Educational Research Journal, 48(6); 1347-1389. https://doi.org/10.3102/0002831211423972        [ Links ]

13.Brown, B.A., Hendersen, B.J., Gray, S., Donovan, B., Sullivan, S., Patterson, A. and Waggstaff, W. (2016). From Description to Explanation: An Empirical Exploration of the African-American Pipeline Problem in STEM. Journal of Research in Science Teaching, 53(10):146-177. https://doi.org/10.1002/tea.21249        [ Links ]

 

 

Correspondence:
Palesa Mothapo
Email: mothapo@sun.ac.za

Published: 10 July 2020