COMMENTARY

 

Assuring the quality of scholarly South African journals: An experiment in journal peer review

 

 

Robin Crewe^{I, II}

^ISenior Research Fellow: Centre for the Advancement of Scholarship, University of Pretoria, Pretoria, South Africa
^IIChair: Committee on Scholarly Publishing in South Africa, Academy of Science of South Africa (ASSAf), Pretoria, South Africa

Correspondence

 

 

---

Keywords: grouped peer review, funding formulae, journal databases

---

 

 

The introduction of the research output incentive scheme for public higher education institutions (HEIs) in the 1980s² was premised on two principles - the first was that rewarding outputs would direct resources to those institutions that were producing graduates at the master's and doctoral levels, and whose staff and students produced published scholarly work in recognised journals, conference proceedings and books. The incentive scheme was seen as a mechanism to encourage improvement in both the productivity and the quality of the research produced by the public HEIs in South Africa. With the introduction of the New Funding Framework for HEIs in the 2004/2005 financial year, the way in which institutions received funding was explicitly laid out³ and research was funded as a separate category with its components funded through awarding units with a particular monetary value.

The component of the formula for research funding that will be explored here relates to the recognition of scholarly publications in journals only. Scholarly publications in books and conference proceedings were assessed by a committee established by the Department of National Education (at that time) that decided on the subsidy units to be allocated to these categories of submission. Journal outputs were handled differently, with the Department indicating that only journals listed in the Institute for Scientific Information (ISI) databases and the International Bibliography of Social Sciences (IBSS) would be automatically recognised. South African journals that were not listed by the ISI or IBSS had to apply for inclusion in the Department of National Education's and then subsequently the Department of Higher Education and Training (DHET)'s list of accredited South African journals (LASAJ). The quality of journal outputs was assessed using the ISI and IBSS databases as a proxy for acceptable quality while South African journal titles that were not included in these lists had to apply to the Department for inclusion in the LASAJ.

Just prior to the change in the formula for the funding of universities, the Department of Science and Technology (DST) decided in 2001 that it would close the Bureau for Scientific Publications which had previously provided funding for a group of South African research journals, and it contracted the Academy of Science of South Africa (ASSAf) to review scholarly publishing in South Africa and to make recommendations to DST and DHET regarding this matter. The ASSAf report¹ that was delivered in 2006, and its fifth recommendation (shown above), proposed the idea of subjecting journals to peer review.

The key element of this recommendation was the systemic peer review and periodic audit of all South African journals and particularly those included in the LASAJ. At the time that this recommendation was made it was entirely novel in that it proposed the peer review of journals in a way that was analogous to the peer review undertaken by journals of the material submitted to them. It was the first time that a peer-review process was being proposed to assess the quality of material in, and editorial processes of, journals. The underlying thrust of the approach was to encourage accountability for the quality of content and also make developmental recommendations directed at improving the quality of the journals reviewed.

Following the acceptance of the ASSAf report, ASSAf undertook the task of implementing the recommendations of the report and the ASSAf Council established the Committee on Scholarly Publishing in South Africa (CSPiSA) which had oversight of ASSAf's Scholarly Publishing Programme (SPP). It was the staff of the SPP that were given the task of making journal peer review a reality.

 

Introduction of peer review of journals

An explicit call for the peer review of South African scholarly journals was made in Recommendation 5 of the ASSAf report¹ with the idea that this be mandated by the Department of Education and DST which were the relevant government departments at that time. Both departments endorsed the idea and indicated that the process could be used to manage South African journals on the LASAJ. This proposal was then discussed with the newly established National Scholarly Editors' Forum (NSEF) and the editors present supported the introduction of this process and were consulted regarding both the process and documentation that would be used. The outcome of this consultation was that guidelines for the peer review of journals were developed and introduced.⁴

There is a clear distinction between the process that ASSAf uses to review journals and makes recommendations regarding their inclusion in the LASAJ and/or their suitability for the SciELO South Africa platform⁵, from that used by other database services such as Web of Science (successor to ISI), IBSS, Scopus and others.

The currently DHET recognised databases outside of South Africa are the Norwegian List, Web of Science, IBSS, and Scopus. These four databases try to ensure that there is some form of quality assurance in relation to the items that are included. However, the form of quality assurance employed usually consists of a set of technical criteria that the journal needs to address in order to be listed. Individual journals may be delisted if they no longer adhere to the stipulated criteria. In addition, some of the companies maintaining these databases have initiated the use of bibliometric data to screen journals for acceptability. An example of the criteria that are used by Elsevier for the Scopus database are:

To be considered for review, all journal titles should meet all of these minimum criteria:

• Consist of peer-reviewed content and have a publicly available description of the peerreview process;

• Be published on a regular basis and have an International Standard Serial Number (ISSN) as registered with the ISSN International Centre;

• Have content that is relevant for and readable by an international audience, meaning: have references in Roman script and have English language abstracts and titles;

• Have a publicly available publication ethics and publication malpractice statement.

In addition to this list of minimum requirements, there is also a set of metrics that the journals have to meet in order to be included or remain on the list.⁶ In addition, Scopus has 'The Scopus Content Selection and Advisory Board (CSAB) [that] is an international group of scientists, researchers and librarians who represent the major scientific disciplines.' The way in which this CSAB operates and how it is constituted is not clearly spelt out.

The peer review of journals undertaken by ASSAf is different in nature, rigour and transparency from the procedures that appear to be used by the other journal databases. The key element of this difference is the process of peer review and the way that it is implemented which follows a standard methodology that has been refined by ASSAf over a number of years and provides the outcomes of its reviews in the form of publicly available reports.

 

ASSAf journal peer review process

The review of a group of journals is initiated by the CSPiSA, which then seeks a group of suitably qualified individuals to constitute a peer-review panel to undertake the review of the chosen set of journals. The CSPiSA solicits the nomination of individuals who are recognised scholars in the disciplinary fields of the journals to be reviewed. In addition, individuals with expertise in journal publishing as well as one individual who is from an unrelated field of study are included in the selection. Generally, 10-12 individuals are nominated and the panel is then constituted of 6-8 members who agree to participate, and whose membership is approved by the ASSAf Council.

The panel elects its own chair and manages, with support from staff of the SPP, the production of a report on the journals being reviewed. The process entails the identification and appointment of the peer reviewers who will review the issues of the journals that have been selected for review. Two or three peer reviewers are appointed for each journal. In addition to the peer reviewers' reports, information is obtained from the editor/editorial board of the journal that details the way in which the journal implements the 'Code of Best Practice in Scholarly Journal Publishing, Editing and Peer Review'⁷ and other details of the journal's practices and business model. In addition, bibliometric data on the performance of the journal may be obtained.

The peer reviewers' comments and recommendations regarding a journal, together with the other data obtained about the journal, are considered by the review panel and a report on the journal is formulated. The report on each journal is based on the data collected about the journal together with the peer reviewers' comments. This synthesis is prepared by the panel and represents their consensus view of the report on the journal.

Once the report on a particular journal has been approved by the review panel, it is sent to the editor/editorial board of the journal for comment. The request for comment is not regarded as an opportunity for rebuttal or explanation of the findings of the review panel, but rather an opportunity for the journal's editor to point out any errors of fact in the report or to draw the attention of the panel to any oversights that may have occurred. The panel considers the comments that it receives to this draft of the report and may make appropriate revisions if these are considered to be necessary.

The panel is also asked to make recommendations about the following:

1. Should the editor/publisher of the journal be invited to consider placing the journal on the SciELO South Africa platform if it meets the criteria for inclusion?

2. Make a recommendation to DHET regarding the suitability of the journal for LASAJ.

3. If the recommendation in (1) and/or (2) is negative, then guidance should be provided about what the journal needs to do to meet the criteria for inclusion.

4. Any additional recommendations for improvement and enhanced functionality.

Reports on all the journals considered in a particular disciplinary group are then consolidated into a single report which is prepared for publication. Prior to publication, the report is reviewed by the members of CSPiSA and if they are satisfied with its contents, the report is referred to the ASSAf Council for approval. Once approved by Council, the report is published and publicly disseminated, and copies made available for downloading from the ASSAf website.

The data in Table 1 indicate that over a period of a decade, the various review panels have managed to review and make recommendations about journals covering a broad sweep of disciplines. An explicit set of quality assurance mechanisms for assessing the quality of South African journals has been provided. Indeed, in a recent report on scholarly publishing in South Africa⁸, the following observation was made: 'We need ongoing analyses of SA publication practices to identify cases of questionable publication and again to alert the DHET and university research offices to such practices.' The report goes on to comment: 'Amongst a large number of interesting and relevant findings were disturbing indications of predatory publishing and questionable editorial practices.' The peer review of journals undertaken by ASSAf provides a level of oversight for South African journals that can mitigate the effects of some of the dubious publication practices identified. The fact that these practices are not more widespread amongst South African journals may at least partly be attributable to the advice provided by the National Scholarly Editors' Forum and the role that journal peer review has played in monitoring journal performance and encouraging reform where deficiencies are identified.

 

 

References

1. Academy of Science of South Africa (ASSAf). Report on a strategic approach to research publishing in South Africa. Pretoria: ASSAf; 2006. http://dx.doi.org/10.17159/assaf/0038        [ Links ]

2. Pouris A. Effects of funding policies on research publications in South Africa. S Afr J Sci. 1991;87:78-81.         [ Links ]

3. Steyn AGW, De Villiers AP. Public funding of higher education in South Africa by means of formulae. In: Review of higher education in South Africa. Pretoria: Council on Higher Education; 2007. p. 11-51. Available from: https://pdf4pro.com/view/review-of-higher-education-in-south-africa-28a64b.html        [ Links ]

4. Academy of Science of South Africa (ASSAf). Periodic peer review of South African scholarly journals: Approved process guidelines and criteria [webpage on the Internet]. No date [cited 2020 Aug 17]. Available from: https://www.assaf.org.za/index.php/programmes/scholarly-publishing-programme/peer-review-panels        [ Links ]

5. ASSAf. SciELO South Africa [webpage on the Internet]. No date [cited 2020 Aug 17]. Available from: https://assaf.org.za/index.php/programmes/scholarly-publishing-programme/open-access-scielo-south-africa        [ Links ]

6. Elsevier. Content policy and selection [webpage on the Internet]. No date [cited 2020 Aug 17]. Available from: https://www.elsevier.com/solutions/scopus/how-scopus-works/content/content-policy-and-selection        [ Links ]

7. Code of Best Practice in Scholarly Journal Publishing, Editing and Peer Review [webpage on the Internet]. c2018 [cited 2020 Aug 17]. Available from: https://sites.google.com/view/assaf-nsef-best-practice        [ Links ]

8. Academy of Science of South Africa (ASSAf). Twelve years later: Second ASSAf report on research publishing in and from South Africa. Pretoria: ASSAf; 2019. https://doi.org/10.17159/assaf.2018/0030        [ Links ]

 

 

Correspondence:
Robin Crewe
Email: robin.crewe@up.ac.za

Published: 21 September 2020

 

 

Recommendation No 5: that ASSAf be mandated jointly by the Departments of Education and Science and Technology to carry out external peer review and associated quality audit of all South African research journals in 5-year cycles, probably best done in relation to groups of titles sharing a particular broad disciplinary focus, in order to make recommendations for improved functioning of each journal in the national and international system. Academy of Science of South Africa (ASSAf)'