This document describes the evaluation criteria, policies and procedures adopted by the SciELO Network Platform (http://www.scielo.org) for inclusion of scientific journals in the SciELO SA online collection.
The general goal of the SciELO system is to contribute to the development of scientific research by improving and expanding its means of dissemination, publication, and evaluation through the intensive use of electronic publishing, focused on a series of national platforms of emerging or developing countries.
In the short term, the objective of the SciELO SA project is to increase the national and international visibility, accessibility and impact of South African journals by publishing them on the internet as national collection embedded in the SciELO system.
In order to meet the SciELO SA objectives, it is essential to promote the improvement of every aspect of scientific communication in the country, seeking to identify, stimulate, and develop a core collection of scientific journals with the same quality standards as leading scientific journals elsewhere. Therefore, the evaluation criteria for featured journals as well as the policies and procedures for their application are the main constituents' of the local SciELO Project.
The 'Committee for Scholarly Publishing in South Africa' (CSPiSA) is currently chaired by Professor Robin Crewe, a nationally esteemed scholar and previous President (2009–2012) of the Academy of Science of South Africa (ASSAf). The CSPiSA and its Chair manage the appointment process of discipline–grouped journal 'Peer Review Panels' (PRPs), and the appointments are approved by the ASSAf Council.
ASSAf's Scholarly Publishing Programme (SPP) carries out an external independent peer review and associated quality audit of all South African research journals in 5–year cycles. This is done in relation to groups of titles sharing a particular broad disciplinary focus in order to make recommendations for improved functioning of each journal in the national and international system.
The periodic, grouped, quality assurance–directed peer review of South African research/scholarly periodicals functions analogously to the institutional quality audits of the Council on Higher Education (through its Higher Education Quality Committee (formerly HEQC), and focuses on: the quality of editorial and review processes; fitness of purpose; positioning in the global cycle of new and older journals listed and indexed in selective, established databases; financial sustainability; and scope and size issues. The ASSAf Panels carrying out the reviews each comprise 6–8 experts, at least half of whom are not directly drawn from the areas concerned.
The methodology comprises a detailed (mainly logistic) questionnaire sent to the editors, direct multiple, independent peer reviews of the journals in terms of content, and a panel meeting to review these materials and all other available evidence in order to make appropriate findings and recommendations.
The reports with recommendations are considered by the ASSAf's CSPiSA, and conveyed to the publishers and editors of the journals concerned, as well as to other stakeholders such as national associations, the South African Departments of Science and Technology and of Higher Education and Training (DHET), the Council for Higher Education (CHE), the National Research Foundation (NRF), Universities South Africa (USAf) and all research–active institutions.
ASSAf Peer Review Panels have reviewed, or are busy reviewing, journals in the following subject areas: To view the subject areas click here.
For the process guidelines for setting up panels, peer reviewers, panel meetings and reports for the subject peer reviews of journals (revised in July 2012) see point 10 below.
The SciELO SA collection includes scholarly and scientific journals featuring articles resulting from original research and other original works. As outlined above the review and approval of articles published in all these journals is carried out by peers, and is subject to authoritative editorial discretion.
The titles in the SciELO SA collection grow as each journal is approved for inclusion in the collection and recommended by the PRPs, the Committee on Scholarly Publishing in South Africa, and the Academy Council. The articles and other works should preferably be written in English.
Journals published by the SciELO SA collection can be published in print or online editions.
Only South African Open Access journals with no 'embargo' are considered for the SciELO SA collection. If such a journal is accredited by the South African Department of Higher Education and Training and has received a favourable peer review by the PRPs described above, then it qualifies to be included in the SciELO SA collection.
The DHET evaluates journals according to their accreditation criteria and policy.
Journals will continue to be featured in the SciELO SA collection subject to monitoring on the basis of performance indicators such as content quality and regular publication.
Inclusion of a journal in the SciELO SA collection is made after a favourable report is issued by the SciELO SA Advisory Committee, and confirmed by the ASSAf Council. The SciELO SA Operations Manager then extends an invitation to the editor of the journal to join the SciELO SA Open Access platform. Some editors prefer not to follow the open access route though.
The Peer Review Advisory Committee may, in addition to approval of inclusion, make recommendations for improvements and modifications to the journals.
The CSPiSA approved the establishment of a SciELO SA Advisory Committee to be responsible for this task. As a rule, all the indicators adopted in the evaluation process for inclusion of titles in SciELO SA are also applied as initial inputs into the decision on its continuation in the collection.
Secondly, implementation of the Committee's recommendations for improvement of a journal will be considered as necessary for the journal to remain in the collection.
Thirdly, specific continuation indicators associated with the specific performance of journals in SciELO SA will be used to justify their continuation or not in the collection.
Punctuality is measured by the arrival of the electronic files of a journal at SciELO SA. File submission must strictly comply with the agreed periodicity of the journal.
Journals that arrive late will be evaluated by the SciELO SA Advisory Committee, and will be subject to possible exclusion from the SciELO SA collection if not corrected without delay.
Journal use is measured monthly by the number of times it is accessed or viewed online. When the use of a journal is consistently low and/or tends to decrease when compared with other journals featuring the same subjects, the SciELO SA Advisory Committee will decide if the title should remain in the collection. The Committee will, however, first then initiate a study to establish the reasons for the failure and will either propose possible solutions, or recommend forthwith that the journal be excluded from the SciELO SA collection.
The impact indicator of a journal title is measured by the average number of citations received by all articles in a set period following publication. This must be assessed together with the titles of journals in the same field of expertise.
An increase in the impact factor or retention of the same average as that of similar journals will be considered productive performance.
Exclusion of a journal from the SciELO SA collection can only be carried out after a decision report is issued by the SciELO SA Advisory Committee.
In the case of unfavourable results obtained in the performance appraisal, the journal will be notified by the Committee of the aspects to be improved upon within the period/time stipulated.
Journal editors may appeal at any time against a decision by the SciELO SA Advisory Committee, both in relation to inclusion and exclusion processes.
Appeals will be examined by the 'Advisory Committee' and exclusion of the journal concerned may be re–examined. The Committee's report will be sent to the journal editor.
Journals excluded from the SciELO SA collection can be re–introduced provided that they meet the inclusion criteria required by SciELO SA. The SciELO SA Advisory Committee will analyse possible re–inclusion.
Re–inclusion of a journal title will not be made effective immediately after its exclusion from the SciELO SA collection. In order to be re–evaluated, a journal must demonstrate its compliance with the criteria for at least four consecutive issues or one year.
The quality assurance system for journals is conducted primarily through discipline–grouped peer reviews carried out by a series of purpose–appointed PRPs drawn from the ranks of researchers and other experienced scholars in and around the fields concerned in each case, and also include persons with practical (technical) knowledge of publishing. The proposed ASSAf PRPs are overseen by the CSPiSA, but appointed by the Academy Council. Their draft reports are sent to relevant stakeholders for comment and input, before finalisation by the PRP concerned, and ultimate consideration sequentially by the CSPiSA and the ASSAf Council.
The following quote from the 2006 ASSAf Report clarifies the approach to be followed in the review of the journals and some aspects of the approach proposed:
The periodic, grouped quality assurance–directed peer review of South African research periodicals would function analogously to the quality audits of the CHE, would be developed as an outcome of the Editors' Forum, and would focus on: the quality of editorial and review process; fitness of, and for purpose; positioning in the global cycle of new and old journals listed and indexed in databases; financial sustainability; and scope and size issues. The ASSAf panels carrying out the reviews would each comprise 6 – 8 experts, some of whom would not be directly drawn from the areas concerned, and would require data–gathering, interviews, and international comparisons, before reports with recommendations are prepared, approved, and released to stakeholders such as national associations, the Departments of Science and Technology and of Education, the CHE, the NRF and USAf.
It must be emphasised that the main purpose of the ASSAf review process of journals is to improve the quality of scholarly publishing in the country and not an attempt to control these publications in any way. ASSAf respects the independence and freedom of researchers and of the research process itself as important preconditions for the critical and innovative production of new knowledge. At the same time, the work of South African researchers has to be assessed as part of the global community of scholars and scientists and in this respect ASSAf has an obligation to contribute to the improvement of quality of such work where possible.
A number of criteria were explored in the part of the 2006 ASSAf Report (Chapter 4) that dealt with the survey of the then over 200 editors of accredited South African scholarly journals. Other possible criteria were proposed in other sections of the Report, or have since been suggested by members of the CSPiSA or the National Scholarly Editors' Forum; these are grouped and listed below – they have been consolidated in the Questionnaire sent to the editors of the journals being peer reviewed.
Existence of a peer review process (e.g. by a professional association). The questionnaire used by independent peer reviewers to evaluate journal can be viewed here.
An ASSAf Project Officer of the Scholarly Publications Unit is assigned to support each Panel Chair, but reports to the Director of the Scholarly Publishing Unit in terms of review logistics and the production of draft and final review reports. The project officer is responsible for the following issues and activities:
The proposed PRPs is chaired by an ASSAf Member appointed by Council, who assumes accountability for the Panel's work in helping to developing a credible quality assurance mechanism for South African scholarly journals.10.8.1 Selecting Panel Members
Persons selected as panel participants will typically be drawn from ASSAf members, academic institutions, science councils and consultants.10.8.3 Conflict of interest
Organization of the panel is conducted by its Chair, supported by the assigned Projects Officer. The activities related to organization typically include:
Independent peer reviewers are selected on the basis of their eminence and activity in their disciplines, drawing from the ASSAf membership, registers of grant holders from the National Research Foundation (NRF), Medical Research Council (MRC) and others, and from the leadership and general membership of scholarly associations. Once they have agreed to serve, they are provided with a set of questions (Appendix B) to be answered in examining all the issues of particular journals that have appeared during the preceding two to three years, or not fewer than eight issues. They are asked to examine print or e–copies of the journals, and to submit a confidential report including, if possible, comments in each of the areas specified in the question list, plus on any other relevant matter.
As the individual reviews are subsumed in the process of drafting the consolidated consensus review, they are in effect anonymous and confidential.
A detailed and motivated draft report of each PRP's findings and recommendations are prepared by the assigned Projects Officer, working closely with the panel chair, and in consultation with the CSPiSA. The drafting of the consensus review from the individual reviews is key, and is overseen by the panel chair and the Director of the SPU. Draft materials are circulated to all panel members for review and comment before draft consensus reports are prepared. The relevant excerpts are sent to the editors and publishers for comment and correction of misconceptions and inaccuracies, after which the final versions of each report are prepared for consideration by the CSPiSA and subsequently the ASSAf Council. If approved, the reports are published by the Academy and made generally available. Specific submissions and recommendations are made to the DHET.
The finally agreed record should be submitted to the CSPiSA for approval, before submission in turn to the ASSAf Council, and public release.
Academy of Science of South Africa (ASSAf)
Switchboard: +27 12 349 6600