This document describes the evaluation criteria, policies and procedures adopted by the SciELO Project (http://www.scielo.org) for inclusion of scientific journals in the SciELO SA (www.scielo.org.za) electronic collection.
The general goal of the SciELO system is to contribute to
the development of scientific research by improving and expanding its means of
dissemination, publication, and evaluation through the intensive use of
electronic publishing, focused on a series of national platforms of emerging or
In the short term, the objective of the SciELO SA project is radically to increase the national and international visibility, accessibility and impact of South African journals by publishing them on the internet as national collection embedded in the SciELO system.
In order to meet the SciELO SA objectives, it is essential to promote the improvement of every aspect of scientific communication in the country, seeking to identify, stimulate, and develop a core collection of scientific journals with the same quality standards as leading scientific journals elsewhere. Therefore, the evaluation criteria for featured journals as well as the policies and procedures for their application are the main constituents' of the local SciELO Project.
The role of 'Advisory Committee' for the SciELO SA collection is played by the 'Committee for Scholarly Publishing in South Africa' (CSPiSA), currently chaired by Professor Wieland Gevers, a nationally esteemed scholar and previous President (1998–2004) and Executive Officer (2004–2008) of the Academy of Science of South Africa (ASSAf). The CSPiSA and its Chair manage the appointment process of discipline–grouped journal 'Peer Review Panels' (PRPs) , and the appointments are approved by the ASSAf Council.
ASSAf's Scholarly Publishing Programme (SPP) carries out an external independent peer review and associated quality audit of all South African research journals in 5–year cycles. This is done in relation to groups of titles sharing a particular broad disciplinary focus in order to make recommendations for improved functioning of each journal in the national and international system.
The periodic, grouped, quality assurance–directed peer review of South African research/scholarly periodicals functions analogously to the institutional quality audits of the Council of Higher Education (CHE) through its Higher Education Quality Committee ( HEQC), and focuses on: the quality of editorial and review processes; fitness of purpose; positioning in the global cycle of new and older journals listed and indexed in selective, established databases; financial sustainability; and scope and size issues. The ASSAf Panels carrying out the reviews each comprise 6–8 experts, at least half of whom are not directly drawn from the areas concerned.
The methodology comprises a detailed (mainly logistic) questionnaire sent to the editors, direct multiple, independent peer reviews of the journals in terms of content, and a panel meeting to review these materials and all other available evidence in order to make appropriate findings and recommendations.
The reports with recommendations are considered by the ASSAf 'Committee on Scholarly Publishing in South Africa', and released to the publishers and editors of the journals concerned, as well as to other stakeholders such as national associations, the Departments of Science and Technology and of Higher Education and Training, the CHE/HEQC, the National Research Foundation (NR) and 'Higher Education South Africa' (HESA), and all research–active institutions.
ASSAf Peer Review Panels have reviewed, or are busy reviewing, journals in the following subject areas:
Click on a subject to view a list of journals
For the process guidelines for setting up panels, peer reviewers, panel meetings and reports for the subject peer reviews of journals (revised in July 2012) see point 10 below.
The SciELO SA collection includes scholarly and scientific journals featuring articles resulting from original research and other original works. The review and approval of articles published in all these journals is carried out by peers, and is subject to authoritative editorial discretion.
The titles in the SciELO SA collection grow as each journal is approved for inclusion in the collection and recommended by the Peer Review Panels, the Committee on Scholarly Publishing in South Africa, and the Academy Council. The articles and other works may be written in English or Afrikaans.
Journals published by the SciELO collection can be published in print or online editions.
Only South African Open Access journals with no 'embargo' are considered for the SciELO SA collection. If such a journal is accredited by the South African Department of Higher Education and appears either on the ISI or IBSS lists and/or or has received a favourable peer review by the Peer Review Panels described above, then it qualifies to be included in the SciELO SA collection.
The South African Department of Higher Education and Training evaluates journals according to their accreditation criteria and policy.
Journals will continue to be featured in the SciELO SA collection subject to monitoring on the basis of performance indicators such as content quality and regular publication.
When a title does not automatically qualify for inclusion in SciELO, it is evaluated by peers (as described above) according to the following (summarised) criteria:
For the COMPLETE questionnaire used by independent peer reviewers to evaluate journals click here.
Periodicity is important for the flow of scientific production and the speedy communication of research. The SciELO Portal suggests the following minimum and desirable periodicity as well as the minimum and desirable number of articles per year a journal is expected to publish. SciELO SA aspires to reach towards a similar periodicity, where possible.
Inclusion of a journal in the SciELO collection is made after a favorable report is issued by the SciELO Advisory Committee, and confirmed by the ASSAf Council. The Director of the Academy of Science of South Africa's Scholarly Publishing Programme then extends an invitation to the editor of the journal to join the SciELO SA Open Access platform. Some editors prefer to not follow the open access route though.
The Peer Review Advisory Committee may, in addition to approval of inclusion, make recommendations for improvements and modifications to the journals.
As a rule, all the indicators adopted in the evaluation process for inclusion of titles in SciELO South Africa are also applied as initial inputs into the decision on its continuation in the collection.
Secondly, implementation of the Committee's recommendations for improvement of a journal will be considered as necessary for the journal to remain in the collection.
Thirdly, specific continuation indicators associated with the specific performance of journals in SciELO will be used to justify their continuation or not in the collection.
Punctuality is measured by the arrival of the electronic files of a journal at SciELO SA. File submission must strictly comply with the agreed periodicity of the journal. SciELO SA shall report any delays in file submission.
Journals that arrive late will be evaluated by the SciELO Advisory Committee, and will be subject to possible exclusion from the SciELO collection if not corrected without delay.
Journal use is measured monthly by the number of times it is accessed or viewed online. When the use of a journal is consistently low and/or tends to decrease when compared with other journals featuring the same subjects, the SciELO Advisory Committee will decide if the title should remain in the collection. The Committee will, however, first then initiate a study to establish the reasons for the failure and will either propose possible solutions, or recommend forthwith that the journal be excluded from the SciELO–SA collection.
The impact indicator of a journal title is measured by the average number of citations received by all articles in a set period following publication. This must be assessed together with the titles of journals in the same field of expertise.
An increase in the impact factor or retention of the same average as that of similar journals will be considered productive performance.
Exclusion of a journal from the SciELO collection can only be carried out after a decision report is issued by the SciELO 'Advisory Committee'.
In the case of unfavourable results obtained in the performance appraisal, the journal will be notified by the Committee of the aspects to be improved upon within the period/time stipulated.
Journal editors may appeal at any time against a decision by the SciELO 'Advisory Committee', both in relation to inclusion and exclusion processes.
Appeals will be examined by the 'Advisory Committee' and exclusion of the journal concerned may be re–examined. The Committee's report will be sent to the journal editor.
Journals excluded from the SciELO collection can be re–introduced provided that they meet the inclusion criteria required by SciELO South Africa. The SciELO Advisory Committee will analyze possible re–inclusion.
Re–inclusion of a journal title will not be made effective immediately after its exclusion from the SciELO collection. In order to be re–evaluated, a journal must demonstrate its compliance with the criteria for at least four consecutive issues or one year.
The quality assurance system for journals is conducted primarily through discipline–grouped peer reviews carried out by a series of purpose–appointed peer review panels (PRPs) drawn from the ranks of researchers and other experienced scholars in and around the fields concerned in each case, as well as persons with practical (technical) publishing experience. The proposed ASSAf PRPs are overseen by the CSPiSA, but appointed by the Academy Council. Their draft reports are sent to relevant stakeholders for comment and relevant inputs, before finalisation by the PRP concerned, and final consideration sequentially by the CSPiSA and the ASSAf Council.
It must be emphasized that the main purpose of the ASSAf review process of journals is to improve the quality of scholarly publication in the country and not an attempt to control these publications in any way. ASSAf respects the independence and freedom of researchers and of the research process itself as important preconditions for the critical and innovative production of new knowledge. At the same time, the work of South African researchers has to be assessed as part of the global community of scholars and scientists and in this respect ASSAf has an obligation to contribute to the improvement of the quality of such work where possible.
The quality assurance system for journals is conducted primarily through discipline–grouped peer reviews carried out by a series of purpose–appointed peer review panels (PRPs) drawn from the ranks of researchers and other experienced scholars in and around the fields concerned in each case, and also include persons with practical (technical) knowledge of publishing. The proposed ASSAf PRPs are overseen by the CSPiSA, but appointed by the Academy Council. Their draft reports are sent to relevant stakeholders for comment and relevant inputs, before finalisation by the PRP concerned, and final consideration sequentially by the CSPiSA and the ASSAf Council.
The following quote from the 2006 ASSAf Report clarifies the approach to be followed in the review of the journals and some aspects of the approach proposed:
The periodic, grouped quality assurance–directed peer review of South African research periodicals would function analogously to the quality audits of the CHE/HEQC, would be developed as an outcome of the Editors' Forum, and would focus on: the quality of editorial and review process; fitness of, and for purpose; positioning in the global cycle of new and old journals listed and indexed in databases; financial sustainability; and scope and size issues. The ASSAf panels carrying out the reviews would each comprise 6–8 experts, some of whom would not be directly drawn from the areas concerned, and would require data–gathering, interviews, and international comparisons, before reports with recommendations are prepared, approved, and released to stakeholders such as national associations, the Departments of Science and Technology and of Education, the CHE/HEQC, the NRF and HESA.
The first ASSAf PRPs reviewed and assessed sets of journals in two selected fields, and simultaneously assisted to refine the initial criteria and process guidelines (see below), for use in the subsequent reviews of further sets of journals by other panels.
It must be emphasised that the main purpose of the ASSAf review process of journals is to improve the quality of scholarly publishing in the country and not an attempt to control these publications in any way. ASSAf respects the independence and freedom of researchers and of the research process itself as important preconditions for the critical and innovative production of new knowledge. At the same time, the work of South African researchers has to be assessed as part of the global community of scholars and scientists and in this respect ASSAf has an obligation to contribute to the improvement of quality of such work where possible.
A number of criteria were explored in the part of the 2006 ASSAf Report (Chapter 4) that dealt with the survey of the then over 200 editors of accredited South African scholarly journals. Other possible criteria were proposed in other sections of the Report, or have since been suggested by members of the CSPiSA or the National Scholarly Editors' Forum; these are grouped and listed below – they have been consolidated in the Questionnaire sent to the editors of the journals being peer reviewed.
Existence of a peer review process (e.g. by a professional association).The questionnaire used by independent peer reviewers to evaluate journal can be viewed here.
An ASSAf Projects Officer of the Scholarly Publications Unit is assigned to support each Panel Chair, but reports to the Director of the Scholarly Publishing Unit in terms of review logistics and the production of draft and final review reports. The project officer is responsible for the following issues and activities:
The proposed PRPs is chaired by an ASSAf Member appointed by Council, who assumes accountability for the Panel's work in helping to developing a credible quality assurance mechanism for South African scholarly journals.10.8.1 Selecting Panel Members
Persons selected as panel participants will typically be drawn from ASSAf members, academic institutions, science councils and consultants.10.8.3 Conflict of interest
Organization of the panel is conducted by its Chair, supported by the assigned Projects Officer. The activities related to organization typically include:
Independent peer reviewers are selected on the basis of
their eminence and activity in their disciplines, drawing from the ASSAf
membership, registers of grant holders from the National Research Foundation
(NRF), Medical Research Council (MRC) and others, and from the leadership and
general membership of scholarly associations. Once they have agreed to serve,
they are provided with a set of questions (Appendix B) to be answered in
examining all the issues of particular journals that have appeared during the
preceding 2–3 years, or not fewer than eight issues. They are asked to examine
print or e–copies of the journals, and to submit a confidential report
including, if possible, comments in each of the areas specified in the question
list, plus on any other relevant matter.
As the individual reviews are subsumed in the process of drafting the consolidated consensus review, they are in effect anonymous and confidential.
A detailed and motivated draft report of each PRP's findings and recommendations are prepared by the assigned Projects Officer, working closely with the panel chair, and in consultation with the CSPiSA. The drafting of the consensus review from the individual reviews is key, and is overseen by the panel chair and the Director of the SPU. Draft materials are circulated to all panel members for review and comment before draft consensus reports are prepared. The relevant excerpts are sent to the editors and publishers for comment and correction of misconceptions and inaccuracies, after which the final versions of each report are prepared for consideration by the CSPiSA and subsequently the ASSAf Council. If approved, the reports are published by the Academy and made generally available. Specific submissions and recommendations are made to the Department of Higher Education and Training.
The finally agreed record should be submitted to the CSPiSA for approval, before submission in turn to the ASSAf Council, and public release
Academy of Science of South Africa (ASSAf)
Switchboard: +27 12 349 6600