ARTICLES

 

In Search of the Perceived Quality and Impact of Accredited South African Law Journals: Exploring the Possibility of a Ranking System. A Baseline Study: 2009 - 2014

 

 

M Carnelley*

North-West University South Africa

 

 

---

ABSTRACT

The DHET Research Output Policy (2015) indicates that there has been a change in the government's approach to research funding. Previously all research published in any accredited journal was rewarded equally. A decision has been taken, however, that a shift will be made towards rewarding better quality and higher impact peer-review research. Additional mechanisms such as biometric/bibliometric data, including citations, assessments by discipline-specific panels of experts and/or post-publication reviews may be used to determine the quality and impact of publications. The policy notes that the DHET may distinguish between "high" and "low" impact journals after proper consultation.
This article highlights the need for consultation by the legal fraternity with the DHET about the implementation of these possible mechanisms in the light of the special considerations applicable to the evaluation of law journals: most journals publish mainly local legal content, there is a limited number of active legal academics, the nature of legal research is not empirical, and a premium is placed on the writing of books.
The research evaluates the available data between 2009 and 2014 in an attempt to assess if it would be appropriate to introduce a legal journal ranking system in South Africa. The article discusses direct and indirect forms of quality evaluation to inform possible ranking systems. This includes the data from the ASSAf expert panel evaluation of law journals in 2014 and other bibliometric data based on whether the journal is featured in international accredited lists, the size of its print-run, author prominence, rejection-rate, usage studies, and evaluations based on citations. An additional ranking system is considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for rating.
The article concludes that a law journal ranking system would be inappropriate for South Africa. None of the systems meet the minimum requirements for a trustworthy ranking of South African law journals, as the data available are insufficient, non-verifiable and not based on objective quality-sensitive criteria. Consultation with the DHET is essential and urgent to avoid the implementation of inappropriate measures of quality and impact assessment.

Keywords: Ranking; law journals; Department of Higher Education; bibliometric data; citations; quality.

---

 

 

1 Introduction: A changing research landscape

There is something fundamentally absurd about the idea of ranking research. At the same time, no one can seriously argue that all research is equal in importance and quality. Either way, we are doubtless witnessing a dramatic change in the management and organisation of research. One aspect of the change is a move towards the ranking of research.¹

The assessment of the quality of research in higher education is a relatively new international phenomenon.² Academics are required to publish more, to publish faster,³ and to demonstrate the quality and significance of their research to their employers and state funders.⁴ This trend is prevalent in South Africa as well, and is part of a larger corporatisation movement to introduce managerial mechanisms. Initially developed to measure performance in profit-making enterprises, quality assessment was extended into academia to "improve efficiency and economy"⁵ at universities and to advance capacity, quality and innovation.⁶ The merits of this trend and its potential impact on academic freedom are excluded from this discussion.⁷

In developing any national research policy:

It ... is worth considering three fundamental issues: first, whether there is a need to assess the quality of research outputs and, if so, whether it is better to control the assessment process centrally or devolve the process to individual institutions; second, if research assessment is deemed useful, then what form it should take; third, whether outcomes should be explicitly linked to the distribution of research funding.⁸

The need for the quality assessment of research output is recognised and accepted in South Africa as well as internationally, and a form of such assessment is already in use.⁹ Research assessment is seen as useful and necessary, and is currently practised in the form of rewarding all contributions to accredited peer-reviewed journals in the form of the government "publication output subsidy".¹⁰ Over the past decade or so public universities have successfully encouraged their academics to publish peer-reviewed¹¹ research in accredited journals in line with this policy, even though the actual number of academics has remained fairly stable.¹² Official opinion will have it that although the policy has resulted in increased output, it has not necessarily led to an improvement in quality.¹³

Academic performance measurement generally leads to a growth in measurable output but seldom results in a higher quality of the research or direct reallocation of funds to the best performers.¹⁴

Both the 2013 White Paper for Post-School Education and Training¹⁵and particularly the 2015 DHET Research Output Policy¹⁶indicate that a different government approach is to be adopted in an attempt to reward better quality and higher impact peer-reviewed research.¹⁷

Leaving aside the problematic definitions of "quality" and "impact",¹⁸ the question may well be asked why the specific focus on quality and impact is currently so prevalent. Should not the fact that an article has been peer-reviewed and published in an accredited journal be a sufficient indication that the publication meets the set minimum quality standards?¹⁹ After all, ASSAf does engage in random expert post-publication evaluations of accredited South African law journals. The ASSAf review of law journals in 2014 resulted in a finding that all (but one) of the accredited law journals on the DHET list were in line with the DHET policies and had met the minimum standards vis-à-vis inter alia the peer-review process.²⁰ Thus, all these accredited journals had been found, in principle, to be publishing appropriate research, in line with the DHET policy as "original, systematic investigation undertaken to gain new knowledge and understanding".²¹

Although the journals may meet the minimum criteria, it has become evident that peer-reviewed published research is not always of uniform standard.

While peer-review is one of the most fundamental indicators of the quality of a research journal, the way it is applied is what reflects the journal's standards and indicates the overall quality of the research presented in its pages.²²

The variation in the standard of published research could be ascribed to numerous factors such as the diverse peer-review processes employed by editors, differences in the assessment standards used by peer-reviewers, and/or possibly the proliferation of accredited peer-reviewed journals recognised by the DHET, each with its own selection nuances. ²³ Another factor that could negatively influence the standards of output is the increase in the number of predatory publishers and journals, some of which also appear on accredited lists. These seem to focus on profit rather than on an adherence to strict peer-review quality standards, making it easier for authors to get published.²⁴

In addition, there is a negative perception of the quality of South African journals.²⁵ Because of the small pool of academics in any specific discipline, it is inevitable that the number of journal submissions as well as of peer reviewers will be limited - casting doubt over the independence of the peer-review system.²⁶

The negative perceptions about quality prompted the DHET to amend its research assessment policy as from 2016 to ensure that it received value for its subsidy investment in academic research.²⁷ In terms of this policy, the DHET will continue to determine the quality of research output by proxy.²⁸However, from 2016 the quality assessment of research output may include additional mechanisms such as biometric/bibliometric data (including citations),²⁹ discipline-specific panels of experts and post-publication reviews by the DHET.³⁰ The DHET "may consider introducing such measures as the categorising of journals as 'high' or 'low' impact; citation indexes or other relevant and appropriate quality measurements as prerequisites, after due and extensive consultation process with this sector."³¹ No formal consultation process with legal academia on this issue has yet begun at the time of the writing of this article, but certain universities are already differentiating between legal journals on national and international accredited lists.³²

It must be stated that whatever form any additional research assessment takes, it could never be truly objective.³³ Then, the additional burdens of the cost of the implementation of such a system and the inconvenience to individuals and universities by diverting attention to non-core business issues must also be reckoned with.³⁴

The task facing the South African legal fraternity is to agree on additional quality improvement measures that would generally be regarded as suitable and that could be implemented successfully as a means of determining the impact or quality (or perceptions of the impact or quality) of the South African legal research output, or as a means of distinguishing between the exceptional and the average. The consequences of non-engagement may result in the DHET determining measure(s) that may or may not be appropriate for legal academics.

When one considers the terms "high" and "low" quality and "impact", it stands to reason that a ranking system, identifying the better quality journals could be introduced. ³⁵ However, such a system has not been universally acceptable for all legal jurisdictions.

In the USA, for instance, quality is assessed through a direct journal ranking system as opposed to an assessment of individual articles.³⁶ This system is well developed with an extensive published discourse about the types of journal ranking systems employed. More about these will form the basis of the discussion in the latter part of this article.

Belgium and Australia have experimented with ranking systems, albeit not too successfully.³⁷ Van Gestel notes that in Belgium the 2004 ranking list was so controversial that it was never implemented and in Australia the 2010 formal journal rankings were abandoned by 2011.³⁸ The Netherlands and the UK have not even attempted to implement a ranking system.³⁹ The Netherlands uses "qualitative reviews by panels of international experts for its external reviews" to assess research outputs.⁴⁰ The UK implemented an external post-publication evaluation process where selected individual articles submitted by universities are assessed on merit.⁴¹ The 2014 UK Law sub-committee confirmed the view "that peer review remains the most reliable method of assessing research quality in law".⁴² In the UK the external evaluation process of selected research output submissions, rather than the ranking of journals, serves as quality-control.⁴³ In 2001 and again in 2008 the law panel in the Research Assessment Exercise (England) concluded (my emphasis):

Work of internationally-recognised excellence was found in a wide range of types of output and places, and in both sole and jointly authored works .... First-rate articles were found in both well-known journals and relatively little-known ones. Conversely, not all the submitted pieces that had been published in 'prestigious' journals were judged to be of international excellence. These two points reinforced the Panel's view that it would not be safe to determine the quality of research output on the basis of the place in which they have been published or whether the journal was 'refereed'.⁴⁴

It is submitted that should the DHET in South Africa implement a system of differentiating among journals based on their "low" and "high" impact, whether by a panel of experts and/or the use of bibliometric data and/or through other methods, it is conceivable that a journal ranking system would follow, either officially or unofficially, unless a suitable alternative could be found.

 

2 Journal ranking systems

It has been argued that a journal ranking system could serve to increase the quality and impact of scholarship and create incentives for journal editors to select and publish only quality submissions; which, in turn, would motivate academics to strive to produce work of higher quality.⁴⁵ However, this is not always the case as "ground-breaking 'must read' articles are as likely to be published in less prestigious journals as those held in particular high regard".⁴⁶ Research has shown that a ranking system may "stifle diversity and innovation", as journal editors may prefer to publish mainstream articles to increase their rankings rather than new and experimental research.⁴⁷ An unintended consequence (also of incentivising editors to publish only high quality outputs) may be a chilling effect on young academics, wary of rejection, or, where those young academics do submit outputs, editors may reject them. In the light of the above observation regarding the relatively small pool of persons working in legal fields in South Africa, this could be catastrophic going forward. And this would be even more devastating in respect of the project of transforming academia in respect of the development of black academics.

Academics could benefit from a ranking system as it could act as a guide to their choice of journal.⁴⁸ Publication in higher-ranking journals would afford prestige, as it would signal potential superior quality,⁴⁹ which could lead to favourable outcomes regarding promotions and career paths.⁵⁰ The reputations of law schools would benefit⁵¹ if their journals obtained high-ranking status.⁵² Higher-ranked journals in turn could benefit other stakeholders: they would be more widely purchased,⁵³ read and cited, with accompanying benefits, as serious scholars are likely to prefer making use of more prestigious journals.⁵⁴ From a journal editor's perspective, it could also translate into potential reviewers approached for review being more likely or inclined to review submissions.

For funders and managers, exceptional quality could be rewarded and promoted.⁵⁵ For the journals, a negative change in ranking may give rise to self-evaluation and reflection.⁵⁶

If journal rankings become established and respected in the legal and academic community, they can have a significant effect on the content of legal scholarship produced nationwide. This conclusion suggests that attempts to rank journals are extremely significant to the scholarly enterprise ...⁵⁷

According to Perry⁵⁸ a ranking system should adhere to certain minimum requirements.⁵⁹ First, it should be based on quality-sensitive criteria.⁶⁰ This could be problematic because, as mentioned above, one journal may contain both excellent and mediocre articles.⁶¹ Secondly, the ranking methodology must be sensitive to changes in quality and must make allowance for regular revision and updating.⁶² Thirdly, the ranking must be based on objective criteria, free from bias,⁶³ and practical, with enough available data to fulfil the goals of the ranking.⁶⁴ The data should be readily verifiable and not susceptible to manipulation.⁶⁵ For a ranking system to be successful, it should thus be "carefully designed. If the ranking method is not defensible, then the resultant ranking will not fulfil its goals".⁶⁶ These requirements will be used hereunder to evaluate the ranking systems discussed.

It has been acknowledged that a multi-factor combination ranking rather than a single factor system is preferred,⁶⁷ although using a combination of factors may be simply "too burdensome" - especially where a single factor data system has not yet been collected and coded.⁶⁸ The use of a multi-factor method is, however, not beyond criticism, as the person determining the ranking has:

... to determine how the different factors should be combined to generate the ultimate ranking. The weight that is assigned to each factor is crucial, and since this determination is purely subjective (and most likely controversial), a complex ranking method can [also] never be objective.⁶⁹

The question arises whether it is possible to achieve a successful ranking system in South Africa, given the limited number of law journals published in the country, including those of a highly specialised nature, and taking into account the limited number of legal academics in the country.

The ASSAf Report on Grouped Peer Review of Scholarly Journals in Law notes the following special considerations for the evaluation of law journals.⁷⁰ One: legal content is more locally-orientated than other disciplines, as the legal principles under scrutiny are mostly jurisdiction-specific.⁷¹ Two: the limited number of active academics in a specific sub-area impacts on the number of possible submissions⁷² and qualified peer-reviewers. Three: the type of research is generally not primary research or empirical in nature⁷³ and can be carried out with minimal resources, unlike other types of research that require extensive funding.⁷⁴ Four: a premium is placed on the writing of books and chapters in books, resulting in the fact that not all legal scholarship can be found in journal articles.⁷⁵ And five: research can be focused on applied legal practice, where research academics are agents for legal changes or developments.⁷⁶ Law is always a "discipline in transition" where quality and impact should not rely solely on historical accuracy and data.⁷⁷ Legal commentary is aimed at the legal profession and a case note suggesting an alternative approach may have a profound effect on the law that more "lengthy and academic papers published in prestigious journals do not have".⁷⁸ Similarly, textbooks clarifying a complex legal issue in an accessible manner may "appear trivial to researchers from other disciplines [but] are in fact highly valuable and more sophisticated than they may seem at first glance".⁷⁹

Such work, [law review articles] ... has earned the real respect of the bench. We admire the law review for its scholarship, its accuracy, and, above all, for its excruciating fairness. We are well aware that the review takes very seriously its role as judge of judges - and to that, we say, more power to you. By your criticisms, your views, your appraising cases, your tracing the trends, you render the making of 'new' law a little easier. In a real sense, you thus help to keep our system of law an open one, ever ready to keep pace with the changing patterns.⁸⁰

Taking the above into consideration, the aim of this article is to focus on the consequences of a possible law journal ranking system for South Africa in determining perceived quality and impact. The author will assess how the existing journal information for the period from 2009 to 2014 would have been evaluated, had the various US-type journal ranking systems been imposed. The article includes direct and indirect quality evaluations.⁸¹ The direct evaluation consists of the ASSAf expert panel evaluation. Indirect evaluations through biometric/bibliometric data are based on whether the journal is featured in internationally accredited lists, the size of its print run, author prominence, the rejection rate, usage studies based on the perusal of library and electronic databases, and evaluations based on citations - in other journals as well as by the courts. An additional and uniquely South African ranking system is also considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for an NRF rating.

In this article the principles, advantages and disadvantages, and the outcome of the principles applied (with some nuances) to the South African law journals will be discussed. Finally, the spread of the rankings per journal and an average of all the rankings are shown, flawed as they may be. The article concludes with an assessment of whether the application of the various systems resulted in a consistent ranking outcome or whether the results showed a marked difference in ranking depending on the ranking system used. This information could form the basis of a more informed decision about the viability of ranking systems for law journals in South Africa or whether an alternate evaluation system by the DHET is called for.

Of the twenty-three peer-review law journals that met the ASSAf minimum criteria for accreditation, twenty-one will be considered for this article.⁸² It should be noted from the outset that the immediate problem was the "lack of reliable, comparable and comprehensive data",⁸³ and this article should therefore be treated as exploratory - as a starting point for a debate about the quality and impact evaluation of South African law journals. It is not intended to be comprehensive and neither the parts nor the whole is without fault or beyond criticism.⁸⁴ The various aspects could and should be improved upon by additional research and debate.⁸⁵ That said, the author expects to make a useful contribution by exposing the dangers of a ranking system and the need for engagement with the DHET.

 

3 Direct quality and impact assessment through expert panels

In an ideal world, direct quality evaluation for all law journals would be performed by a panel of experts who regularly evaluate journal contributions objectively, according to prescribed criteria.⁸⁶ However, it is unrealistic to expect academics who are over-burdened as it is and not expert in all areas of the law to find the time to devote to additional and continuous quality peer reviewing.⁸⁷ Even where sub-specialisation panels are utilised and the number of specialised journals is limited, the process would remain time-consuming, subjective and therefore problematic.⁸⁸ The logical conclusion would be that academics would evaluate their own articles as well as those of their peers, making the system inherently subjective, biased⁸⁹ and potentially "self-perpetuating".⁹⁰

[Peer-review, w]hile this is an important step in the right direction ... can, nevertheless give unreliable assessments. Studies have shown peer reviews can produce inconsistent results . ⁹¹

The process of ranking journals is complex and even experts might find it challenging to evaluate the varying quality of different journals.⁹²

The South African ASSAf panels mentioned above have been active for some time and periodically assess sample journals through the ASSAf Committee on Scholarly Publishing in South Africa to determine whether, post-publication, journals still meet the minimum set criteria for inclusion in the accreditation lists.⁹³ This process is not a journal ranking system and it does not designate "low" or "high" impact status. For now, it provides the best available data for the purposes of this paper.

South African law journals were assessed by the ASSAf in 2014 and the results were published in their Report on Grouped Peer Review of Scholarly Journals in Law. The evaluation of the journals was based on the best practices set out in the ASSAf Editorial Process-related Criteria,⁹⁴which are aimed at eventually promoting available quality open-access online research.⁹⁵ As mentioned earlier, the criteria used by the panels included editorial process-related criteria set out in the Code of Best Practices,⁹⁶as well as business-related criteria⁹⁷ and bibliometric assessments.⁹⁸

Although the process does not rank journals, the panels do comment on the quality of the contributions in their individual assessments. Descriptions of the quality of the journals are used in this article to determine if and/or how the journals could/should be ranked. They were, however, not standardised and objective, and as such neither very accurate nor very helpful.⁹⁹ For instance, would a journal described as "high" quality be the same as "generally high"? Is there a difference in quality between being "good overall" and "generally good"? If the first is inconsistent and of varied quality, and the other included "very good contributions", does that mean that the latter should be rated higher? Where the evaluating panel described a journal as "generally good" would that imply that some contributions were poorer and others better?

If the ranking of law journals should become inevitable, these panels may be in the best position to carry out such a task, although a truly accurate ranking may remain elusive and subjective.

For the ranking of journals based on the available information, flawed and subjective as it may be, five different groups are identified - a sort of scale of perceived quality as expressed by the ASSAf Report:

The SALJ referees were unanimous that the SALJ publishes articles of a high quality.¹⁰⁰ In fact, it noted that the SALJ is "South Africa's premier law journal".¹⁰¹

The second group included the Annual Survey and the ILJ, which were regarded as examples of "the best work done in South Africa".¹⁰²

The SAJHR,¹⁰³the SAYIL¹⁰⁴ and Fundamina¹⁰⁵fell into the third group, being described as of a "high" quality without reservations.

The fourth group, including Stell LR, LDD, TSAR, JJS, SACJ, Obiter, THRHR and CILSA, was described as publishing good material, subject to a qualification. Articles in Stell LR¹⁰⁶ received special mention and the quality was labelled "generally high". LDD¹⁰⁷ articles were noted to be "generally very good". The quality of the articles in TSAR¹⁰⁸ was described as "good overall" and those in JJS,¹⁰⁹ SACJ¹¹⁰ and Obiter¹¹¹as "generally good". The contributions in the THRHR¹¹²were also described as "high", although there was concern that some articles seemed to be primarily descriptive and to have very little theoretical content. The CILSA publications were described as of a "high quality", but concerns were raised about the lack of variety and about not keeping pace with changes in the area.¹¹³

The fifth group consisted of journals that were regarded as publishing articles of varying quality, but as being nonetheless worthy of accreditation. This group includes AHRLJ, De Jure, PER, SA Public Law, Acta Juridica, Merc LJ and Speculum Juris. The Report noted that in AHRLJ¹¹⁴the quality of the contributions varied between and within issues, with some being excellent whilst others are average, but most were judged as being "good". The articles in De Jure,¹¹⁵ PER,¹¹⁶ SA Public Law ¹¹⁷ and Acta Jurídica¹¹⁸were described as varied, ranging from adequate, acceptable or average to good. SA Public Law occasionally had "very good contributions",¹¹⁹ and the Merc LJ was "generally good with some exceptions".¹²⁰Speculum Juris¹²¹contributions were described as "a mixture of more academic and more practical articles [that] compares well with general national law journals in Europe, America and the UK".

In conclusion, it is submitted that the information generated through the ASSAf Report does not meet the Perry minimum standards for a trustworthy and acceptable ranking system, mainly because it is neither objective nor free from bias, but also as it is impractical and not easily verifiable. A ranking based on the five groups was nonetheless included in the summary of the data under 5 hereunder.

As an aside, this article would not be complete without mentioning that an alternative method of determining the perceived quality of journals exists in the form of perception-based questionnaires or surveys similar to those of the Crespi¹²² and Campbell, Goodacre and Little¹²³ studies. However, this system is also controversial because of its subjectivity and the fact that discretionary viewpoints cannot be standardised.¹²⁴ In addition, the respondents may not be equally familiar with all of the journals¹²⁵ and research has shown that "geographical origin, research orientation and affiliation with a journal" play a significant role in the assessment made by of the respondent.¹²⁶ As far as the outcome of these studies is concerned, there may be some consensus about who should make the list, but the ranking of the journals remains unclear.¹²⁷ As no South African academics or other role players have taken part in such surveys between 2009 and 2014, none could be included in this article.¹²⁸

 

4 Indirect quality evaluation by bibliometric data

Not everything that counts can be counted, and not everything that can be counted counts.¹²⁹

4.1 Introduction

Although there has been a rise in the use of bibliometric indicators in legal scholarship, it is not yet regarded as being on a par with or as effective as the expert peer review process.¹³⁰ This type of data is also biased in favour of English publications and older legal sub-disciplines.¹³¹ Van Gestel and Vranken¹³² ask the following question:

What problems are bibliometric research indicators really meant to solve? ... The purpose seems to have shifted [from furthering the scholarly quality of individual publications] towards creating an instrument for oversight, management and policy, which is just as ineffective in guaranteeing a lasting high quality of scholarly publications as substantive assessment by peers. Implementing both systems cumulatively would only add to the burden on the time and efforts of researchers to justify their work, leaving less time for research and education.

Bibliometric data serve the purpose of gauging a journal from another perspective to get an indication of the productivity and depth of impact amongst discipline peers. It is regarded as being more objective, as it circumvents the "old boy's network", is cheaper and more transparent.¹³³The systematic use of bibliometric data is, by its very nature, rooted in history. It assesses the past as a possible indication of future performance, but excludes new discoveries, new researchers and new universities.¹³⁴

Obtaining reliable data is problematic¹³⁵ because law journals themselves do not always present a full picture of academic endeavour. As mentioned earlier and discussed later, books and chapters in books are also important sources of information in the legal field.¹³⁶

With the available information, the following eight bibliometric systems were considered for this article: inclusion in international accredited lists; the print-run; author prominence; the rejection rate; library usage; the citation index; court citations; and NRF rating choices.

4.2 International accredited lists

Although this article is limited to the ranking of law journals on the DHET list of accredited journals, some of these journals also appear on international accreditation lists recognised by the DHET. Could it be argued that journals appearing on international lists should be ranked higher? It may well be that inclusion on numerous accreditation lists could have a positive influence on their impact because it makes the journals more accessible. But are these journals necessarily of a higher quality?

On the one hand it could be argued that they are not necessarily so, as the criteria used for inclusion in any of the lists are similar to those used by the DHET. No information is available about the reasons why all the journals are not on international lists. For instance, did they apply and were they rejected, or did they not apply at all?

Certain South African public institutions award greater financial incentives to academics who publish in journals accredited in international lists, indicating perceptions of their better quality or greater impact.¹³⁷ Most importantly, to be accepted for and remain on these international lists the journal must undergo an additional systematic and continuous evaluation by experts, using set criteria of scholarly expertise,¹³⁸ including peer-reviewed¹³⁹ high quality analytical research under an international editorial board of academics. Most of these lists have their own areas of expertise that journals must adapt to for inclusion. NSD and ISI focus on the diversity of authorship. ISI specifically considers the citation index of the journal within the context of the discipline. SciELO is focused on the DHET criteria as confirmed by the ASSAf evaluation panel reports; and Scopus on indexing and citations.

For the purposes of this article and because these journals have been subjected to additional external scrutiny and evaluation and have been found to meet their specific criteria, the journals that appear on international lists, and the number of times they appear are ranked higher for the purposes of this article.

Only one South African law journal appeared in four additional lists in 2014,¹⁴⁰ namely the SAJHR, which is ranked first. AHRLJ, CILSA and PER were each listed in two of these lists,¹⁴¹ and are jointly ranked second. The titles of ten journals appeared in only one international list: SALHR, SALJ, JJS, THRHR and AHRJ in IBSS; and SAJHR, SAYIL and the SACJ in the NSD list. The SciELO list also included De Jure, Fundamina and LDD.

In conclusion, it is reiterated that this system is not a true reflection of the quality of a journal's research output and as such does not meet the Perry minimum requirement for quality-sensitive criteria. It is nevertheless included in the summary under 5 hereunder.

4.3 Ranking based on the print run

The print run of a journal used to be an indication of its popularity, coverage and visibility, including in South Africa, as market forces generally differentiate between journals based on their relevance and impact.¹⁴²

The ASSAf Report noted the print-run of all hard-copy law journals, but the information could not be verified independently. Print run as a measuring tool has limited usefulness, as the number of copies of the journals printed does not necessarily indicate the quality of the research per volume. Serial journal subscription should also be taken into account. Moreover, in later years some journals have decreased their print runs as they became available electronically. Others, like PER, have never been available in hard copy, while TSAR was unwilling to release its print run information to ASSAf,¹⁴³ this making the inclusion of these two journals in the ranking impossible.

The stated number of hard copies printed is as follows: SALJ (1150),¹⁴⁴ILJ (1000),¹⁴⁵Annual Survey (750),¹⁴⁶AHRLJ (650),¹⁴⁷Acta Jurídica (600),¹⁴⁸THRHR and Merc LJ (550 each),¹⁴⁹SAYIL (400),¹⁵⁰SAJHR (391),¹⁵¹LDD (375),¹⁵²SACJ (350),¹⁵³Obiter and Stell LR (315 each),¹⁵⁴CILSA (300), Fundamina (300),¹⁵⁵De Jure (260),¹⁵⁶JJS (250),¹⁵⁷SA Public Law and Speculum Juris (200 each).¹⁵⁸

Although seemingly straightforward, print run is not an effective measuring tool and it is evident that it does not meet the Perry minimum requirements for an acceptable ranking system, in that the criteria are not quality-sensitive. In addition, in the light of the move towards open online access, the print run will become less relevant. It is still included in the ranking data hereunder at 5, however.

4.4 Ranking based on author prominence

We do not believe that we need to provide a detailed justification. Right or wrong, good or bad, justified or unjustified, prestige speaks volumes in the legal - and legal academic - world ... Accordingly, we think our decision to attempt a prestige-based ranking of specialised reviews will strike most readers as intuitive.¹⁵⁹

In 1997 Jarvis and Coleman¹⁶⁰ ranked law reviews in the US by author prominence and in 1999 George and Guthrie¹⁶¹ did the same for speciality law journals.¹⁶² The rationale for this ranking methodology is that it "reflects the common-sense intuition that the prestige of a journal depends largely upon the prestige of the authors whose articles it publishes".¹⁶³ Law journals were ranked over a five-year period, using a 1 000-point contributor scale according to the prestige of the authors of lead articles at the time of publication.¹⁶⁴ The scale ranged from 1 000 points for an article by the US President to 750 for a US Circuit Court judge to 625 for a law professor at a first-tier law school.¹⁶⁵ Although the creators acknowledged that the scale itself was subjective, they argued that the exact points were not as important as the consistency in comparing journals.¹⁶⁶

The main flaw in this method is its subjectivity,¹⁶⁷ which is highlighted by Crespi,¹⁶⁸ Perry¹⁶⁹ and Korobkin.¹⁷⁰ It is the status or prominence of the author which determines the rankings rather than the scholarliness or quality of the article published.¹⁷¹ Author prominence does "not necessarily correlate with creativity, innovation, profundity, style, usefulness, or impact on legal thought or practice".¹⁷² It incentivises editors to select articles based on the prestige the author might lend the journal, rather than the quality of the submission.¹⁷³

Applying the system in South Africa, a slightly amended scale based on the Jarvis and Coleman contributor scale is adopted for this article.¹⁷⁴ However, it remains subjective, unsatisfactory and open to criticism.¹⁷⁵

 

Table 1

 

The average contributor score for each volume per year was used for the calculations.

 

Table 2

 

Should this system be used for ranking, TSAR and Annual Survey would be ranked jointly first (586), with THRHR third (551), Fundamina fourth (549) and SA Public Law fifth (546). The next group would contain Acta Juridica (527), De Jure (525), Merc LJ (524) and Stell LR (524) with SACJ (521) next.

In conclusion, even though this ranking system does not meet the Perry minimum criteria for its lack of quality-sensitivity, it is nonetheless included under 5 hereunder. Furthermore, in the South African historical context focussing on author prominence as the basis of a ranking system would not be an appropriate measure. It might dis-incentivise editors from publishing articles by younger (black) academics as they may not yet register high on the prominence scale, resulting in the possibility of constraining the academic transformation process.

4.5 Ranking based on the rejection rate

Another ranking system for journals might be based on the rejection or acceptance rates of submitted articles. The argument is that the more selective the journal, the higher its rejection rate, which could indicate a higher quality of scholarly publication. Perry¹⁷⁶ argues that this method is unreliable for the following reasons: cooperation from the editors is required in collecting and producing the data, and the information may be unreliable and unverifiable; and differences in rejection policies may distort the rejection rates and result in the artificial alteration of the rates. He asks, for instance, if the response "revise and re-submit" is a rejection or not?¹⁷⁷ Is the non-acceptance of a quality article because of systematic preferences a rejection? The rejection rate focuses only on the selection made by the journal among the articles submitted to it, and not the selection made by the authors of the journals to which they will submit their articles; and the solicitation of articles may decrease the rejection rate, which would therefore not represent a true reflection of the quality of the material.¹⁷⁸These arguments are also relevant vis-à-vis South African law journals. In addition, one should not lose sight of the fact that a journal with a greater output of volumes per annum may receive more submissions¹⁷⁹ and that the subjective nature of peer review may influence the rejection rate.¹⁸⁰

I refer yet again to the ASSAf Report on Grouped Peer Review of Scholarly Journals in Law as the only available information on this topic. The Report reflects the number of rejections by most of the law journals, although the data could not be independently verified.

 

Table 3

 

The ranking according to the rejection rate result in Merc LJ being ranked first, AHRLJ second and JJS third. SALJ and LDD would be ranked fourth and fifth respectively.

In conclusion, apart from the data's being unverifiable and subjective, it does not meet the Perry minimum requirement of quality-sensitivity. The data are nonetheless included under 5 hereunder.

4.6 Ranking based on library and electronic database usage

This type of ranking is based on the library usage of journals - either through actual physical viewing of the journal in the library or online by way of electronic downloads.²⁰⁰ This method cannot be regarded as very effective in South Africa, as there are no national or other verifiable statistics available for the physical access to journals. Research in the US has shown that the data would be university-, region- or database-specific, depending on the interest and focus area of academics at a specific university.²⁰¹ The results would be demand-driven and dependent on what the "hot" topics²⁰² are, and could ultimately be manipulated by an in-house academic who requires his/her students to use specific materials.²⁰³ Online databases are not very reliable sources of statistics, as companies are competitive and secretive.²⁰⁴When accessing information from outside the system, there is always the possibility of a manipulated result because, without individual logon identification, multiple views or author views may not be highlighted.²⁰⁵ The frequency of downloading is also not a true reflection of the frequency of actual use²⁰⁶ and may not be indicative of the quality or impact of the article.²⁰⁷

As it was not possible to obtain any data of electronic usage per journal for the period, no rating could be made on this basis.

4.7 Ranking based on journal citations

One of the standard ways of determining the quality of a journal for ranking purposes in various disciplines is to look at the number of citations it receives:²⁰⁸

The overall impact of a given journal is equivalent to the aggregate impact of all articles that were published in it during the relevant time period. Consequently, the citation frequency of all articles published in a certain journal in a given set of subsequent texts constitutes a rough measure of that journal's impact on the professional discourse within a specific circle, although ... it needs to be adjusted to serve as an approximate measure of the academic value of this journal.²⁰⁹

The argument is that the more often the journal is cited, the higher the quality and the more influential the journal.²¹⁰ The citation frequency of law journals can be seen as an indication of important scholarship, although it may not always be the case.²¹¹ Although citation is regarded as a more objective measure of quality,²¹² it is not without its problems, such as technical difficulties in the "citing-cited matching process"²¹³ underlying the bias²¹⁴ and limitations that underpin the system.²¹⁵

The quality of each journal may vary and an article published in a frequently cited journal may not necessarily be of a high quality itself.²¹⁶ In addition, this ranking system is "overly sensitive to the presence of one or two remarkable articles" getting numerous citations.²¹⁷ In addition, editors could be tempted to publish articles that they know would be controversial and thus more likely to be cited²¹⁸ or to publish articles by more famous authors likely to be cited and not necessarily because of the quality of the article itself.²¹⁹ A journal may also be cited for the "wrong" reason.²²⁰ Korobkin inter alia names a few instances where the citing of a journal "provides little or no evidence of scholarly value." For instance, the article may contain an unoriginal argument that is easy to access; the source may be a friend or ally of the author; the author may wish to impress the source;²²¹ the source may be incorrect or present an opposite view from that of the author.²²²Other negatives of the citation system have been noted by Tome and Lipu:²²³ Scholars working in the same field tend to cite one another, resulting in "strong geographical and regional tendencies"²²⁴ and favouring English as the medium.²²⁵

Other problems are that the citations may include negative citations, self-citation, collegial citation or selective citation without necessarily reflecting the quality of the journal.²²⁶

Citations are inherently predisposed towards older journals,²²⁷ although the number of citations may decline over the years.²²⁸ The extent of the size of the publication may impact on the number of citations because of an increased word-count, even though the material published may be more mediocre.²²⁹ Articles available online in their entirety are more likely to be cited than those available in hard copy only.²³⁰

Impact factors also differ for each subject field, document type and journal size.²³¹ The citation conventions in the legal field vary: articles are comprehensively referenced and every citation may not be of equal scholarly value.²³² In addition, there may be inconsistencies in the citations of general law journals vis-à-vis specialist law journals.²³³ Certain areas of the law could be regarded as more interesting or popular at any given time and this could impact positively on the number of citations.²³⁴

Low impact indices are not unusual in law. The London School of Economics Maximising the impact of academic research noted that the average h-index score in law is 1.25 as opposed to 4.83 in Economics and 5.04 in Geography.²³⁵

Research in the US has highlighted that various citation indices could provide inconsistent results.²³⁶

Those concerned with citations²³⁷ in different disciplines have attempted to nuance or adapt their citation systems accordingly to mitigate the negatives.²³⁸ Eisenberg and Wells argued that the use of ISI is inappropriate in the legal field as it is under-represented and slanted towards the sciences, and may be detrimental to scholarship.²³⁹

The most popular social sciences index in South Africa is the Web of Science SSCI Social Science Citation Index, but a certain preference for country and discipline is present.²⁴⁰ As only the SAJHR was included in this list during the relevant period, this index is disregarded in this article.

The Washington and Lee citation databases count only citations by US Law journal articles in the Westlaw Database.²⁴¹ As a result, the citation counts of smaller legal jurisdictions such as South Africa, which tend to be jurisdiction-specific, are not well-represented.²⁴²

In this article the existing Washington and Lee Law Journal Ranking Project²⁴³ data will be used as an example. It has been argued that its so-called Combined Factor balances the raw citations with the impact factor to give a more realistic measure of influence.²⁴⁴ This system considers the wider readership of legal materials such as practitioners, judges, academics and policy makers.²⁴⁵ It should be noted that compared to ISI it has been described as "surprisingly inconsistent, with no statistically significant correlation".²⁴⁶

The rankings described in this article are based on citations found in the full-text Westlaw journals database of journals published in the preceding eight years.²⁴⁷ For our purpose, it is important to note that fourteen of the relevant journals appear on the list. By 2014 the South African journals were ranked as follows:

 

Table 4

 

In conclusion, the citation system does not meet the Perry minimum requirements: comprehensive South African data is unavailable and it is not quality-sensitive. In addition, it may not be productive to rank South African law journals and pit the editors against each other in such a small pool of legal academia:

The notion of competition among editors has its disturbing side, in that behaviour modification to manipulate a journal's citation ranking may not be a positive feature. Without improving the intellectual quality of a general law review it is possible to increase a journal's citation ranking by excluding less popular topics . ²⁵¹

4.8 Court citations

The first legal citation index dates back to the Shepard's Citations of 1873, wherein the index listed court cases and publications referred to.²⁵²Because of the nature of legal research, it was argued that it may be possible to rank the journals according to the number of citations of the journal by the courts.²⁵³ As such, it may be a measure of the value of legal academic scholarship,²⁵⁴ especially if the aim of legal research is seen to be legal change and the development of the law.

The question, however, is which courts should be used for the purposes of this research. In the US "there are at least two paradigmatic versions of the method":²⁵⁵ one, the citations in the appellate courts, which would reflect the influence the journal exercises on the understanding and development of the law and especially the resolving of practical legal problems - which shows creativity and innovation;²⁵⁶ and two, the frequency of citations in the trial courts because of the nature and variety of work done in these courts - reflecting the analysis of existing law.²⁵⁷ The court citation may also be negative.²⁵⁸

A rudimentary attempt was made to ascertain the citation rate of journals by using the Juta, Lexis Nexis and Saflii judgment databases²⁵⁹ in an online search. It included all the online facilities in 2014. The search was not limited to a specific number of years, resulting in a possible slant towards older journals.²⁶⁰ This was counteracted by considering the number of years a journal has been in existence. The citations of certain journals could not be calculated because the title picked up references to the word in other contexts as well, such as Obiter, De Jure and PER. These three had to be ignored for the purposes of this exercise. These searches did not make provision for any possible duplication of reported cases in the various databases.

 

Table 5

 

Although the data obtained are flawed and incomplete, if this system is used the ILJ would be ranked first, with SALJ second, THRHR third and TSAR and SAJHR fourth and fifth respectively. The high ranking of the ILJ is not unexpected as it includes labour judgments, and possibly because the databases searched include a high number of CCMA awards and where labour law sources are cited a slanting towards more citations from the ILJ is inevitable.

Because of the unreliability of the information, this cannot be a proper basis for an official ranking system. It does not meet the Perry minimum requirements of reliability and quality-sensitivity. The data are nonetheless included under 5 hereunder.

4.9 Ranking based on the outputs chosen by successful NRF-rating applicants as their best²⁶¹

In terms of the NRF rating application requirements, applicants who apply for rating must submit what they regard as their five best publications of the past eight years for peer-evaluation.²⁶² These submissions may include locally or internationally published books, chapters in books, or journal articles.

When one looks at the NRF-rating data over a period of six years, 20092014, the raw data reveal the frequencies with which the journals relevant to this study were selected by successful NRF-rating applicants: SALJ (40); TSAR (32); Merc LJ (31); THRHR (29); Stell LR (26); SAJHR (24); PER (20); CILSA (17); SA Public Law (16); De Jure (12); Obiter (9); SACJ (8); AHRLJ (8); Acta Juridica (7); Fundamina (6); LDD (6); ILJ (5); JJS (5); SAYIL (5) and Speculum Juris (5). It is not surprising to note that specialised journals such as Fundamina, SACJ and the ILJ were selected less frequently than the generalist law journals, keeping in mind that there are fewer specialist academics in South Africa.

When one breaks down the data further into rating categories, the information becomes more nuanced. The A-rated legal academics, who are in the minority, displayed a preference for submitting internationally published books and international journal articles for the NRF review process.²⁶³ B-rated academics, in contrast, submitted a variety of materials. Apart from a modicum of conference proceedings and reports, there was an almost equal distribution on average of South African journal articles, international articles and chapters in books.²⁶⁴ There was also a noteworthy number of book submissions in this class (15%).²⁶⁵ The C-rated academics concentrated on submissions published in South African journals - at all levels.²⁶⁶ Although chapters in books and international journals also played a role, these were substantially fewer than the South African journals.²⁶⁷The submission of chapters and books increases as the rating becomes higher.²⁶⁸ P- and Y1-rated scientists generally included more international journals in their submission lists.²⁶⁹

From the above it may be inferred that there seems to be a trend that the higher an academic's NRF-rating, the less significant the role South African law journals play in the assessment of career-defining publications. This does not detract, however, from the submission above, namely that the esteem in which a journal is held is reflected in the number of highly-rated academics quoting it.

An additional caveat should be added to this analysis, namely that fewer than twenty per cent of law academics in South Africa were rated in 2014.²⁷⁰A reason for this could be inter alia a pre-conception that applicants will be given a C-rating, irrespective of their real standing in the field nationally, as many fields in law do not lend themselves to publication in international journals. Whatever the merit of the argument, rating is not compulsory at most universities, and many legal academics are not interested in applying for a rating. And, going forward, recent developments at the NRF to reduce incentive funding may suggest that very few academics will apply for rating in future.

 

Table 6

 

If this system were to be used as a guide, the ranking of the top five journals would be as follows: SALJ, TSAR, Merc LJ, THRHR and Stell LR. However, apart from the reasons given above, the data do not meet the Perry minimum requirements for an overall quality-sensitive measurement. The data are nevertheless included under 5 hereunder.

 

5 Consolidated data

The evidence above that none of the systems discussed is flawless, trustworthy and quality-sensitive, but it also highlights the problems related to ranking systems in general as well as specific ranking systems in particular.

The question can rightly be asked whether the various systems should not be weighted, as some of the measures could be regarded as inherently more important than others. The choice of systems and weightings could be in the interests of credibility and trustworthiness.²⁷¹ However, any exclusions or weightings would remain purely subjective.²⁷² For the purposes of illustration, all the available data are included.

To consider the consistency in the outcome of the various systems of ranking South African law journals, the spread of the outcomes per journal and the average of all the imperfect results are shown.

 

Table 7

 

So what does all of this mean? The spread per journal of the outcome of the various ranking systems used shows that there is very little consistency in the outcome of the data.²⁷³ The ranking of a journal will thus depend largely on the ranking system(s) chosen, making the use of ranking as a quality and impact measure random, unfair, irrational and unreliable.

 

6 Conclusion

The aim of this article is to consider whether a journal ranking system would be useful to assist the DHET in determining quality and impact in the legal field. None of the journal ranking systems discussed meets the Perry minimum requirements for a reliable and trustworthy system. None can claim to be objective, with quality-sensitive criteria free from bias. In addition the data are difficult to retrieve, unverifiable and possibly open to manipulation. For South Africa as a small jurisdiction the negatives clearly outweigh any possible benefits of the use of a data ranking system.²⁷⁴

The short response to the proposal of a ranking system for South African law journals would be negative, making consultation with the DHET urgent, especially since the decision is directly linked to government funding. Not acting could result in an inappropriate and prejudicial system's being imposed on the legal fraternity, especially in the light of the tendency to use journal citations as a fall-back position.

More research and debate are required. These may indicate that expert panels should be appointed to determine quality and impact, which may or may not result in the giving of an expanded brief to ASSAf. New developments on the electronic front, particularly around databases, and a usable, verifiable and trustworthy citation system may in future be developed for law to create indicators of quality and impact.

The quest for a solution should be part of a serious national debate on substantive quality and impact and the development of a matrix to give effect to the complexity of the legal discipline, as suggested by Van Gestel.²⁷⁵ The South African Law Deans' Association may have a role to play in this regard.

Whatever the final decision, there are deep concerns currently about a law journal ranking system for South Africa and I can only agree with the sentiment eloquently expressed by Svantesson and White:

[N]one of the ranking methodologies ... has the sort of scientific rigour that can be expected from an exercise with such profound implications. ... [T]he results... are bound to be influenced by intentional and/or unintentional biases. Consequently, there may be a lack of 'procedural fairness' as to who will be the winners and who will be the losers under any particular ranking scheme. Finally, we are concerned about the consequences that inevitably will flow from research ranking. Those consequences may be particularly detrimental for a small jurisdiction . ²⁷⁶

[In addition ...] we have little faith that existing research ranking schemes are capable of producing accurate and scientifically valid ranking of research. At the same time, we are convinced that attempts to rank research carry with them serious detrimental effects on research diversity, and potentially, research quality. This is particularly so where research funding is based on the outcome of the ranking exercise.²⁷⁷

In addition, any ranking system could probably be open to challenge should it have a detrimental effect for any given journal. Administrative justice requires, at least, rationality, and it would not appear that the implementation of any of the systems discussed here and found to be wanting.

 

Bibliography

Literature

Altbach PG "Academic Freedom: International Realities and Challenges" 2001 Higher Education 205-219        [ Links ]

ASSAf Report on a Strategic Approach to Research Publishing in South Africa (ASSAf Pretoria 2006)        [ Links ]

ASSAf Report on Grouped Peer Review of Scholarly Journals in Law and Related Legal Fields (ASSAf Pretoria 2014)        [ Links ]

Brophy AL "Law [Review]'s Empire: The Assessment of Law Reviews and Trends in Legal Scholarship" 2006 Connecticut Law Review 101-107        [ Links ]

Budden AE et al "Double-Blind Review Favours Increased Representation of Female Authors" 2008 Trends Ecol Evol 4-6        [ Links ]

Campbell K, Goodacre A and Little G "Ranking of UK Law Journals. An Analysis of the Research Assessment Exercise (RAE) 2001 Submissions and Results" 2006 J L & Soc'y 335-363        [ Links ]

Carnelley M "Identifying Predatory Open-Access Academic Journal Publishers, in Light of the South African Department of Higher Education and Training's Decision to Retrospectively De-accredit Certain Journals" 2015 Obiter 519-538        [ Links ]

Crespi GS "Ranking International and Comparative Law Journals: A Survey of Expert Opinion" 1997 Int'l Law 869-886        [ Links ]

Crespi GS "Ranking the Environmental Law, Natural Resources Law and Law Use Planning Journals: A Survey of Expert Opinion" 1998 Wm & Mary Envtl L & Pol'y Rev 273-297        [ Links ]

Crespi GS "Ranking Specialised Law Reviews: A Methodological Critique" 1999 Fla St U L Rev 837-849        [ Links ]

Currie RR and Pandler GS "Survey. Finance Journal Rankings and Tiers: An Active Scholarly Assessment Methodology" 2011 J Bank Finance 7-20        [ Links ]

Curtis B "The Performance-Based Research Fund: Research Assessment and Funding in New Zealand" 2008 Globalisation, Societies and Education 179-194        [ Links ]

Datt K, Tran A and Tran-Nam B "Ranking Tax Journals - The Way Forward" 2009 ATF 341-369        [ Links ]

Department of Higher Education and Training Research Output Policy (DHET Pretoria 2015)        [ Links ]

Department of Higher Education and Training White Paper for Post-School Education and Training: Building an Expanded, Effective and Integrated Post-School System (DHET Pretoria 2013)        [ Links ]

Department of Higher Education and Training Directorate: University Policy and Development Support Communiqué 2 of 2015 (DHET Pretoria August 2015)        [ Links ]

Doyle J "Ranking Legal Periodicals and some other Numeric Uses of the Westlaw and Lexis Periodical Databases" 2004 Leg Ref Serv Q 1-396        [ Links ]

Editorial 2012 "Impact Factor - The Food is Bad and What's More There is not Enough of It: EJIL - The Beginning of the Existential Debate; Masthead Changes; In this Issue" 2012 EJIL 607-612        [ Links ]

Editorial 2015 "Nature Journals Offer Double-Blind Review" 2015 Nature 274        [ Links ]

Eisenberg T and Wells MT "Ranking Law Journals and the Limits of Journal Citation Reports" 2014 Economic Inquiry 1301-1314        [ Links ]

European Union Expert Group on Assessment of University-based Research Assessing University-based Research (Publications Office of the EU Luxembourg 2010)        [ Links ]

Fuld SH "A Judge Looks at the Law Review" 1953 NYU L Rev 915-921        [ Links ]

George TE and Guthrie C "An Empirical Evaluation of Specialised Law Reviews" 1999 Fla St U L Rev 813-836        [ Links ]

George TE and Guthrie C "In Defence of Author Prominence: A Reply to Crespi and Korobkin" 1999 Fla St U L Rev 877-896        [ Links ]

Given LM, Kelly W and Wilson R "Bracing for Impact: The Role of Information Science in Supporting Societal Research Impact" in Proceedings of the 78th ASIS&T Annual Meeting: Information Science (610 November 2015 St Louis, MO) 1 -10        [ Links ]

Grossman JL "Feminist Law Journals and the Ranking Conundrum" 2003 Colum J Gender & L 522-531        [ Links ]

Higher Education Funding Council for England Research Excellence Framework 2014: Overview Report by Main Panel C and Sub-panels 16 to 26 (HEFCE Bristol 2015)        [ Links ]

Higher Education Funding Council for England Research Assessment Exercise 2001 Law Panel General Overview (HEFCE Bristol 2001)        [ Links ]

Jarvis RM and Coleman PG "Ranking Law Reviews: An Empirical Analysis Based on Author Prominence" 1997 Arizona L Rev 15-24        [ Links ]

Jarvis RM and Coleman PG "Ranking Law Reviews by Author Prominence - Ten Years Later" 2007 L Libr J 573-588        [ Links ]

Kahn M "A Bibliometric Analysis of South Africa's Scientific Outputs - Some Trends and Implications" 2011 107(1/2) SAJS 1-6        [ Links ]

Korobkin R "Ranking Journals: Some Thoughts on Theory and Methodology" 1999 Fla St U L Rev 851 -876        [ Links ]

Liefner I "Funding, Resource Allocation and Performance in Higher Education System" 2003 Higher Education 469-489        [ Links ]

Mallard G, Lamont M and Guetzkow J "Fairness as Appropriateness. Negotiating Epistemological Differences in Peer Review" 2009 Sci Technol Hum Values 573-606        [ Links ]

Marashi S-A et al "Can Scientific Journals be Classified Based on their 'Citation Profiles'?" 2015 SAJS 1-3        [ Links ]

Maru O "Measuring the Impact of Legal Periodicals" 1976 Am B Found Res J 227-249        [ Links ]

McWhirter DA The Legal 100: A Ranking of Individuals Who Have Most Influenced the Law (Birch Lane Press Secaucus 1998)        [ Links ]

Ministry of Education Policy and Procedures for Measurement of Research Output of Public Higher Education Institutions (Department of Education Pretoria 2003)        [ Links ]

Moed HF, Van Leeuwen Th N and Reedijk J "A New Classification System to Describe the Ageing of Scientific Journals and Their Impact Factors" 1998 J Doc 387-419        [ Links ]

Mouton J A Bibliometric Analysis of the State of Research at UKZN. Final Report (Centre for Research on Evaluation, Science and Technology Stellenbosch 2015)        [ Links ]

Mouton J and Valentine A "The Extent of the South African Authored Articles in Predatory Journals" 2017 SAJS 1-9        [ Links ]

NRF/RISA Evaluation and Rating. Guidelines and Instructions for Completing the Application for Evaluation and Rating via the NRF Online Submission System (NRF/RISA Pretoria 2015)        [ Links ]

Online Computer Library Centre A Comparative Review of Research Assessment Regimes in Five Countries and the Role of Libraries in the Research Assessment Process. A Pilot Study (OCLC Truro 2009)        [ Links ]

Osterloh M and Frey BS Research Governance in Academia: Are There Alternatives to Academic Rankings? (Institute for Empirical Research in Economics Zurich 2009)        [ Links ]

Perry R "The Relative Value of American Law Reviews: A Critical Appraisal of Ranking Methods" 2006 Va J Law Technol 1 -40        [ Links ]

Pouris AEM and Pouris A "An Assessment of South Africa's Research Journals: Impact Factors, Eigenfactors and Structure of Editorial Boards" 2015 111(3/4) SAJS 1-8        [ Links ]

Pritchard RMO "Academic Freedom and Autonomy in the United Kingdom and Germany" 1998 Minerva 101-124        [ Links ]

Ramsay I and Stapledon GP "A Citation Analysis of Australian Law Journals" 1997 MULR 676-692        [ Links ]

Smyth R "Academic Writing and the Courts: A Qualitative Study of the Influence of Legal and Non-Legal Periodicals in the High Court" 1998 U Tas LR 164-182        [ Links ]

Smyth R "Who Publishes in Australia's Top Law Journals?" 2012 UNSWLJ 201-245        [ Links ]

Steinbuch R "On the Leiter Side: Developing a Universal Assessment Tool for Measuring Scholarly Output by Law Professors and Ranking Law Schools" 2009 Loy LA L Rev 87-123        [ Links ]

Svantesson DJB "International Ranking of Law Journals - Can it be done and at What Costs?" 2009 Legal Studies 678-691        [ Links ]

Svantesson DJB and White P "Entering an Era of Research Ranking - Will Innovation and Diversity Survive?" 2009 Bond LR 173-191        [ Links ]

Van Gestel R "Sense and Non-Sense of a European Ranking of Law Schools and Law Journals" 2015 Legal Studies 165-185        [ Links ]

Van Gestel R and Vrancken J "Assessing Legal Research: Sense and Nonsense of Peer Review Versus Bibliometrics and the Need for a European Approach" 2011 GLJ 901 -929        [ Links ]

University of the Witwatersrand Faculty of Commerce, Law and Management Faculty Research Committee Research Publication Incentive (RINC) Policy (WITS Johannesburg circa 2015)        [ Links ]

Yoon AH "Editorial Bias in Legal Academia" 2013 JLA 309-339        [ Links ]

Internet sources

Akademie van Wetenschappen 2014 Standard Evaluation Protocol 2015 -2021 http://www.knaw.nl/nl/actueel/nieuws/wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek accessed 10 October 2017        [ Links ]

Elsevier date unknown About Scopus https://www.elsevier.com/solutions/scopus accessed 10 October 2017        [ Links ]

International Bibliography of the Social Sciences 2013 IBSS Editorial Policies and Principles http://media2.proquest.com/documents/IBSS+Editorial+Policies+and+Principles.pdf accessed 10 October 2017        [ Links ]

London School of Economics 2010 Maximising the Impact of Academic Research http://blogs.lse.ac.uk/impactofsocialsciences/files/2010/11/chart3.jpg accessed 10 October 2017        [ Links ]

Norwegian Register for Scientific Journals, Series and Publishers date unknown Criteria for Inclusion of New Scientific Publication Channels https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?request_locale=en accessed 10 October 2017        [ Links ]

New Zealand Performance-Based Research Fund date unknown Funding and Performance http://www.tec.govt.nz/Funding/funding-and-performance/ accessed 10 October 2017        [ Links ]

Scientific Electronic Library Online date unknown 'SciELO SA Criteria': Criteria, Policy and Procedures for Admission and Permanence of Scientific Journals in the South African Collection of SciELO http://www.scielo.org.za/avaliacao/avaliacao_en.htm accessed 10 October 2017        [ Links ]

Testa J 2016 Journal Selection Process http://wokinfo.com/essays/journal-selection-process/ accessed 10 October 2017        [ Links ]

Tome L and Lipu S 2004 Research and Development Discussion Paper #6: Indicators of Journal Quality http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.897.6809&rep=rep1&type=pdf accessed 19 December 2017        [ Links ]

UKZN Research Office date unknown DHET Accredited Journals http://research.ukzn.ac.za/DoHETAccreditedJournals.aspx accessed 10 October 2017        [ Links ]

Washington and Lee University 2009-2014 Law Journals: Submissions and Ranking, 2009 - 2014 http://lawlib.wlu.edu/LJ/ accessed 10 October 2017        [ Links ]

University of the Witwatersrand date unknown South African Journal on Human Rights https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf accessed 10 October 2017        [ Links ]

LIST OF ABBREVIATIONS

AHRLJ African Human Rights Law Journal

Am B Found Res J American Bar Foundation Research Journal

Am Sociol Rev American Sociological Review

Annual Survey Annual Survey of the South African Law

Arizona L Rev Arizona Law Review

ASSAf Academy of Science of South Africa

ATF Australian Tax Forum

Bond LR Bond Law Review

CARSA Child Abuse Research in South Africa

CCMA Commission for Conciliation, Mediation and Arbitration

CESifo Centre for Economic Studies Publishing Group

Chi-Kent L Rev Chicago-Kent Law Review

CILSA Comparative and International Law Journal of Southern Africa

Colum J Gender & L Columbia Journal of Gender and the Law

Conn L Rev Connecticut Law Review

DHET Department of Higher Education and Training

EJIL European Journal of International Law

Fla St U L Rev Florida State University Law Review

GLJ German Law Journal

Haifa L Rev Haifa Law Review

HEFCE Higher Education Funding Council for England

IBSS International Bibliography of the Social Sciences

ILJ Industrial Law Journal

Ind LJ Indiana Law Journal

Int'l J Legal Info International Journal of Legal Information

Int'l Law The International Lawyer

ISI Thomson Reuters Web of Science (formerly ISI Web of Knowledge)

IT Information Technology

J Bank Finance Journal of Banking and Finance

J Doc Journal of Documentation

J L & Soc'y Journal of Law and Society

JJS Journal of Juridical Science

JLA Journal of Legal Analysis

L Libr J Law Library Journal

LDD Law, Democracy and Development

Leg Ref Serv Q Legal Reference Services Quarterly

Loy LA L Rev Loyola of Los Angeles Law Review

Merc LJ South African Mercantile Law Journal

MULR Melbourne University Law Review

NMMU Nelson Mandela Metropolitan University

NRF National Research Foundation

NSD Norwegian Register for Scientific Journals, Series and Publishers

NYU L Rev New York University Law Review

NZ New Zealand

OCLC Online Computer Library Centre

PBRF New Zealand Performance-Based Research Fund

PER Potchefstroom Elektroniese Regstydskrif

RISA Research and Innovation Support and Advancement

SACJ South African Journal of Criminal Justice

SAJELP South African Journal of Environmental Law and Policy

SAJHR South African Journal of Human Rights

SAJS South African Journal of Science

SALJ South African Law Journal

SAYIL South African Yearbook of International Law

Sci Technol Hum Values Science, Technology and Human Values

SciELO Scientific Electronic Library Online

Stell LR Stellenbosch Law Review

THRHR Tydskrif vir Hedendaags Romeins-Hollandse Reg

Trends Ecol Evol Trends in Ecology and Evolution

TSAR Tydskrif vir die Suid-Afrikaanse Reg

U Tas LR University of Tasmania Law Review

UK United Kingdom

UKZN University of KwaZulu-Natal

UNSWLJ University of New South Wales Law Journal

US / USA United States of Am erica

Va J Law Technol Virginia Journal of Law and Technology

WITS University of the Witwatersrand

Wm & Mary Envtl L & Pol'y William and Mary Environmental Law and Rev Policy Review

 

 

Date of submission: 8 November 2017
Date published: 19 January 2018

 

 

Editor Prof C Rautenbach
* Marita Carnelley. BA LLB LLM PhD (Amsterdam). Professor, North-West University, South Africa. Email: marita.carnelley@nwu.ac.za. The author wishes to thank the National Research Foundation (NRF) for providing the data about the NRF rating submissions and Ms Erica Wille for her assistance with the sourcing of the information about the prominence of authors.
1 Svantesson and White 2009 Bond LR 173.
2 OCLC Research Assessment Regimes 5 discusses inter alia the quality evaluation processes used in the Netherlands, Ireland, the UK, Denmark and Australia. Evaluating research is regarded as a complex process with no single European measurement having been accepted as meeting all the requirements of quality determination, as well as the need for accountability and transparency (European Union Expert Group on Assessment of University-based Research Assessing University-based Research) 9.
3 Mouton and Valentine 2017 SAJS 1.
4 Given, Kelly and Willson "Bracing for Impact" 1; Currie and Pandler 2011 J Bank Finance 7; Liefner 2003 Higher Education 486.
5 Curtis 2008 Globalisation, Societies and Education 180; Osterloh and Frey Research Governance in Academia.
6 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 9.
7 Altbach 2001 Higher Education 216; Pritchard 1998 Minerva 119.
8 OCLC Research Assessment Regimes 8.
9 Ministry of Education Policy and Procedures.
10 Ministry of Education Policy and Procedures 4: "While the policy recognises different types of research output for purposes of subsidy, it does not support differentiation within types of output." South African public universities are rewarded by the DHET for the number of accredited publications their academic staff publishes (Mouton Bibliometric Analysis 9).
11 Peer-review is not necessarily double-blind peer-review (Budden et al 2008 Trends Ecol Evol 4; Editorial 2015 Nature 274).
12 This is applicable not only to the field of law, but also to other disciplines. See the discussion of Kahn 2011 SAJS 2-5 regarding the increased number of publications in the sciences as a result of the increased rewards.
13 ASSAf Report on a Strategic Approach to Research Publishing 5.
14 Van Gestel 2015 Legal Studies 170.
15 DHET White Paper 4.4.
16 DHET Research Output Policy 2.1.
17 DHET Research Output Policy 2.2.
18 Given, Kelly and Wilson notes the difficulty with the determination of what "impact" is, as it is fraught with difficulties: how does one measure the level of impact, what should the place of impact be - society, academia or other stakeholders - and who should decide these issues (Given, Kelly and Wilson "Bracing for Impact" 4)? The stakeholders and users of the research are various and have diverse needs. They include policy makers, government agencies, universities, research organisations, graduate schools, employers, civil society, the courts, the judiciary and the profession (European Union Expert Group on Assessment of University-based Research Assessing University-based Research 26). "Measuring impact and benefits is an emerging methodology and additional work needs to be done to identify appropriate indicators, but also to develop mechanisms to collect accurate and comparable data" (European Union Expert Group on Assessment of University-based Research Assessing University-based Research 41).
19 Korobkin 1999 Fla St U L Rev 860.
20 See the discussion hereunder. ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 4.
21 DHET Research Output Policy 2.4; Ministry of Education Policy and Procedures 34.
22 ASSAf Report on Strategic Approach to Research Publishing 69.
23 The lists of journals approved for subsidy include one, the DHET list of approved South African journals; two, the International Bibliography of the Social Sciences (IBSS); and three, the Thomson Reuters Web of Science (formerly ISI Web of Knowledge) (ISI). From 2016, it was expanded to include three more: four, SciELO; five, Scopus and six, the Norwegian Register for Scientific Journals, Series and Publishers (NSD) (DHET Directorate Communiqué 1). For a copy of these accredited lists, see for example UKZN Research Office date unknown http://research.ukzn.ac.za/DoHETAccreditedJournals.aspx.
24 See in general Mouton and Valentine 2017 SAJS 2; Carnelley 2015 Obiter 519-538.
25 ASSAf Report on Strategic Approach to Research Publishing 29.
26 The increase of specialised journals may also have exacerbated this problem of a lack of an experienced pool of academics per discipline which is in line with research in Canada and the USA that noted that law reviews generally publish articles from their own faculty rather than outsiders - even if they are cited less frequently. This is indicative of what is referred to as "editorial bias in legal academia" (Yoon 2013 JLA 336). In South Africa, DHET has attempted to ameliorate this problem by requiring from 2016 that, to qualify for subsidy, in-house journals should not publish a volume where at least 75% of the submissions emanate from multiple external institutions (DHET Research Output Policy Clause 5.10(c)).
27 The same is true for other jurisdictions (Van Gestel and Vrancken 2011 GLJ 905). The OCLC Report confirms that in the absence of evidence to government that their research funding results "in good value of quality and impact", it is difficult to objectively defend research budgets (OCLC Research Assessment Regimes 8).
28 The assessment was and is done through ASSAf in terms of the Ministry of Education Policy and Procedures (until 2015), and from 2016 in terms of the DHET Research Output Policy 22.2.
29 ASSAf Report on Strategic Approach to Research Publishing 7.
30 DHET Research Output Policy 2.4.
31 DHET Research Output Policy 2.7.
32 WITS for example allocates R10 000 for journal publications in DHET-accredited local journals, but R20 000 for journals in ISI or IBSS indexed journals (WITS Research Publication Incentive (RINC) Policy 2).
33 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 12. See the discussion hereunder.
34 OCLC Research Assessment Regimes 9; European Union Expert Group on Assessment of University-based Research Assessing University-based Research 117.
35 Van Gestel 2015 Legal Studies 177.
36 See the discussion hereunder.
37 Van Gestel 2015 Legal Studies 176. After the research impact pilot project completed in 2012, the Australian government considered a move towards the UK tradition (Given, Kelly and Wilson "Bracing for Impact" 1; also see the European Union Expert Group on Assessment of University-based Research Assessing University-based Research 84).
38 Van Gestel 2015 Legal Studies 176; Van Gestel and Vrancken 2011 GLJ 917; Eisenberg and Wells 2014 Economic Inquiry 1301.
39 Van Gestel and Vrancken 2011 GLJ 917.
40 OCLC Research Assessment Regimes 9; European Union Expert Group on Assessment of University-based Research Assessing University-based Research 117. See Akademie van Wetenschappen 2014 http://www.knaw.nl/nl/actueel/nieuws/wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek for the standard evaluation protocol 2015-2021. France and Finland also use a peer-review system (European Union Expert Group on Assessment of University-based Research Assessing University-based Research 96, 91), whilst Germany adds another dimension by supplementing the peer-review with a matrix and panels (European Union Expert Group on Assessment of University-based Research Assessing University-based Research 99).
41 Svantesson and White 2009 Bond LR 177. HEFCE Research Excellence Framework. The units submitted are assessed on a scale of 1 - 4. New Zealand has a system similar to that of the UK with a Performance-Based research fund (PBRF) to ensure that research excellence in universities is rewarded. The research performance of the institutions is measured and funding is based on performance (See PBRF date unknown http://www.tec.govt.nz/funding/funding-and-performance/; Curtis 2008 Globalisation, Societies and Education 179). The possibility of using a process of peer-review of individual articles in South Africa, such as that in use in the UK and NZ, is excluded from this article and is a topic for research at some other time.
42 HEFCE Research Excellence Framework 75.
43 Perry 2006 Va J Law Technol 4. For a full discussion see Campbell, Goodacre and Little 2006 J L & Soc'y 335 onwards. Svantesson and White 2009 Bond LR 182 with reference to HEFCE Research Assessment Exercise 2008 Subject Overview Report (2009).
44 Svantesson and White 2009 Bond LR 182 quoting the HEFCE Research Assessment Exercise 2001 Law Panel General Overview (2001).
45 Perry 2006 Va J Law Technol 4; Korobkin 1999 Fla St U L Rev 853; Van Gestel 2015 Legal Studies 177; Brophy Connecticut Law Review 104. Brophy recognises other trends that may assist with increased quality: the increased online availability of legal materials, serious legal blogs and increased participation in law review decisionmaking (Brophy Connecticut Law Review 105-107).
46 Svantesson 2009 Legal Studies 680; Grossman 2003 Colum J Gender & L 526; Brophy Connecticut Law Review 103; Perry 2006 Va J Law Technol 27.
47 Smyth 2012 UNSWLJ 206.
48 Van Gestel 2015 Legal Studies 176; Grossman 2003 Colum J Gender & L 522.
49 Korobkin 1999 Fla St U L Rev 857.
50 Perry 2006 Va J Law Technol 4; Korobkin 1999 Fla St U L Rev 858.
51 Examples of this link are traditionally seen with the Stell LR, which is linked to the University of Stellenbosch, PER to the North West University, and TSAR to the University of Johannesburg, mainly because of the affiliations of the editors-in-chief.
52 Perry 2006 Va J Law Technol 5, Brophy Connecticut Law Review 103. There is no official Law School ranking system in South Africa.
53 Libraries may use rankings when prioritising the acquisition of material within a limited budget (Perry 2006 Va J Law Technol 6; Van Gestel 2015 Legal Studies 177).
54 Korobkin 1999 Fla St U L Rev 858; Perry 2006 Va J Law Technol 5; Van Gestel 2015 Legal Studies 177.
55 Van Gestel 2015 Legal Studies 176.
56 Van Gestel 2015 Legal Studies 177.
57 Korobkin 1999 Fla St U L Rev 859.
58 Perry 2006 Va J Law Technol 6-7. Datt, Tran and Tran-Nam 2009 ATF 364 argue that ranking methodologies should be objective, rigorous, comprehensive, valid, verifiable and practical with the outcome plausible and acceptable.
59 Ranking systems generally distinguish between general journals and specialised journals. In South Africa the number of specialised journals is limited and this distinction is not made in this article.
60 Perry 2006 Va J Law Technol 6.
61 Perry 2006 Va J Law Technol 6, Van Gestel 2015 Legal Studies 178.
62 Perry 2006 Va J Law Technol 6, 38.
63 Perry 2006 Va J Law Technol 7, Datt, Tran and Tran-Nam 2009 ATF 364.
64 Perry 2006 Va J Law Technol 7.
65 Perry 2006 Va J Law Technol 7.
66 Perry 2006 Va J Law Technol 7.
67 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 56-58; 13. The EU Report notes that it is good practice to combine peer-review, bibliometric information and self-evaluation (European Union Expert Group on Assessment of University-based Research Assessing University-based Research 13, 58). Also see Perry 2006 Va J Law Technol 38.
68 George and Guthrie 1999b Fla St U L Rev 880.
69 Perry 2006 Va J Law Technol 38.
70 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 17. This is borne out by the NRF data as set out in 4.9 hereunder.
71 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 17; European Union Expert Group on Assessment of University-based Research Assessing University-based Research 37.
72 The Report notes that in certain instances there may be criticism that a specific journal accepts too many submissions from a particular university, such as UNISA. However, that university may have many academics working in that area (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 17).
73 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 17.
74 Svantesson and White 2009 Bond LR 189.
75 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 17; European Union Expert Group on Assessment of University-based Research Assessing University-based Research 26.
76 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 17.
77 Van Gestel 2015 Legal Studies 179.
78 Svantesson and White 2009 Bond LR 189.
79 Svantesson and White 2009 Bond LR 189.
80 Maru 1976 Am B Found Res J 228 quoting Judge Stanley H Fuld's 1953 article "A Judge looks at the Law Review" (Fuld 1953 NYU L Rev 918).
81 The Perry framework is adopted for this article (Perry 2006 Va J Law Technol 7-37). Also see Currie and Pandler Journal of Banking and Finance 7.
82 The ASSAf Report on Grouped Peer Review of Scholarly Journals in Law evaluated 24 law journals and recommended that 23 remain accredited. The scope of the article is limited to journals publishing predominantly legal articles. The two multi-disciplinary journals, Acta Criminologica and CARSA, were excluded from this discussion. SAJELP was also excluded as it was found not to meet the ASSAf criteria by being out of date at that time, although this has subsequently been rectified.
83 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 15.
84 Perry 2006 Va J Law Technol 39.
85 It would be difficult to have rankings of speciality law journals in South Africa as there is only one (or maybe two at most) journal in each specific speciality area. For example, Fundamina is the only legal history journal. See in general the division used in ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 21.
86 Perry 2006 Va J Law Technol 7-8.
87 Perry 2006 Va J Law Technol 8, 10; Van Gestel 2015 Legal Studies 177.
88 Perry 2006 Va J Law Technol 8, 10; Van Gestel 2015 Legal Studies 176.
89 Van Gestel 2015 Legal Studies 176; Perry 2006 Va J Law Technol 8.
90 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 20. It is regarded as self-perpetuating, as these systems are susceptible to so-called "'gaming' which occurs when respondents deliberately downgrade competitors or upgrade their assessments to influence the outcome".
91 Svantesson and White 2009 Bond LR 183. See in general Mallard, Lamont and Guetzkow 2009 Sci Technol Hum Values 599, noting the peer-review "roadblocks to distributional 'fairness caused by non-scientific influences such as politics, friendship networks, or common institutional positions'".
92 Perry 2006 Va J Law Technol 8.
93 DHET Research Output Policy 3.12.
94 The Forum of Editors of Law Journals of Southern Africa also subscribe to these best practices.
95 This quality assurance process is seen as a precursor to the identification of journal titles to be loaded onto the open access platform Scientific Electronic Library Online (SciELO)-South Africa. Journals of a sufficiently high quality will be included in this fully indexed, free online, multi-national platform featured on the Thomson-Reuters Web of Knowledge portal (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 7). Journals not already on SciELO, were invited to SciELO as "SciELO will become an important tool for the DHET to consider articles for subsidy purposes". (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 18). In the legal publishing scenario this is problematic, as some of the journals and/or their publishers indicated upfront that they are not interested in making use of the SciELO platform (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 18). The invitation to join the platform was nonetheless made, but fewer than a quarter of the law journals are listed on the SciELO platform (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 18).
96 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 12. The criteria include: the longevity of the journal; the number of original peer-reviewed papers published per year during the last five years; the number of manuscripts submitted and rejected out of hand or after peer review; the average length of published papers; the "author demography" of the papers submitted and published; the number and nature of the peer reviewers used per manuscript and per year, including the institutional and national/international spread; the quality and average length of the peer review reports; the average delay before publication; the frequency of publication; the professional stature and experience of the editor, his selection and length of service; the success in addressing the major issues in the field; the number and professional stature and experience of the editorial board members, the selection process, turnover and involvement; the mix from developed or developing countries; the editorial policy and guidelines; the conflict of interest policy; the annual errata published; value-adding features; the number of pages per issues; the peer review process and professional associations (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 12-13).
97 Business-related criteria include the frequency, regularity and punctuality of publication; the print-run, distribution patterns and the redundant stock; the production model and service providers; advertising and sponsorship; the subscription base, marketing and costs; e-subscriptions, accessibility and searchability; the format and the use of multimedia. In addition, the annual income and expenditure; the distribution to international destinations; and indexing in Thomson ISI and/or IBSS, or any other international database are considered (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 13).
98 These refer to the citation practice and the number of authors listed; ISI-type impact factors; whether reviews are a regular feature and if the articles are not in English, whether an English abstract is mandatory (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 13).
99 Perry 2006 Va J Law Technol 10.
100 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 22.
101 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 21.
102 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 59, 81 and 57 respectively.
103 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 49.
104 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 79.
105 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 64.
106 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 42.
107 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 54.
108 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 34.
109 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 38.
110 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 73.
111 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 40.
112 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 36.
113 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 75.
114 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 44.
115 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 25.
116 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 27.
117 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 51.
118 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 61.
119 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 51.
120 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 83.
121 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 32.
122 Crespi 1997 Int'l Law 869-886; Crespi 1998 Wm & Mary Envtl L & Pol'y Rev 273. Also see the discussion in Doyle 2004 Leg Ref Serv Q 6 and Perry 2006 Va J Law Technol 10.
123 Campbell, Goodacre and Little 2006 J L & Soc'y 340 onwards.
124 Datt, Tran and Tran-Nam 2009 ATF 351; Perry 2006 Va J Law Technol 10.
125 Datt, Tran and Tran-Nam 2009 ATF 350.
126 Datt, Tran and Tran-Nam 2009 ATF 351; Perry 2006 Va J Law Technol 11.
127 See in general McWhirter Legal 100.
128 Korobkin 1999 Fla St U L Rev 872; Perry 2006 Va J Law Technol 10. Their argument is as follows: here are potential problems when selecting participants for such a study. Few users would be familiar with all the journals, making the responses potentially random. Even if they were knowledgeable, their understanding of the criteria may vary resulting in the outcome being fraught with inconsistencies and even possible bias.
129 European Union Expert Group on Assessment of University-based Research Assessing University-based Research Report 18, quoting Einstein.
130 Van Gestel and Vrancken 2011 GLJ 915.
131 Van Gestel 2015 Legal Studies 172. The European Union Expert Group on Assessment of University-based Research Assessing University-based Research 37-38 gives the example of the difference between the available information on Roman law and on Information Technology (IT) law.
132 Van Gestel and Vrancken 2011 GLJ 920.
133 Osterloh and Frey Research Governance in Academia 8-9.
134 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 39-40.
135 Osterloh and Frey Research Governance in Academia 10. In the UK and Australia expert review remains important to ensure that legal academics are not treated unfairly, as bibliometric data are not readily available (Van Gestel and Vrancken 2011 GLJ 916).
136 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 39-40.
137 At NMMU the same subsidy is paid for all accredited publications, but for the awarding of the "Researcher of the Year", ISI and IBSS weigh more. As mentioned earlier, at WITS more credit is given to articles published in journals on the ISI or IBSS lists.
138 The details about the editorial policies and principles for inclusion in these lists can be accessed on their websites: IBSS at IBSS 2013 http://media2.proquest.com/documents/IBSS+Editorial+Policies+and+Principles.pdf; ISI at Testa 2016 http://wokinfo.com/essays/journal-selection-process/; and NSD at NSD date unknown https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?request_locale=en; SciELO and Scopus information can be found at SciELO date unknown http://www.scielo.org.za/avaliacao/avaliacao_en.htm and Elsevier dateunknown https://www.elsevier.com/solutions/scopus.
139 IBSS requires submissions to be "ideally peer-reviewed" (IBSS 2013 http://media2.proquest.com/documents/IBSS+Editorial+Policies+and+Principles.pdf), although NSD's requirements are more stringent: "a system of quality assurance, generally a double peer-review system" (NSD date unknown https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?Requestlocale=en).
140 ISI, IBSS, NSD and Scopus.
141 All three journals are listed in IBSS. CILSA is also in the NSD list and AHRLJ and PER are in SciELO.
142 Van Gestel 2015 Legal Studies 177; Perry 2006 Va J Law Technol 6.
143 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 29 and 35 respectively.
144 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 23.
145 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 82.
146 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 60.
147 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 46.
148 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 63.
149 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 37 and 84 respectively.
150 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 76 and 80 respectively.
151 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 49.
152 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 56.
153 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 73.
154 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 41 and 43 respectively.
155 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 66.
156 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 26.
157 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 39.
158 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 53 and 32 respectively.
159 George and Guthrie 1999b Fla St U L Rev 881.
160 Jarvis and Coleman 1997 Arizona L Rev 15-24. Also see their follow-up article Jarvis and Coleman 2007 L Libr J 573-588.
161 George and Guthrie 1999a Fla St U L Rev 813-836; George and Guthrie 1999b Fla St U L Rev 877-896.
162 Specialist law reviews in the US are generally peer-reviewed and faculty edited rather than the generalist law reviews that are edited by graduate students (George and Guthrie 1999a Fla St U L Rev 819). Korobkin 1999 Fla St U L Rev 860. In South Africa none of the DHET law journals are edited by students.
163 George and Guthrie 1999a Fla St U L Rev 826.
164 Jarvis and Coleman 1997 Arizona L Rev 16.
165 Jarvis and Coleman 1997 Arizona L Rev 16.
166 Jarvis and Coleman 1997 Arizona L Rev 16 fn 7.
167 Perry 2006 Va J Law Technol 13, Crespi 1999 Fla St U L Rev 848.
168 Crespi 1999 Fla St U L Rev 837-849.
169 Perry 2006 Va J Law Technol 12-13.
170 Korobkin 1999 Fla St U L Rev 851-876.
171 Perry 2006 Va J Law Technol 13. As the peer-review process is theoretically blind, the acceptance of an article would depend not on the status of the author in South Africa but on the quality of the article.
172 Perry 2006 Va J Law Technol 13.
173 Svantesson 2009 Legal Studies 682.
174 Although this was considered, no difference was made between law faculties in South Africa as no official faculty ranking system exists. The seniority of judges, magistrates and attorneys was not considered as this information is not readily available. A and B-NRF ratings were considered to be of greater prominence, although C-rated academics were placed at their appointment level.
175 Jarvis and Coleman 1997 Arizona L Rev 17-18; Jarvis and Coleman 2007 L Libr J 575-578. Notwithstanding the limited number of specialised journals in South Africa, the assessment here is of all the journals, general and specialised, although the outcome of the rating could be impacted by this decision.
176 Perry 2006 Va J Law Technol 14-17.
177 Perry 2006 Va J Law Technol 14.
178 Perry 2006 Va J Law Technol 14-17. Annual Survey is an example of the phenomenon of solicitation.
179 Campbell, Goodacre and Little 2006 J L & Soc'y 343.
180 Sometimes the review process can be a lottery as a result of the subjective nature of peer review. For example, an unfortunate contributor may be faced with two highly critical reviewers in the double blind review process, and an article may be rejected which would otherwise have survived less exacting reviewers.
181 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 61.
182 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 45.
183 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 59.
184 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 74.
185 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 24.
186 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 63.
187 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 35.
188 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 37.
189 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 33.
190 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 54.
191 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 39.
192 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 27.
193 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 72.
194 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 47.
195 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 21.
196 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 82-83.
197 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 77.
198 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 31.
199 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 41.
200 It is accepted that physical counting is no longer reliable due to the electronic accessibility of journals (Perry 2006 Va J Law Technol 19).
201 Svantesson 2009 Legal Studies 683; Perry 2006 Va J Law Technol 19; Korobkin 1999 Fla St U L Rev 871; Doyle 2004 Leg Ref Serv Q 5 with reference to the 1997 University of Illinois study.
202 Datt, Tran and Tran-Nam 2009 ATF 352.
203 Perry 2006 Va J Law Technol 19, Datt, Tran and Tran-Nam 2009 ATF 352.
204 Perry 2006 Va J Law Technol 20; Svantesson 2009 Legal Studies 683.
205 Perry 2006 Va J Law Technol 20.
206 Perry 2006 Va J Law Technol 20; Datt, Tran and Tran-Nam 2009 ATF 352.
207 Perry 2006 Va J Law Technol 21.
208 See Pouris and Pouris 2015 SAJS 1-8 for a discussion of the South African scientific journals.
209 Perry 2006 Va J Law Technol 22; European Union Expert Group on Assessment of University-based Research Assessing University-based Research 71.
210 Ramsay and Stapledon 1997 MULR 680-683.
211 Korobkin 1999 Fla St U L Rev 864, 868.
212 Korobkin 1999 Fla St U L Rev 865; Osterloh and Frey Research Governance in Academia 8. Eisenberg and Wells Economic Inquiry 1302 note that the original idea of citations was to track ideas and not to assess the quality of the article.
213 Osterloh and Frey Research Governance in Academia 10. Newer software can exclude self-citations that may distort the ranking (Perry 2006 Va J Law Technol 28).
214 Bias in favour of prominent authors was noted by Datt, Tran and Tran-Nam 2009 ATF 350.
215 Campbell, Goodacre and Little 2006 J L & Soc'y 339.
216 Datt, Tran and Tran-Nam 2009 ATF 350.
217 Perry 2006 Va J Law Technol 26.
218 Korobkin 1999 Fla St U L Rev 865.
219 Korobkin 1999 Fla St U L Rev 868.
220 Osterloh and Frey Research Governance in Academia 11; Datt, Tran and Tran-Nam 2009 ATF 350.
221 Campbell, Goodacre and Little 2006 J L & Soc'y 340 refers to this as a "gesture of respect to influential academics".
222 Korobkin 1999 Fla St U L Rev 865-866. Steinbuch 2009 Loy LA L Rev 92, 100. See also Ramsay and Stapledon 1997 MULR 678; Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.897.6809&rep=rep1&type=pdf 4; Marashi et al 2015 SAJS 3.
223 Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.897.6809&rep=rep1&type=pdf 4-5.
224 Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.897.6809&rep=rep1&type=pdf 4. Perry notes that specialised journals are inclined to self-cite (Perry 2006 Va J Law Technol 27; Ramsay and Stapledon 1997 MULR 683). Some authors exploit the citation system to increase their impact factor. This practice is known as "gaming the system" and this is especially problematic if impact factors are used as the main quality measure (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 96).
225 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 71.
226 Perry 2006 Va J Law Technol 25-26.
227 Perry 2006 Va J Law Technol 30; Ramsay and Stapledon 1997 MULR 683.
228 Doyle 2004 Leg Ref Serv Q 16-18. It has been argued that any citation system should be limited to recent years (Perry 2006 Va J Law Technol 30). He recommends between 4 and 10 years (Perry 2006 Va J Law Technol 31).
229 Perry 2006 Va J Law Technol 33. To alleviate the problem of the varied number of articles per journal, it has been suggested that the solution would be to divide the number of submissions by the number of publication outlets (Ramsay and Stapledon 1997 MULR 683). This solution was adopted in this article.
230 Doyle 2004 Leg Ref Serv Q 18.
231 Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.897.6809&rep=rep1&type=pdf 4, quoting Moed, Van Leeuwen and Reedijk 1998 J Doc 416; European Union Expert Group on Assessment of University-based Research Assessing University-based Research 73.
232 Korobkin 1999 Fla St U L Rev 867.
233 Korobkin 1999 Fla St U L Rev 868.
234 Perry 2006 Va J Law Technol 31; Korobkin 1999 Fla St U L Rev 869. Even if an article on gambling law is excellent, it is unlikely to be cited by scholars with no interest in the area. An article on the equality clause in the Constitution will be cited by numerous academics in various areas, even if it may be of a lesser quality (examples of Korobkin 1999 Fla St U L Rev 869 adapted for a South African scenario). Also see Steinbuch 2009 Loy LA L Rev 92, 101.
235 London School of Economics 2010 http://blogs.lse.ac.uk/impactofsocialsciences/files/2010/11/chart3.jpg.
236 Datt, Tran and Tran-Nam 2009 ATF 348.
237 For a list of articles until 1999, see George and Guthrie 1999a Fla St U L Rev 824 fn 59.
238 See for example Marashi et al 2015 SAJS 1-3, which proposed to classify biomedical publications and analyse their citation profiles by locating the citation within the structure of the article; specifically the protocol, the methodology, the descriptive section or the theoretical framework (Marashi et al 2015 SAJS 2).
239 Eisenberg and Wells 2014 Economic Inquiry 1313.
240 Datt, Tran and Tran-Nam 2009 ATF 344. It is for this reason that the ISI citation analysis is not used for the Humanities cluster that includes law as a discipline.
241 Datt, Tran and Tran-Nam 2009 ATF 350; Van Gestel 2015 Legal Studies 168.
242 Datt, Tran and Tran-Nam 2009 ATF 350.
243 For a discussion of the system, see Doyle 2004 Leg Ref Serv Q 25-27; Datt, Tran and Tran-Nam 2009 ATF 345; Steinbuch 2009 Loy LA L Rev 107-112. Van Gestel describes their ranking as the "most influential and comprehensive" (Van Gestel 2015 Legal Studies 168). Steinbuch argues that this system is "highly right tailed", meaning that very few journals score highly. The clear majority are "clumped in a relatively narrow band" (Steinbuch 2009 Loy LA L Rev 108). The rankings are also unsuitable for books (Steinbuch 2009 Loy LA L Rev 109). A search for the South African law journals on the Hein Online "Most cited law journals" as proposed by Doyle 2004 Leg Ref Serv Q 4 did not show any of these journals. A search of the citation index of the Web of Science only referenced the SAHRJ.
244 Eisenberg and Wells 2014 Economic Inquiry 1032; Editorial 2012 EJIL 608.
245 Eisenberg and Wells 2014 Economic Inquiry 1311-1312.
246 Eisenberg and Wells 2014 Economic Inquiry 1301. The inconsistency is also illustrated by the various rankings of the SAJHR as set out on their website (WITS date unknown https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf): In terms of Thompson Reuters the journal was ranked first of the South African law journals with a combined score of 100. It is second, with an impact factor of 0.04, in the Washington and Lee University's Law Journals: Submissions and Rankings System (2006-2013). It has a 'B' rating under the CERES system for research valuation with an impact factor of 0.051 (and a 5-year impact factor of 0.170) with 61 total citations in the Journal Citation Reports 2013. This increased to an impact factor of 0.392 (with a 5-year impact factor of 0.36) in the Journal Citation Reports 2015.
247 Doyle 2004 Leg Ref Serv Q 25. It also includes the number of citations in full-text US state and federal case databases (Doyle 2004 Leg Ref Serv Q 25-26). Although Google Scholar could be another possibility, this is excluded from this article as the 2014 data are no longer available.
248 The Washington and Lee website (Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/) defines the combined-score as follows: "The combined-score is a composite of each journal's impact-factor and total cites count. The combined-score is, by default, weighted with approximately a third of the weight given to impact-factor and two-thirds given to total cites. The resulting score is then normalized."
249 The Washington and Lee website (Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/) defines impact factor as: "Impact-factor shows the average number of annual citations to articles in each journal (rounded to two decimal places). The method by which impact-factor is calculated is to conduct each of the Westlaw searches for citing articles in 8 separate yearly slices."
250 The Washington and Lee website (Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/) defines currency as: "Currency-factor aims to compare journals on how rapidly their articles become cited. It examines a three-year interval looking at how many articles in Westlaw's JLR database, made available during those three years, cite items published by a journal and dated during those same three years."
251 Doyle 2004 Leg Ref Serv Q 9.
252 Datt, Tran and Tran-Nam 2009 ATF 344.
253 Ramsay and Stapledon 1997 MULR 679 fn 26. Smyth had already ranked journals by their citation in high court cases between 1993 and 1997 as included in the Commonwealth Law Reports (Smyth 1998 U Tas LR 164 onwards).
254 Smyth 2012 UNSWLJ 208. It is possible that the courts may use legal material without citing it (Smyth 2012 UNSWLJ 209).
255 Perry 2006 Va J Law Technol 23.
256 Perry 2006 Va J Law Technol 23-24.
257 Perry 2006 Va J Law Technol 24.
258 Perry 2006 Va J Law Technol 25-26.
259 The Lexis Nexis databases included the All SA Law Reports, the Constitutional Court Law Reports, the Labour Law Reports, Pension Law Reports and Judgments Online. The Juta databases included the South African Law Reports, South African Criminal Law Reports, South African Appellate Division Law Reports and Juta's Unreported Judgments. SAFLII included the Competition Appeal Court; Competition Tribunal, Companies Tribunal, Constitutional Court, Court of the Commissioner of Patents, Commercial Crime Court, Eastern Cape High Courts: Bhisho, Grahamstown, Mthatha, East London and PE; Electoral Court, Equality Court, Free State High Court, Bloemfontein; High Courts Gauteng: North and South; High Courts, KwazuluNatal: Durban and Pietermaritzburg, Land Claims Court; Limpopo High Court, Thohoyandou; Northern Cape High Court, Kimberley; North West High Court, Mafikeng; Supreme Court of Appeal, Tax Court, Water Tribunal and the Western Cape High Court, Cape Town.
260 Perry 2006 Va J Law Technol 30.
261 Although this application is based on the South African system, the idea originated from a study done by Campbell, Goodacre and Little 2006 J L & Soc'y 335-344 in the UK relating to articles submitted by academics for submission to their HEFCE RAE Panels.
262 NRF/RISA Evaluation and Rating 3.
263 This is similar to the research done by Campbell, Goodacre and Little 2006 J L & Soc'y 346 in the UK, which found that the percentage of journal submissions in Law is generally less than in other disciplines.
264 South African journals (29%), international articles (25%) and chapters in books (26%).
265 There was no significant difference between the sub-categories of the B-rated scientists, although B2 academics seem to publish less in South African journals and more in international journals.
266 C1 (53%), C2 (56%) and C3 (65%).
267 With C-rated academics, international journals are 10% (C1), 36% (B2) and 8% (B3), whilst chapters in books are 18%, 15% and 8% respectively. Books represent 13%, 12% and 4% of the submissions.
268 The submission of chapters and books increased from 7% and 6% respectively for C3 rated academics to 20% and 14% respectively for C1 rated academics.
269 For other Y-rated academics, just under half the submissions (46%) consisted of South African law journals, with about 10% books and 10% chapters in books. The remainder were mostly international journals.
270 Although the actual number of legal academics differs annually, the number in 2014 was more than 670 according to the various faculty websites. This number is conservative as a few of the faculties do not display full details of all their academics on their websites.
271 European Union Expert Group on Assessment of University-based Research Assessing University-based Research 36.
272 Perry 2006 Va J Law Technol 38.
273 One may also be tempted to look at the table and conclude that TSAR is consistently ranked high, but this does not take into consideration its refusal to divulge to ASSAf the print run information that may have impacted the outcome.
274 Svantesson and White 2009 Bond LR 191.
275 Van Gestel 2015 Legal Studies 179, 184.
276 Svantesson and White 2009 Bond LR 191.
277 Svantesson and White 2009 Bond LR 186.