RESEARCH LETTER
http://dx.doi.org/10.1590/sajs.2014/20140192

 

Durham versus Durban: quantifying productivity in astrophysics research

 

 

Matthew Hilton

Astrophysics and Cosmology Research Unit, School of Mathematics, Statistics and Computer Science, University of KwaZulu-Natal, Durban, South Africa

Correspondence

 

 

---

ABSTRACT

Quantifying and rewarding research productivity is a contentious issue. In South Africa, there are at least two systems in wide use: peer assessment (as used by the National Research Foundation in providing researchers with individual ratings) and a simple publication count (used by the Department of Higher Education and Training to incentivise research output). At the University of KwaZulu-Natal (UKZN), the latter is used to grade the research performance of staff; however, this metric penalises those academics who work in large teams, as is increasingly common in astronomy. To test for correspondence between this metric and perceived research quality, I conducted a case study of the Extragalactic and Cosmology Group at Durham University in the UK, which is one of the leading astrophysics research groups in the world. I found that 44-74% of the permanent academic staff within this research group would not meet the research productivity target applied at UKZN in 2014. Given the disparity between this result and the esteem in which the research of the Durham group is held, I suggest that alternative methods of recognising and rewarding research output by funding agencies and universities should be explored, with an emphasis on quality rather than quantity.

Keywords: bibliometrics; citation counts; publication counts; research funding; University of KwaZulu-Natal

---

 

 

Research productivity

Research productivity at the University of KwaZulu-Natal is measured in terms of publication units (PUs). Following the Department of Higher Education and Training scheme, 60 PUs are awarded per publication in a journal on the Thomson Reuters Web of Science (WoS) Science Citation Index.¹ However, this number is divided by the number of co-authors - hence, a paper with 10 authors will result in 6 PUs being awarded to each staff member who is an author. The graduation of a PhD student also amounts to an award of 60 PUs in this system.

In this article, the focus is on astrophysics. In this field, like in many other areas of science, there has been a steady increase in the average number of authors per journal publication over the last few decades.^2-4 Hence, long lists of authors are increasingly common (on average, seven authors per paper in the period 2 0 06-201 0⁴), and the authors of such papers are awarded small numbers of PUs. The productivity of researchers in the College of Agriculture, Engineering and Science at UKZN is judged according to whether staff have met a target number of PUs in the calendar year, and the target does not vary with discipline. Peer evaluation and citation metrics are not used to assess research quality. Table 1 lists the research productivity targets at UKZN in 2014. Note that higher targets are set for higher academic ranks (i.e. the target increases proportionally from Lecturer to Senior Lecturer to Associate Professor to Professor).

 

 

A case study: Durham University

In order to determine if the system used to quantify research productivity at UKZN is a good predictor of the esteem in which researchers are held, I have applied it to the permanent academic staff in the Extragalactic and Cosmology Group at Durham University.⁵ According to Thomson Reuters, Durham was the top-rated university in Europe in space sciences research over the period 1998-2008 in terms of citations per paper.⁶ Moreover, Durham University is ranked 66 in the world according to the Times Higher Education university rankings in physical sciences 2013-2014.⁷ I make the assumption here that Durham's placing in these rankings is as a result of the efforts of a majority of the staff, and is not driven by one or two individuals.

In astronomy, the standard bibliographic database in use is the NASA Astrophysics Data System (ADS).⁸⁹ This database provides a variety of information, including citation counts; but, in this article, I used only its author count function in order to calculate the number of PUs per paper that would be awarded to an author based at UKZN. A Python script was used to extract the 2013 publications list for a given author from ADS, and to calculate their total number of PUs and the corresponding performance management (PM) score according to their academic rank (Table 1). Note that only articles published in journals on the WoS list were considered.

I performed these calculations for all 27 permanent staff listed as being part of the Extragalactic and Cosmology Group at Durham.⁵ (In the case of one staff member, the calculation was based on the publications for 2013 listed on their staff profile page, rather than ADS output; the ADS query is based on only the author's surname and first initial, so common names such as Li (or Smith) can lead to spuriously large PUs and PM scores by matching multiple authors.) Emeritus staff were also excluded. Readers in the UK system were assumed to be equivalent to Associate Professors in the South African university sector.

 

Results and discussion

Figure 1 shows the number of PUs produced by each staff member at Durham in 2013 in comparison to the target level (PM score = 3) for each academic rank at UKZN (Table 1).

As can clearly be seen, academics in the Durham Extragalactic and Cosmology Group do not meet the research productivity target applied at UKZN: based on publications, 37% of the staff produced fewer than 60 PUs (the expected performance of a Lecturer at UKZN) in 2013, despite only 26% of the staff being at Lecturer level. Taking into account the different performance levels expected at higher academic ranks, 74% of Durham staff produced fewer than the expected number of PUs. It is clear that the UKZN research productivity metric does not reflect the high regard in which this research group is held.⁶⁷ I also note that the correlation between academic rank and research output is weak, with only Lecturers producing significantly fewer PUs on average (median: 51.4 PUs) than higher ranks (78.4, 75.2 and 77.7 PUs for Senior Lecturers, Associate Professors and Professors, respectively).

The above analysis does not, however, take into account the graduation of PhD students. This figure is more difficult to quantify, but can be estimated using information on the Extragalactic and Cosmology Group's staff web page.⁵ In June 2014, a total of 48 postgraduate students are listed as being part of the Durham group, resulting in an average of 1.7 students per academic staff member. For the purposes of this estimate I will assume that all listed students are at PhD level. Assuming that students are distributed equally among all 27 staff members, that the number of students remains constant, and that a student graduates every 3 years (an optimistic estimate¹⁰), then 16 students should graduate per year, boosting each staff member's research productivity by 35.6 PUs on average. The result of this boost is shown as the dashed line in Figure 1. The inclusion of PhD student graduation clearly results in a significant improvement towards the UKZN productivity target. Nevertheless, even with this optimistic estimate of the PhD graduation rate, 44% of Durham staff would be below the expected research productivity level at UKZN.

Finally, I note that I have not taken into account that some of the staff classified as Lecturers are in fact currently Senior Research Fellows with (presumably) limited teaching duties, and therefore it is likely they have more time to devote to research than the average Lecturer at UKZN. Thus, even with the optimistic figures used in this analysis, it is clear that the research productivity metric applied at UKZN is not well aligned with the productivity of researchers based at world-leading research centres in astrophysics, such as at Durham University.

 

Conclusion

I have performed a simple analysis of the research productivity of the Extragalactic and Cosmology Group at Durham University using the metric applied at UKZN. I found that, in 2013, almost half of the staff in this group attained fewer PUs than would be expected of them at UKZN, even with an optimistic estimate of the PhD student graduation rate at Durham. However, the Durham group is regarded as producing world-leading research in astrophysics.⁶ I conclude that the quantity of research produced, as measured using the PUs system, does not necessarily reflect the quality of research, and it is the latter that leads to international recognition such as that enjoyed by the Durham group. I suggest that alternative methods of recognising and rewarding research output at South African institutions should be explored. These methods might include citation metrics, in which case caution should be applied, as the typical number of citations per article varies widely across fields, and even within sub-fields. Peer evaluation, through reading of actual articles, while difficult, subjective and time consuming, is perhaps the fairest way for the value of research to be assessed.

 

Acknowledgements

I acknowledge many thought-provoking and useful discussions with colleagues at UKZN and thank the anonymous reviewers for several useful comments that improved this manuscript.

 

References

1. Thomson Reuters. Source publication list for Web of Science Science Citation Index expanded 2014 [document on the Internet].         [ Links ] c2013 [cited 2014 July 29]. Available from: http://ip-science.thomsonreuters.com/mjl/publist_sciex.pdf.

2. Schulman E, French JC, Powell AL, Eichhorn G, Kurtz MJ, Murray SS. Trends in astronomical publications between 1975 and 1996. Publ Astron Soc Pac. 1997;109:1278. http://dx.doi.org/10.1086/134008        [ Links ]

3. Henneken EA. Publication trends in astronomy: The lone author [document on the Internet].         [ Links ] c2012 [cited 2014 June 06]. Available from: http://arxiv.org/pdf/1202.4646v1.pdf

4. Milojevic S. Principles of scientific research team formation and evolution. Proc Natl Acad Sci USA. 2014;111:3984. http://dx.doi.org/10.1073/pnas.1309723111        [ Links ]

5. Current Members, Extragalactic & Cosmology Group, Durham University [homepage on the Internet].         [ Links ] No date [cited 2014 June 05]. Available from: http://astro.dur.ac.uk/index.php?content=Staff/Staff.

6. Times Higher Education. Institutional rankings in space sciences [homepage on the Internet].         [ Links ] c2008 [cited 2014 Oct 01]. Available from: http://www.timeshighereducation.co.uk/news/institutional-rankings-in-space-sciences/403363.article

7. Times Higher Education. Top 100 universities for physical sciences 20132014 [homepage on the Internet].         [ Links ] c2014 [cited 2014 June 06]. Available from: http://www.timeshighereducation.co.uk/world-university-rankings/2013-14/subject-ranking/subject/physical-sciences.

8. The NASA Astrophysics Data System [homepage on the Internet].         [ Links ] No date [cited 2014 June 05]. Available from: http://www.adsabs.harvard.edu/.

9. Kurtz MJ, Eichhorn G, Accomazzi A, Grant CS, Murray SS, Watson JM. The NASA Astrophysics Data System: Overview. Astron Astrophys Suppl Ser. 2000;143:41. http://dx.doi.org/10.1051/aas:2000170        [ Links ]

10. Higher Education Funding Council for England. Rates of qualification from postgraduate research degrees: Projected study outcomes of full-time students starting postgraduate research degrees in 2010-1 1, ref: 201 3/17 [homepage on the Internet] c201 3 [cited 201 4 June 06].         [ Links ] Available from: http://www.hefce.ac.uk/pubs/year/2013/201317/name,82794,en.html.

 

 

Correspondence:
Matthew Hilton
Astrophysics and Cosmology Research Unit
School of Mathematics, Statistics and Computer Science
University of KwaZulu-Natal
Westville Campus
Durban 4041
South Africa
EMAIL: hiltonm@ukzn.ac.za

Received: 06 June 2014
Revised: 29 July 2014
Accepted: 14 Sep. 2014