SciELO - Scientific Electronic Library Online

 
vol.109 issue9-10Contamination of the water supply to the town of Carolina, Mpumalanga, January 2012An H5N1 influenza DNA vaccine for South Africa author indexsubject indexarticles search
Home Pagealphabetic serial listing  

South African Journal of Science

On-line version ISSN 1996-7489

S. Afr. j. sci. vol.109 n.9-10 Pretoria Jan. 2013

 

RESEARCH LETTER

 

Assessing the scientific relevance of a single publication over time

 

 

Philipp A. Bloching; Harald Heinzl
Medical University of Vienna - Center for Medical Statistics, Informatics, and Intelligent Systems, Vienna, Austria

Correspondence

 

 


ABSTRACT

Quantitatively assessing the scientific relevance of a research paper is challenging for two reasons. Firstly, scientific relevance may change over time, and secondly, it is unclear how to evaluate a recently published paper. The temporally averaged paper-specific impact factor is defined as the yearly average of citations to the paper until now including bonus citations equal to the journal impact factor in the publication year. This new measure subsequently allows relevance rankings and annual updates of all (i.e. both recent and older) scientific papers of a department, or even a whole scientific field, on a more objective basis. It can also be used to assess both the average and overall time-dependent scientific relevance of researchers in a specific department or scientific field.

Keywords: journal impact factor; paper-specific impact factor; time course; single paper evaluation; researcher evaluation


 

 

Introduction

Identifying the most relevant scientific papers within a certain department or field may seem to be a simple task at first sight, but the resultant need to rank the papers makes it a serious challenge. A selection compiled by superiors or (alleged) experts may be immediately challenged by those whose papers do not feature. Accusations of bias and general discontent may follow. In order to avoid such discord, the selection could alternatively be based on a compromise between all the stakeholders. For instance, each researcher could be allowed to select a certain number of their papers depending on years already spent in research, although such an approach may lead to suboptimal selections from the viewpoint of scientific relevance.

Usually, quantitative measures of the scientific relevance of the publishing journal are considered, with the journal impact factor being the most common choice.1 The journal impact factor can be regarded as an estimate of the average number of citations a journal paper will receive in the first 2 years following the publication year. Most notable is the poor correlation of a journal's impact factor and the actual number of citations of the individual papers within the journal.2 Therefore, measuring the scientific relevance of single papers by the impact factor of the journal in which they appear seems reasonable for papers recently published. However, for all other papers, the use of the journal impact factor is limited by a levelling effect as both highly cited and infrequently cited papers are given the same impact factor. There is therefore a need for a quantitative paper-specific evaluation criterion.

The basic idea behind paper-specific scientific relevance is that highly relevant papers will be cited more often than papers which are less relevant. A simple and straightforward first approach is therefore to sum the number of citations a paper has received to date. The citation h-index for single publications is another approach adapted from the author h-index.3,4 A publication has citation h-index H, if H of its citing papers have been cited at least H times each, and the remainder of the citing papers have been cited no more than H times each. Both the number of citations received and the citation h-index can never decrease over time. The former can be considered a measure of the first-generation spread of a publication and the latter a measure of its second-generation spread. Neither can be used to compare papers with different publication dates, as older papers are obviously favoured over newer ones.

Dividing the number of citations received and the citation h-index by the number of years since the respective publication appeared yields the average number of citations per year and the citation m-quotient, respectively. The latter is analogous with the m-quotient for authors.3 The average number of citations per year is a disadvantage only for recently published papers whose first citing papers have yet to be written, submitted and published. This disadvantage is even more prolonged for the citation m-quotient, for which a second generation of citing papers is required.

However, all these approaches ignore a psychological influence, in that they do not appraise the reputation an author acquires when publishing in a journal with a high impact factor. Such journals are usually flooded with submissions and simply having a paper accepted for publication is considered an achievement sui generis, even if the paper is rarely cited afterwards.

Keeping all these factors in mind, we aimed to derive a measure which

1. considers citations of a paper as crucial

2. avoids disadvantages for a paper just published

3. recognises the reputation of publishing in a journal with a high impact factor

The temporally averaged paper-specific impact factor (TAPSIF) fulfills these requirements. TAPSIF is the paper's average number of citations per year (including the publication year) combined with bonus cites for the publishing journal's prestige - which is taken as the journal impact factor from the publication year. Furthermore, the annual TAPSFF values of all the papers of an author can be appropriately combined to measure the overall scientific relevance of the author over time.

 

Methods

The temporally averaged paper-specific impact factor (JAPSIF) of a scientific paper for a given year is defined as

where pubyear denotes the publication year of the paper, IF(pubyear) denotes the journal impact factor in the publication year, and citations(i) denotes the number of citations the paper has received in the i-th year.

The basic idea is that in the publication year the paper's scientific value is derived from the impact factor - the scientific relevance or prestige of the journal. With the passing of time, the journal's relevance becomes less important and the paper's own performance - determined by the number of citations received - comes to the fore. Note that alternatively the journal impact factor of the paper submission year or of the year for which TAPSIF is computed could be considered. The former would account for the relevance of the journal when it was chosen by the authors, the latter would account for temporal changes in relevance (for details see Bloching5).

Assume that an author has published n(year) papers up to a specific year. The TAPSFF can now be used to define a measure for the scientific relevance of the author for this year - the temporally summarised author-specific impact factor (TSASIF):

where TAPSIFJ(year) denotes the TAPSFF of the y-th paper in the specific year. An annual mean TAPS F for the papers of the author can also be defined - the temporally averaged author-specific impact factor (TAASIF):

 

Results and discussion

For demonstration purposes assume that researchers in a scientific department have published three papers - A, B and C - in three different scientific journals in 2004 (and none before or afterwards). Furthermore assume that the journal impact factors of the journals in which Papers A and B were published were 2.5 and 10 in 2004, respectively. Assume Paper C was published in a local scientific journal or conference proceedings for which no impact factor was available, so a value of zero is used. The assumed citation counts and the resultant TAPSFF values for the three papers for the years 2004 to 2011 are shown in Table 1.

 

 

Papers B and C start with a TAPSFF value equal to their respective journal impact factor in the assumed publication year 2004. Paper A shows a TAPSIF value larger than its journal impact factor as it has already received a citation in the publication year. Paper A regularly receives a small number of citations over the next years. The resulting TAPSFF values vary slightly, but tend to stabilise in the long run. The high impact factor of the journal in which Paper B was published leads to a high TAPSIF value for 2004. Because the paper is not cited thereafter, the TAPSIF constantly declines. Paper B would be considered less relevant than Paper A from 2009 onwards.

Paper C exemplifies the case of an originally unappreciated publication which suddenly attracts attention. The TAPSIF values reflect this pattern over time by grading Paper C as most relevant from 2008 onwards. A paper published in conference proceedings or in a journal without an impact factor is often considered a wasted effort. However, through citations it can gain in value over time as TAPSIF is recomputed each year. With regard to our initial question, such a paper could even, at some point, score highly in a selected ranking.

As citation habits and frequencies are much too different between scientific subject areas, TAPSff values, like journal impact factors, should not be compared between them. In other words, the TAPSFF is ideally suited for ranking scientific papers of a single department (or, more generally, a single scientific field) and not for ranking those of a whole university (or, more generally, across various scientific fields).

The TAPSFF can also provide a reasonable basis for researcher evaluation beyond the popular h-index. Crucial features of the h-index are its monotonic increase over time and its favouring of researchers who steadily publish medium-quality papers over those who rarely publish highly cited papers.

The TAASFF enables a yearly comparison of the individual average scientific relevance across various researchers, irrespective of their seniority and productivity. The TSASIF measures a researcher's overall scientific relevance in a specific year. A researcher's seniority and productivity are considered in the TSASIF. If a researcher's scientific relevance begins to decline, then both TAASIF and TSASIF will decrease over time. As for TAPSIF, the use of TAASIF and TSASIF should be restricted to single departments or single scientific fields for the purposes of comparison.

Authors' contributions

Both authors contributed equally to this paper. The paper is partly based on the diploma thesis of PA.B.; H.H. suggested the three new bibliometric measures and supervised the thesis.

 

References

1. Garfield E. Citation indexes for science. A new dimension in documentation through association of ideas. Science. 1955;122(3159):108-111. http://dx.doi.org/10.1126/science.122.3159.108        [ Links ]

2. Seglen PO. Why the impact factor of journals should not be used for evaluating research. Brit Med J. 1997;314(7079):498-502. http://dx.doi.org/10.1136/bmj.314.7079.497        [ Links ]

3. Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA. 2005;102(46):16569-16572. http://dx.doi.org/10.1073/pnas.0507655102        [ Links ]

4. Schubert A. Using the h-index for assessing single publications. Scien-tometrics. 2009;78(3):559-565. http://dx.doi.org/10.1007/s11192-008-2208-3        [ Links ]

5. Bloching PA. Quantitative Bewertung von medizinischen Publikationen: Temporally averaged paper-specific impact factor [Quantitative assessment of medical publications: Temporally averaged paper-specific impact factor] [thesis]. Vienna: Medical University of Vienna; 2012. German.         [ Links ]

 

 

Correspondence:
Harald Heinzl
Email:harald.heinzl@meduniwien.ac.at

Received: 11 Mar. 2013
Accepted: 05 Apr. 2013

 

 

Medical University of Vienna - Center for Medical Statistics, Informatics Intelligent Systems, Spitalgasse 23, A-1090 Vienna, Austria

Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License