RESEARCH LETTER

 

Assessing the scientific relevance of a single publication over time

 

 

Philipp A. Bloching; Harald Heinzl
Medical University of Vienna - Center for Medical Statistics, Informatics, and Intelligent Systems, Vienna, Austria

Correspondence

 

 

---

ABSTRACT

Quantitatively assessing the scientific relevance of a research paper is challenging for two reasons. Firstly, scientific relevance may change over time, and secondly, it is unclear how to evaluate a recently published paper. The temporally averaged paper-specific impact factor is defined as the yearly average of citations to the paper until now including bonus citations equal to the journal impact factor in the publication year. This new measure subsequently allows relevance rankings and annual updates of all (i.e. both recent and older) scientific papers of a department, or even a whole scientific field, on a more objective basis. It can also be used to assess both the average and overall time-dependent scientific relevance of researchers in a specific department or scientific field.

Keywords: journal impact factor; paper-specific impact factor; time course; single paper evaluation; researcher evaluation

---

 

 

Introduction

Identifying the most relevant scientific papers within a certain department or field may seem to be a simple task at first sight, but the resultant need to rank the papers makes it a serious challenge. A selection compiled by superiors or (alleged) experts may be immediately challenged by those whose papers do not feature. Accusations of bias and general discontent may follow. In order to avoid such discord, the selection could alternatively be based on a compromise between all the stakeholders. For instance, each researcher could be allowed to select a certain number of their papers depending on years already spent in research, although such an approach may lead to suboptimal selections from the viewpoint of scientific relevance.

Usually, quantitative measures of the scientific relevance of the publishing journal are considered, with the journal impact factor being the most common choice.¹ The journal impact factor can be regarded as an estimate of the average number of citations a journal paper will receive in the first 2 years following the publication year. Most notable is the poor correlation of a journal's impact factor and the actual number of citations of the individual papers within the journal.² Therefore, measuring the scientific relevance of single papers by the impact factor of the journal in which they appear seems reasonable for papers recently published. However, for all other papers, the use of the journal impact factor is limited by a levelling effect as both highly cited and infrequently cited papers are given the same impact factor. There is therefore a need for a quantitative paper-specific evaluation criterion.

The basic idea behind paper-specific scientific relevance is that highly relevant papers will be cited more often than papers which are less relevant. A simple and straightforward first approach is therefore to sum the number of citations a paper has received to date. The citation h-index for single publications is another approach adapted from the author h-index.^3,4 A publication has citation h-index H, if H of its citing papers have been cited at least H times each, and the remainder of the citing papers have been cited no more than H times each. Both the number of citations received and the citation h-index can never decrease over time. The former can be considered a measure of the first-generation spread of a publication and the latter a measure of its second-generation spread. Neither can be used to compare papers with different publication dates, as older papers are obviously favoured over newer ones.

Dividing the number of citations received and the citation h-index by the number of years since the respective publication appeared yields the average number of citations per year and the citation m-quotient, respectively. The latter is analogous with the m-quotient for authors.³ The average number of citations per year is a disadvantage only for recently published papers whose first citing papers have yet to be written, submitted and published. This disadvantage is even more prolonged for the citation m-quotient, for which a second generation of citing papers is required.

However, all these approaches ignore a psychological influence, in that they do not appraise the reputation an author acquires when publishing in a journal with a high impact factor. Such journals are usually flooded with submissions and simply having a paper accepted for publication is considered an achievement sui generis, even if the paper is rarely cited afterwards.

Keeping all these factors in mind, we aimed to derive a measure which

1. considers citations of a paper as crucial

2. avoids disadvantages for a paper just published

3. recognises the reputation of publishing in a journal with a high impact factor

The temporally averaged paper-specific impact factor (TAPS_IF) fulfills these requirements. TAPS_IF is the paper's average number of citations per year (including the publication year) combined with bonus cites for the publishing journal's prestige - which is taken as the journal impact factor from the publication year. Furthermore, the annual TAPS_FF values of all the papers of an author can be appropriately combined to measure the overall scientific relevance of the author over time.

 

Methods

The temporally averaged paper-specific impact factor (JAPS_IF) of a scientific paper for a given year is defined as

where pubyear denotes the publication year of the paper, IF(pubyear) denotes the journal impact factor in the publication year, and citations(i) denotes the number of citations the paper has received in the i-th year.

The basic idea is that in the publication year the paper's scientific value is derived from the impact factor - the scientific relevance or prestige of the journal. With the passing of time, the journal's relevance becomes less important and the paper's own performance - determined by the number of citations received - comes to the fore. Note that alternatively the journal impact factor of the paper submission year or of the year for which TAPS_IF is computed could be considered. The former would account for the relevance of the journal when it was chosen by the authors, the latter would account for temporal changes in relevance (for details see Bloching⁵).

Assume that an author has published n(year) papers up to a specific year. The TAPSFF can now be used to define a measure for the scientific relevance of the author for this year - the temporally summarised author-specific impact factor (TSAS_IF):

where TAPSIFJ(year) denotes the TAPSFF of the y-th paper in the specific year. An annual mean TAPS F for the papers of the author can also be defined - the temporally averaged author-specific impact factor (TAAS_IF):

 

Results and discussion

For demonstration purposes assume that researchers in a scientific department have published three papers - A, B and C - in three different scientific journals in 2004 (and none before or afterwards). Furthermore assume that the journal impact factors of the journals in which Papers A and B were published were 2.5 and 10 in 2004, respectively. Assume Paper C was published in a local scientific journal or conference proceedings for which no impact factor was available, so a value of zero is used. The assumed citation counts and the resultant TAPS_FF values for the three papers for the years 2004 to 2011 are shown in Table 1.

 

 

Papers B and C start with a TAPS_FF value equal to their respective journal impact factor in the assumed publication year 2004. Paper A shows a TAPS_IF value larger than its journal impact factor as it has already received a citation in the publication year. Paper A regularly receives a small number of citations over the next years. The resulting TAPS_FF values vary slightly, but tend to stabilise in the long run. The high impact factor of the journal in which Paper B was published leads to a high TAPS_IF value for 2004. Because the paper is not cited thereafter, the TAPS_IF constantly declines. Paper B would be considered less relevant than Paper A from 2009 onwards.

Paper C exemplifies the case of an originally unappreciated publication which suddenly attracts attention. The TAPS_IF values reflect this pattern over time by grading Paper C as most relevant from 2008 onwards. A paper published in conference proceedings or in a journal without an impact factor is often considered a wasted effort. However, through citations it can gain in value over time as TAPS_IF is recomputed each year. With regard to our initial question, such a paper could even, at some point, score highly in a selected ranking.

As citation habits and frequencies are much too different between scientific subject areas, TAPS_ff values, like journal impact factors, should not be compared between them. In other words, the TAPS_FF is ideally suited for ranking scientific papers of a single department (or, more generally, a single scientific field) and not for ranking those of a whole university (or, more generally, across various scientific fields).

The TAPS_FF can also provide a reasonable basis for researcher evaluation beyond the popular h-index. Crucial features of the h-index are its monotonic increase over time and its favouring of researchers who steadily publish medium-quality papers over those who rarely publish highly cited papers.

The TAAS_FF enables a yearly comparison of the individual average scientific relevance across various researchers, irrespective of their seniority and productivity. The TSAS_IF measures a researcher's overall scientific relevance in a specific year. A researcher's seniority and productivity are considered in the TSAS_IF. If a researcher's scientific relevance begins to decline, then both TAAS_IF and TSAS_IF will decrease over time. As for TAPS_IF, the use of TAAS_IF and TSAS_IF should be restricted to single departments or single scientific fields for the purposes of comparison.

Authors' contributions

Both authors contributed equally to this paper. The paper is partly based on the diploma thesis of PA.B.; H.H. suggested the three new bibliometric measures and supervised the thesis.

 

References

1. Garfield E. Citation indexes for science. A new dimension in documentation through association of ideas. Science. 1955;122(3159):108-111. http://dx.doi.org/10.1126/science.122.3159.108        [ Links ]

2. Seglen PO. Why the impact factor of journals should not be used for evaluating research. Brit Med J. 1997;314(7079):498-502. http://dx.doi.org/10.1136/bmj.314.7079.497        [ Links ]

3. Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA. 2005;102(46):16569-16572. http://dx.doi.org/10.1073/pnas.0507655102        [ Links ]

4. Schubert A. Using the h-index for assessing single publications. Scien-tometrics. 2009;78(3):559-565. http://dx.doi.org/10.1007/s11192-008-2208-3        [ Links ]

5. Bloching PA. Quantitative Bewertung von medizinischen Publikationen: Temporally averaged paper-specific impact factor [Quantitative assessment of medical publications: Temporally averaged paper-specific impact factor] [thesis]. Vienna: Medical University of Vienna; 2012. German.         [ Links ]

 

 

Correspondence:
Harald Heinzl
Email:harald.heinzl@meduniwien.ac.at

Received: 11 Mar. 2013
Accepted: 05 Apr. 2013

 

 

Medical University of Vienna - Center for Medical Statistics, Informatics Intelligent Systems, Spitalgasse 23, A-1090 Vienna, Austria