Print version ISSN 0256-9574
SAMJ, S. Afr. med. j. vol.99 n.4 Cape Town Apr. 2009
Does the impact factor have too much impact?
Are all thy conquests, glories, triumphs, spoils Shrunk to this little measure?
William Shakespeare: Julius Caesar, Act III, scene i
The impact factor (IF) was developed in the 1960s by Eugene Garfield of the Institute for Scientific Information (ISI) in the USA. The IF for a journal is calculated according to the following formula:
The source articles include all original research, case reports and reviews. Letters (except in some cases where original work is published in the form of letters), opinions, editorials and opinion pieces are not included in the denominator, but are included in the numerator. In the case of both the numerator and denominator, the only journals that are considered are those included in the ISI indices. In the case of the South African Medical Journal, for example, the number of cites in 2007 to articles published in 2005 and 2006 was 189 and the number of source articles in 2005 and 2006 was 149. Hence the IF=189/149=1.268.
The index was developed for a particular purpose, namely to inform decisions about what journals should be included in the Sciences Citation Index. However, its use has expanded enormously. It is now used to evaluate the quality of specific articles in a journal, and even the work of the authors of papers published in the journal. Decisions about promotion, appointment and research funding are informed by the IF of the journals in which the applicant has published.
Opposition to the use of the IF has been growing in recent years. Steven Lock, previously editor of the British Medical Journal, had this to say about the matter: 'It is remarkable that scientists may rely upon such a non-scientific method for the evaluation of the scientific quality of a paper as the impact factor of the journal in which it is published.'1
What are the objections to the use of the IF to assess the quality of a paper? Some conceptual objections are immediately apparent from the definition of the IF. There is the assumption that papers are cited because they are of high quality. However, there are a number of other reasons why a paper may be cited, one of which is that it is of low quality. A paper with substantial flaws is likely to spawn a flurry of papers in which the flaws are explicated. Another reason that a paper may be cited frequently is utility; for example it introduces a new biochemical technique or psychometric scale, or is a review that is frequently cited by authors who seek a short-cut when situating their work in the context of the current state of the field. Of course, such a paper will often be of high quality, but this is not necessarily so. Also, a paper may be cited relatively frequently because it is accessible, for example through online accessibility or provision at no extra cost to members of a scientific or clinical society. Another conceptual problem occurs because of the 2-year time window. While it is certainly the case that for a subset of papers in a subset of journals this time period may be appropriate, there are numerous examples in science where the real significance of a paper has emerged only some years after initial publication. Also, the IF ignores the impact of a paper on clinical practice, policy, legislation, and industrial or commercial innovation. Finally, the impact of books is not addressed in the IF. This is a substantial shortcoming, especially for disciplines in which books are an important vehicle for the transmission of new knowledge.
In addition to these conceptual problems, there are a number of technical objections to the use of the IF to assess the quality of papers or their authors. The most obvious is that the ISI database includes only a very small percentage (less than 4%) of scientific journals published worldwide.2-5 The ISI is not entirely transparent about criteria for inclusion. However, it would appear to be the case that journals with certain characteristics are more likely to be included, namely those that publish in the English language, are based in the USA, have high circulations, and are from certain disciplines. Another technical issue is that there are a number of disciplinary differences that have an effect on the IF of journals across disciplines. Some disciplines have a tradition of citing more frequently than others, while some tend to move more slowly. Clinical and epidemiological research, for example, tends to move more slowly than basic laboratory sciences, partly because of the practical and logistical challenges that arise when implementing such studies. It is therefore less likely that a clinical or epidemiological paper will be cited within the 2year window than is the case with basic laboratory sciences. Finally, the IF refers to a mean - it does not follow that the impact of a particular paper in a journal correlates to the IF of that journal. Indeed, the overwhelming majority of citations to a particular journal are generated by a small proportion of papers in that journal. In the case of the journal Nature, for example, an analysis of citations in 2004 to papers published in the previous 2 years concluded that 89% of the citations were to just 25% of the papers.6
One consequence of these conceptual and technical problems is that the IF is liable to manipulation. Journal editors are subject to pressure from publishers, authors, advertisers and professional societies to increase the IF of their journals. An editorial decision was taken at the Australia and New Zealand Journal of Psychiatry (ANZJP) to decrease the frequency of publication of case reports, partly influenced by the limited contribution they make to the IF. However, it was stated that a 'sense of editorial integrity and ethical responsibility forbade the ANZJP making any other concessions in the quest to boost its IF'.7 It is clearly difficult to know where to draw the line between such responsibility and strategic decisions that have an impact on the wellbeing (or even viability) of a journal. Very few editors have publicly admitted to making editorial policy changes with a specific view to the effect on the IF.
The strategies that are available to journal editors who wish to increase the IF of their journals are provided in Table I. The IF can increase either by increasing the numerator or decreasing the denominator in the formula to calculate the IF, and the effect on each of these is indicated in the table. It is important to note that I am not proposing that editors adopt these strategies; rather, I provide this list to emphasise the point that it is possible for an editor to manipulate the IF. Of course, some of these strategies may be appropriate for other reasons, of which the most obvious example is improving the quality of papers.
Finally, use of the IF to evaluate the quality of journals or authors' work has a number of unintended adverse effects. Assumptions may be made about the quality of an author's contribution on the basis of the IFs of the journals in which he or she publishes. If the IFs of such journals are low, this may compromise a person's reputation, because of the assumption that the quality of an author's output correlates with the IF of the journals in which it is published. Related to this, authors may submit to the journals with the highest IFs, as opposed to the most suitable journals. This may be most evident for clinical researchers, since the IFs of journals reporting clinical research are generally lower than those of general medical journals or basic scientific journals. While the author may thus attain the kudos that accrues from publication in highimpact journals, clinicians in that field may be less likely to read the paper, and the impact on clinical practice may thus be attenuated. Also, authors may be less likely to publish in non-American journals, since these generally have lower IFs, which may have the effect of weakening journals in Africa and elsewhere. The effects of such publication habits may become amplified as postgraduate students and junior researchers emulate their supervisors and mentors.
What is the appropriate response to this scenario? Some argue that the IF should be discarded altogether. Gareth Williams, Dean of the Faculty of Medicine and Dentistry at the University of Bristol, adheres to this point of view: 'The impact factor is a pointless waste of time, energy, and money, and a powerful driver of perverse behaviours in people who should know better. It should be killed off, and the sooner the better. Academics should now acknowledge that we have been conned for long enough, and the academic community as a whole should now agree to consign the impact factor to the dustbin.'8 Others, while fully aware of the shortcomings of the IF, propose a more conservative response, in which the IF is modified to address some of these challenges, for example by extending the 2-year period for citations to 5 years. Some argue that such bibliometric measures are necessary to work towards a situation where there are fewer journals, containing fewer papers that are of higher quality.9
In conclusion, there is no 'little measure' that provides a valid assessment of the impact that an author's contribution to knowledge. The IFs of the journals in which an individual has published may comprise one component of such an assessment. However, its interpretation should be tempered by a scepticism that is borne out of an understanding of the shortcomings of the IF that have been addressed above. Ultimately, there is no substitute for a careful assessment of the individual's work (or a sample of their best work) by a set of peers with impeccable academic credentials who have themselves exerted an incontrovertible impact on the field.
I would like to thank Professors Leslie London and Kit Vaughan for discussions about the IF and comments on the manuscript. I have been selective in referencing the statements in the editorial; I should be happy to send more detailed references to anyone who makes this request.
Alan J Flisher
Division of Child and Adolescent Psychiatry and Adolescent Health Research Unit
University of Cape Town, and
Research Centre for Health Promotion
University of Bergen
1. Zetterström R. Impact factor and the future of Acta Paediatrica and other European medical journals. Acta Paediatr 1999: 88: 793-796. [ Links ]
2. Lankhorst GJ, Franchignoni F. The 'impact factor' - an explanation and its application to rehabilitation journals. Clin Rehabil 2001: 15: 115-118. [ Links ]
3. Kurmis AP. Understanding the limitations of the journal impact factor. J Bone Joint Surg 2003; 85-A: 2449-2454. [ Links ]
4. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997; 314: 498-502. [ Links ]
5. Whitehouse GH. Impact factors: facts and myths. Eur Radiol 2002;12: 715-717. [ Links ]
6. Campbell P. Escape from the impact factor. Ethics Sci Environ Polit 2008; 8: 5. [ Links ]
7. Bloch S, Walter G. The Impact Factor: time for change. Aust NZ J Psychiatry 2001; 35: 563-568. [ Links ]
8. Williams G. Should we ditch impact factors? BMJ 2007; 334: 568. [ Links ]
9. Hobbs R. Should we ditch impact factors? BMJ 2007; 334: 569. [ Links ]
Corresponding author: A J Flisher (firstname.lastname@example.org)