LEADER

DORA: The San Francisco Declaration on Research Assessment

Butler-Adam, John.

During the annual meeting of the American Society for Cell Biology held in San Francisco in December 2012, a group of leading scientists, in concert with a group of editors and publishers of scholarly journals, produced what came to be known as the San Francisco Declaration on Research Assessment, or DORA (http://am.ascb.org/dora/files/SFDeclarationFINAL.pdf). The Declaration was released in May this year, an event marked at the time by Bruce Alberts, then the Editor-in-Chief of Science, in an editorial¹ entitled 'Impact factor distortions'.

DORA aims to contextualise journal impact factors, indicating that they were originally and specifically developed to allow journals to understand their own impacts in the world of science and scholarship, and not as means for assessing the performance of individual researchers or institutions of higher learning. To achieve this aim, DORA proposes 18 recommendations to funders, institutions, researchers, publishers and the suppliers of metrics. The recommendations include phasing out metrics at the level of journals and replacing them with article-level measures, being transparent and straightforward about assessment, and judging articles on the basis of real content.

In his editorial, Alberts pointed out that the increasingly widespread use of the impact metrics is highly damaging - to researchers, who are forced to play the numbers game; to journals, like Science, as they come to be overloaded with inappropriate submissions from scientists anxious to have their work published in high impact journals; and to scientific endeavours generally, because they drive research in directions that will earn the researchers (and their universities) high rankings. In an article published in Trends in Ecology and Evolution, Fischer et al.² offer the view that '[t]he modern mantra of quantity is taking a heavy toll on two prerequisites for generating wisdom: creativity and reflection'. They go on to pose the rhetorical question: 'Is it possible to obtain and communicate deep insights via 'twitteresque' research sound bites?'

Almost simultaneously with the Science editorial, Nature carried a news blog, submitted by Assistant News Editor Richard Van Noorden, in which he raised the DORA assertion that impact factors, turned away from their original purpose, have changed scientists' incentives 'leading them to be rewarded for getting into high-impact publications rather than for doing good science'³. Van Noorden also cites Stefano Bertuzzi, the Executive Director of the American Society for Cell Biology as saying '...we created this mess, this perception that if you don't publish in Cell, Nature or Science, you won't get a job'. And although Nature has not endorsed DORA, its Editor-in-Chief, Philip Campbell, has made it clear that the journal has published numerous editorials, from as far back as 2005, which are critical of misuses and excesses in the application of impact factors.

It remains true that, even were impact measures tamed, we would all continue to have subjective assessments of the quality and status of scholarly journals. Yet without the measures as they are currently used, the emphasis would have to move to articles themselves and the production of quality rather than quantity and 'metrics'.

The DORA site indicates that almost 9000 individuals have so far signed the Declaration of whom, interestingly, 94% are from disciplines in the natural sciences and only 4% from the humanities and social sciences - with almost 85% of the signatories being from Europe and North and Central America.

The question to be asked, of course, is whether this ground swell of support for DORA - from researchers, editors and publishers - will result in a cultural change when it comes to using (and misusing) impact metrics. Apart from gradually but firmly ingrained habits and, in some cases, entrenched institutional policies, the main obstacle in the DORA drive is likely to be found in the great 'ranking companies'.

Thomson Reuters (who run the Web of Knowledge, Web of Science (ISI) and Incites), Times Higher Education, Quacquarelli Symonds (who run the Worldwide University Rankings), the Academic Ranking of World Universities (formerly the Shanghai Jiao Tong University Rankings), Elsevier (who run Scopus), the CWTS Rankings (Leiden) and, more recently, the Centre for World University Rankings (in Jeddah, Saudi Arabia) all make (or stand to make) considerable profits from their ranking activities, some appealing not just to universities, but also to potential undergraduate and graduate students, and parents, in relation to institutional status and the selection of suitable places to study.

In other words, the misuse of impact metrics, and of emerging factors, has become the basis for major business operations, which is why the generators and retailers of the data are likely to consider DORA to be a bit of an academic pipe dream. On the other hand, if sufficient numbers of researchers were to reject the present system (and convince their universities to do so too), a change in the ways in which quality is determined might actually become possible. And there are reasons to think that there could well be enough scholars who would signal their support for DORA. As Stephen Curry, a structural biologist at Imperial College London, put it: 'I'm sick of impact factors and so is science'.

References

1. Alberts B. Impact factor distortions. Science. 2013;304(6134):787. http://dx.doi.org/10.1126/science.1240319        [ Links ]

2. Fischer J, Ritchie EG, Hanspach J. Academia's obsession with quantity. Trends Ecol Evol. 2012;27(9):473. http://dx.doi.org/10.1016/j.tree.2012.05.010        [ Links ]

3. Van Noorden R. Scientists join journal editors to fight impact-factor abuse. Nature News Blog. 2013;498(7453).         [ Links ]