COMMENTARY

 

Four proposals for a more reliable scientific literature

 

 

Jason Bosch

Gregor Mendel Institute, Vienna, Austria

Correspondence

 

 

---

Keywords: publishing; reproducibility; retractions; citation

---

 

 

The publication of Richard Harris' new book Rigor Mortis once again brings the issue of poorly conducted science and irreproducible research into focus.¹ This book is the latest in a string of publications which point out flaws in the current scientific system. Earlier examples from the scientific literature include John loannidis' provocatively titled article 'Why most published research findings are false'² and the disclosure that researchers at Amgen could only confirm 6/53 landmark cancer studies³. These articles have suggested and inspired ways to improve the situation that address, among other things, incentives in science⁴, guidelines for better experimental design⁵ and AllTrials greater insistence on the registration and reporting of clinical trials. I think pre-registration would also benefit basic science.

I wish to advance a series of proposals regarding the way in which we publish scientific literature that, I believe, will benefit science by making research rapidly available, easier to search and more reliable.

 

Scientific articles should be short and focused

One should be able to grasp the gist of a paper from its title and abstract alone. However, in a long paper with multiple experiments, many pieces of relevant or interesting information can remain hidden. Often, entire experiments are heavily summarised; compressed into just a few paragraphs or sentences, which can hinder understanding. Over the past three decades, there has been a dramatic increase in the size of the average publishable unit, i.e. the number of pages, figures, references and authors per paper⁶, which makes it more difficult for researchers to gain a true understanding of the paper.

A move to shorter papers will make it easier to identify and judge the contents of a paper and allow authors to clearly explain the experiment and its limitations without worrying about different messages competing for space. Articles would ideally cover a single experiment or a single, carefully defined question. For example, instead of writing a single paper in which one screens for a new molecule, characterises its behaviour and determines its structure, one should publish a separate paper for each step. Having multiple experimental approaches to answer a specific question will increase our confidence in the results but, if experiments appear to contradict one another, may lead to neither experiment being published or only the publication of experiments which favour the authors' hypothesis. Step-by-step publication may help avoid this problem.

I recognise that I am essentially promoting the least publishable unit - the smallest amount of data which can be successfully published - which has been criticised since the early 1980s.⁷ Many of the arguments originally raised against it have already been addressed; diffuse responsibility and unnecessary authorships can both be counteracted by listing author contributions. Criticisms like the 'inflated' number of publications reflect problems with how scientists are evaluated. There are drawbacks to the least publishable unit which should be kept in mind, such as the same data series being used in multiple publications while appearing to be independent sets of data.⁸ These issues can be addressed in other ways, such as publishing and referring back to a data set, and are not general enough to advise against the practice.

Shorter papers could significantly speed up scientific progress. With longer papers, it can be that the first of a series of experiments is completed months or even years before publication. It is quite likely that there are several years' worth of research on a particular topic, relevant to others' ongoing work, sitting unpublished in labs around the world. Rapid publication could prevent researchers from wasting time following dead ends that others have already tried or better inform their approach to a current problem. It also removes the risk that failure of a downstream experiment results in the collapse of the 'story', preventing the publication of earlier, valid results.

Short papers will prove more robust to retractions. A longer paper with more authors means there is a greater chance that one of the authors will do something which leads to a retraction. In such cases, this will have a negative impact on many authors and result in valid experiments being lost as collateral damage. Shorter publications would mitigate these issues.

 

Retractions and corrections should cascade through the literature

Publishers and reference manager software should take advantage of the digital landscape to cascade corrections and retractions. Almost all papers are published digitally and identified by unique tags such as the DOI (digital object identifier). These data can be used to automatically inform researchers of problems in the scientific literature. This is necessary because, despite the importance of such information, the current system does not make it obvious when papers have been retracted.

For example, take the paper by Hunter and Prüss-Ustün⁹ which was published in October 2016 and retracted in May 2017. The web version of the paper links to the retraction notice and the new PDF makes it clear that the paper was retracted. However, the new PDF neither links to the retraction nor does the DOI entry offer such information. Although the Hunter and Prüss-Ustün paper remains cited after its retraction, this fact is neither indicated in previous reference lists nor is this information retrieved by Mendeley with the DOI. If someone had downloaded the paper in the months between publication and retraction, they could, quite easily, never become aware that it was retracted.

This issue could be avoided if reference managers could identify and flag retracted papers using information retrieved from the DOI. Furthermore, digital publishing allows us to follow citations through the literature. This means that papers which cite papers which have been, or later become, retracted could also be identified and flagged. This would inform readers that there are problematic references cited and warn them to look closely at information coming from retracted or corrected sources. Together, this should help to prevent the spread of incorrect information and thereby increase our confidence in the scientific literature.

 

Journals should be required to publish replications of their articles

Although novelty is essential for science to advance, science builds on work that has come before; thus, replication is equally essential to ensure reliability. Therefore, we should not rely on a single publication and place an undue emphasis on novelty. This emphasis leads to absurd situations in which attention-grabbing work is published in a high-profile journal, while a failed replication of that same work is not considered because of a lack of novelty. This results in an asymmetry, in which novel but incorrect research can have a higher impact than less original but correct research. One unfortunate consequence of this asymmetry is the reluctance of scientists to do the important work of replicating previous studies.

I would like to propose that journals should have an obligation to publish scientifically sound replications of work that they have previously published. In addition, building on my previous point, replications should also be linked. Linking would allow readers to see whether someone has attempted to replicate a paper and the result. Scientists being aware that journals will publish replications should help address the problem in which negative results are seldom published, which is important because simulations have shown that publication of negative results is important to prevent incorrect results being accepted as fact.¹⁰ My first three proposals combined would result in a much clearer view of the reliability of a specific piece of knowledge.

 

We should separate data-generating articles from storytelling articles

With an emphasis on short, focused papers and abandoning the idea of complete 'stories', how do we advance conceptually? The answer is by separating the scientific stories from their constituent parts. By complete stories, I mean a series of different but linked experiments which follow logically and build on one another to come to a combined conclusion all within a single paper. As the idea of publishing stories is currently widespread in science, I have no doubt that this proposal will be a controversial notion. Stories have limits, however, and it does not serve the interests of science for researchers to publish only when they believe that they can construct a story or to force results into a story before there is sufficient evidence to support one.

Short, focused articles - ideally linked to replications - will create blocks of data which can stand on their own. It is the function of review or 'story' papers to collect these blocks and combine them into a coherent narrative. The same data will be published as currently but the difference is that the data will come faster and enable the synthesis of up-to-date results from multiple labs instead of many narratives built on incomplete data.

Our narrative explanations of phenomena may change as new data become available, but the original data should remain valid regardless of the interpretation. This alone suggests that it might be wise to separate the data from the narrative as one is likely to remain valid much longer than the other.

The freedom to publish without the limitations of a story will open the way for sharing many more observations. There are journals being established which are supporting such approaches. BMC Research Notes publishes 'scientifically valid research outputs that cannot be considered as full research or methodology articles', and Matters is a journal where 'Stories can wait. Science can't.' In fact, in discussions with co-workers I have been told that this suggestion does not go far enough. Some believe that data papers could be entirely replaced by structured databases, leaving only review or outlook papers.

 

Conclusion

To address concerns about the accuracy and reproducibility of scientific publishing, I have presented a series of proposals which will improve the quality and reliability of scientific publications. Short publications will present data as building blocks which can be combined to form scientific narratives. By keeping papers focused, ensuring replications are published and dynamically linking replications and papers, we ensure that we treat results as pieces of data rather than individual stories. By cascading replications and corrections along a chain of citations, we can build a higher level of confidence in what is published. These proposals will require action from many different parties, but I believe that the benefits of this new system will outweigh the costs.

 

Acknowledgements

I thank my colleagues, particularly Angelika Czedik-Eysenberg, for discussions which led to this article. I also thank Angelika Czedik-Eysenberg, Nancy R. Gough and J. Matthew Watson for providing helpful feedback on this manuscript.

 

References

1. Harris R. Rigor mortis. New York: Basic Books; 2017.         [ Links ]

2. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8), e124, 6 pages. http://dx.doi.org/10.1371/journal.pmed.0020124        [ Links ]

3. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012;483:531-533. http://dx.doi.org/10.1038/483531a        [ Links ]

4. Nosek BA, Spies JR, Motyl M. Scientific utopia II: Restructuring incentives and practices to promote truth over publishability. Perspect Psychol Sci. 2012;7:615-631. http://dx.doi.org/10.1177/1745691612459058        [ Links ]

5. Masca N, Hensor E, Cornelius V Buffa F, Marriott H, Eales J, et al. RIPOSTE: A framework for improving the design and analysis of laboratory-based research. eLife. 2015;4, e05519, 27 pages. http://dx.doi.org/10.7554/eLife.05519        [ Links ]

6. Cordero RJB, De León-Rodriguez CM, Alvarado-Torres JK, Rodriguez AR, Casadevall A. Life science's average publishable unit (APU) has increased over the past two decades. PLoS ONE. 2016;11, e0156983, 14 pages. http://dx.doi.org/10.1371/journal.pone.0156983        [ Links ]

7. Broad WJ. The publishing game: Getting more for less. Science. 1981;211:1137-1139. http://dx.doi.org/10.1126/science.7008199        [ Links ]

8. Dupps WJ, Randleman JB. The perils of the least publishable unit. J Cataract Refract Surg. 2012;38:1517-1518. http://dx.doi.org/10.1016/j.jcrs.2012.07.020        [ Links ]

9. Hunter PR, Prüss-Ustün A. Have we substantially underestimated the impact of improved sanitation coverage on child health? A generalized additive model panel analysis of global data on child mortality and malnutrition. PLoS ONE. 2016;11, e0164571, 17 pages. http://dx.doi.org/10.1371/journal.pone.0164571 [RETRACTED]        [ Links ]

10. Nissen SB, Magidson T, Gross K, Bergstrom CT. Publication bias and the canonization of false facts. eLife. 2016;5, e21451, 19 pages. http://dx.doi.org/10.7554/eLife.21451        [ Links ]

 

 

Correspondence:
Jason Bosch
Email: jason.bosch@gmi.oeaw.ac.at

PUBLISHED: 27 Mar. 2018