Null Results, Replication Studies and Other Important Science Find a Home
Scientists have long recognized that the findings in peer-reviewed journals don’t always reflect the realities of research, and this mismatch can take many forms. For example, scientific journals are much more likely to publish positive findings—that is, statistically significant results supporting the hypothesis being tested—than negative or null results. The trend is increasing: according to one analysis looking at a variety of scientific fields, about 70% of studies published in 1990 reported positive findings, compared to about 86% in 2007. The imbalance extends to citations, as well: a 2017 meta-analysis by epidemiologists found that studies showing a statistically significant effect were 1.6 times more likely to be cited in future studies.
Many journals also prioritize novelty, quite literally. A 2015 study by Dutch researchers found that use of the words ‘novel,’ ‘innovative,’ ‘amazing’ and ‘unprecedented’ in the titles and abstracts of published papers increased by a factor of 9 between the years 1974 and 2014. Reviewers who evaluate grant applications to the National Institutes of Health (NIH) have been asked to estimate an “innovation” score for projects. The potential for impact or novelty can supersede the importance of the work, which means rigorous research that nonetheless contributes to the field may go unreported.
“Perceived novelty or impact is a filter on what is published at many journals that can exclude important work,” says geneticist Christina Cuomo, Ph.D., an institute scientist at the Broad Institute of MIT and Harvard in Cambridge, Mass.
The prioritization of novelty over rigor may be contributing to yet another problem: a growing replicability crisis. In a 2016 survey of 1,500 scientists, conducted by Nature, more than half reported that they had been unable to replicate their own results, and more than 70% had tried to replicate another group’s findings, and failed.
Although these findings may seem to dim the light on the scientific process, they reflect an increasing focus on transparency in the scientific enterprise. And a growing number of researchers, like Cuomo, see such issues as opportunities to address and correct defects in the publication pipeline.
Actions by scientific societies and publishing houses, including ASM, can help to right wrongs, says Cuomo. Awareness of these biases was part of the inspiration for the addition of Microbiology Spectrum, a new open-access journal that prioritizes science over novelty, to ASM’s stable of journals.
“All rigorous science done in microbiology has a home at Microbiology Spectrum,” says Cuomo, the journal’s first editor-in-chief. The journal welcomes research from every domain of basic, applied and clinical microbiology, says Anand Balasubramani, Ph.D., Microbiology Spectrum’s managing scientific editor. “Our goal is to serve the entire microbiology community,” he says. “The scope is broader than what we might think of in other microbiology journals.”
The editorial board of Microbiology Spectrum will consider submissions of studies that previously may not have had a traditional home in publishing, including findings of primary interest to small subfields within microbiology, large datasets that could serve as a valuable community resource or detailed experimental protocols that might be useful to other researchers. They also encourage submissions from authors of studies that describe robust datasets that contradict previously reported finding, or report negative or null results, in which the data do not support the hypothesis.
“You would want to know if someone had already completed a study similar to one you are contemplating and had generated clear results, even if it didn’t confirm a hypothesis,” says Cuomo.
The journal also welcomes replication studies that may prove useful to the field. Replicability is often described as a hallmark of solid science because scientific results should be reproducible by other researchers doing the same experiments under the same conditions. However, efforts at reproducing results remain rare in the literature, and recent surveys of scientists from a range of disciplines, including microbiology, indicate growing concerns about a lack of reproducibility and a growing “replication crisis.”
The editors of Microbiology Spectrum welcome submissions like these—as well as re-analyses of existing datasets that provide new insights, or technically robust datasets that contradict previous findings. “While many journals have not welcomed replication studies based on [lack of] novelty, demonstrating work can be replicated is essential to scientific progress,” says Cuomo. “Our goal is to continue to think about how Spectrum can fill gaps in the publication process.”
The essential, unwavering requirement for Microbiology Spectrum studies, says Cuomo, is they are well controlled and thoroughly documented, with clear methods and openly accessible data that will enable others to reproduce the studies.
The journal will embrace new, forward-looking practices in other areas of publication as well. To ensure diversity on the editorial board—not only of scientific fields but also of race and geographical location—ASM welcomes self-nominations and applications from researchers around the world, “no invitation necessary,” Cuomo says. “We want to encourage this as a way to have an open and global reach. We’re looking for people who are supportive of this model of publishing.”
ASM has also implemented changes to the submission process to streamline review. Papers reviewed by the 10 other ASM journals receive accelerated review at Microbiology Spectrum, and the editors of those journals can accept submitted papers on the behalf of Microbiology Spectrum, which means authors would not need to start from scratch to submit. “It’s a big experiment from the publication side, and a unique model that hasn’t been tried before,” says Balasubramani.
“There’s a lot of volunteerism in publishing and reviewing, and this is a way to ensure that we’re not wasting efforts,” says Cuomo. “We don’t want to prolong the review process.”
These practices and principles, Cuomo says, arise from a growing recognition of the urgent need to retool the scientific publication process so that it’s both more reflective of the state of science and more representative of the diversity of research studies.
For more updates from ASM Journals and the latest scientific content, sign up to get top cited articles and new notable research delivered straight to your inbox.