ASM Addresses the Reproducibility Crisis in New Academy Report
“Promoting Responsible Scientific Research” is the title of a new report just released by the American Academy of Microbiology, a component of ASM. It grew out of an Academy colloquium held last October to tackle an issue that is unfortunately becoming well known both inside and outside scientific circles—the lack of rigor in science. I am delighted that the Academy and ASM are taking on this difficult issue and am grateful to all the participants, the Academy steering committee, and especially to Dr. Arturo Casadevall of Johns Hopkins University, who chaired the colloquium.
A core concept in scientific research is the ability to replicate empirical results. This concept dates back to the birth of the experimental method itself. The Accademia del Cimento (Academy of Experiment) was founded in Florence in 1657 by Galileo’s students and published the first scientific manual of experimentation for data collection and methodological standardization. The motto of the Accademia was provando e riprovando (trying and trying again), emphasizing the importance of replication for scientific experiments.
Fast forward to the present, where scientific discovery proceeds at an impressive pace, and we find that in many instances published research results cannot be replicated (Begley CG, Ellis LM. Raise standards for preclinical cancer research. Nature 483:531-533, 2012; Prinz F et al. 2011. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discovery 10:712, 2011). The causes for the replication crisis have been examined, revealing a complex scenario with multiple determinants (Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature 505:612-613, 2014) ranging from sheer sloppiness (which is inexcusable) to the almost Twitter-length brevity imposed by many glamorous journals on what is allowable in the materials and methods sections of papers. Other culprits include selection bias whereby only positive results are published and a hypercompetitive scientific environment, which forces authors toward sensationalism in reporting their findings. It is important to note here that I am not talking about fraud. That is a whole different issue.
The Academy in its report analyzed these issues and more before setting out 12 recommendations organized into 6 interconnected key areas as the basis for restoring rigor to scientific practice:
- The primacy of rigorous evaluation criteria to recognize and reward high-quality scientific research
- The importance of training in appropriate statistical approaches and in general
- The need for open data as the cornerstone for the scientific enterprise
- The elimination of publication bias by encouraging the publication of negative results
- The establishment of common criteria among journals for retraction criteria to ensure consistency and transparency
- The need for strengthening integrity oversight and training
There are many nuggets of gold in the Academy recommendations, and I encourage all bLog Phase readers to take a look at this short and well-written report. You can read a copy online or request a print copy at ASM; you should also read the mBio editorial on this topic, which you can find here. I would like to highlight a few points.
First, I am proud to report that at ASM we have already taken our own advice. The colloquium report identifies the journal impact factor (JIF) mania as a detrimental influence on researchers. As one of those who thought of and led the San Francisco Declaration on Research Assessment (DORA), I could not agree more. ASM has already stripped its journals and website of all references to JIF, making official our conviction that JIF is a flawed metric for assessing the quality of research and of researchers. We believe that it encourages irresponsible behavior and is not in line with ASM principles. You can read more about the ASM decision to ban JIFs here.
Second, I would like to second the report’s recommendations on data sharing and open data. This must be a cornerstone of all efforts to respond to the reproducibility and rigor issue. The research community needs ready access to the data underlying any experimental figure, not only to judge the validity of the approach and the reported results, but also because detailed data forms a new corpus of scientific knowledge that could accelerate discoveries in a rigorous way. I am convinced that open data associated with peer-reviewed articles would be an invaluable resource for the community. On this front, we should pay attention to what my friend Brian Nosek is doing at the Center for Open Science. Through application programming interfaces (APIs) and other software, Nosek is helping to bring transparency and data availability to the whole community. This could be key to solving the problem of insufficient rigor in science.
Third, ASM already has a strong education program on scientific standards, backed up by valuable resources for members, including videos and articles in the The Academy report calls for a boost in training opportunities, and I look forward to implementing that recommendation. There is so much more that we can do in this area. The ASM Education Department has scheduled a strategic planning retreat for the end of September under the leadership of Amy Chang and Mary Sanchez-Lanier. I’m expecting to hear a lot more from them about what ASM can do on this front.
Finally, let me comment on ASM’s role in shoring up responsible conduct in research. The Academy’s report contains strong recommendations on this topic, and I am pleased to announce that ASM Journals, under the leadership of the indefatigable Barbara Goldman and Tom Shenk, will soon launch a portal on ethics and the responsible conduct of research. It will be an invaluable resource for scientists looking for guidance on all various aspects of research conduct. It will also bring new strength to ASM’s journals, bolstering our efforts to publish outstanding science. This goes to the heart of our journal operations. ASM is a high-quality publisher, focused on presenting rigorous and important science, without chasing impact factors and glitz. Yet we want to help our authors succeed in publishing solid science without making the editorial experience unpleasant or adversarial. This is central to our strategic plan that ASM is “All about you” and that ASM members are at the center of all that we do. This applies especially to our publishing operations, which are here to help researchers present their best science in ASM journals. The new ethics and standards portal will offer the kind of specific advice that authors need in today’s accelerating scientific world. We will be beating the drums loudly at ASM when the portal opens, so stay tuned.
As our science grows more complex and more sophisticated, rigor and reproducibility become all the more essential. Science thrives on openness, rigor, and self-correction. ASM intends to be at the forefront on this issue and for showing us the way forward, so we must thank the Academy of Microbiology!