Scientific Reproducibility: Raising the Standards for Biomedicine

1/3/14

(Page 2 of 2)

investigators would share privately the high-profile work that no-one else could reproduce. However, the issue has probably been amplified and become more public as a result of the convergence of several factors. This includes the plethora of new ‘top tier’ journals spawned over recent decades: Nature alone now has approximately 20 subsidiary journals carrying the ‘Nature‘ label. This means that more experiments are getting the visibility, and scrutiny, that comes with publication in a “top-tier” journal.

The proximity of modern biomedical research to increasingly informed, vigilant and vocal patient-advocate groups, also serves to bring this into sharp focus. These groups expect research ‘breakthroughs’ to be quickly translated into improved human health, an entirely reasonable expectation given that this is funded by the public purse. Many more studies are being highlighted by media outlets, which, again, increases visibility and scrutiny of the underlying research. The attention provided by the media, although important, is not all new. Even as a practicing clinician over 25 years ago I would dread Monday morning clinics when another ‘breakthrough’ had been announced in newspapers over the weekend, and I would have to explain to patients that XYZ finding was still years away from reaching real-world patient care, if ever.

What is driving this?

It is important to acknowledge that the vast majority of researchers want to see their work translated into something that benefits humankind. Equally, industry researchers, journal editors, investors, governments and certainly patients want to see important discoveries made, and turned into improved treatments. As a result, medical research has delivered tremendous benefits to human health: this represents a wonderful return on investment and is unprecedented in human history. And biomedical research will continue to deliver. But there is room for tangible improvements that will provide further benefit.

The problem we have is not a problem of scientific fraud, as some have suggested. It is not a failure of the scientific method. Rather it is a consequence of the lack of rigorous application of the scientific principles in an intensely and increasingly competitive research environment in which scientists are scrambling to get their share of a national research budget that’s shrinking. It is also the consequence of a system that currently turns a blind-eye to a lack of rigor.

What can be done to improve this?

It is instructive to consider how the clinical-trials process has evolved over the last several decades. While there is still room for improvement, now there is an expectation that good quality clinical trials will be performed by investigators who don’t know which patients are getting the experimental treatment and which are in the control group during the course of the study. These studies have appropriate controls, have a pre-specified hypothesis, ensure data is not excluded at the whim of the investigator after the fact, and utilize rigorous statistical analyses.

This is not the current situation in the majority of preclinical biomedical research. Simply addressing investigator bias in preclinical studies would represent a major step forward. However there appears a general reluctance to incorporate this level of objectivity into experimental design of preclinical research, including among the leading scientific journals.

What is being done, and where will this take us?

Although some scientists are clearly able to self-censor and impose a level of experimental rigor to ensure the robustness of their results, many appear unable or unwilling to impose the same level of scientific self-control. As a result, funding agenciesjournals and others are initiating changes to improve data quality. Somewhat disappointingly, although host institutions enjoy the kudos that come when their investigators get results published in top journals, these same institutions have not yet demonstrated a willingness to establish or enforce scientific standards among their investigators. This seems short-sighted as institutions that proactively take the lead will enjoy a distinct advantage as places known for producing high-quality, reproducible research.

It is reasonable to ask what degree of irreproducibility is acceptable as investigators work at the boundary of knowledge and push forward into the unknown? Clearly, we must be prepared to tolerate some level of inaccuracy, in fact it is sometimes the case that a false result gathered in the heat of competition can drive further advances as the error is corrected by other scientists later. But unlike the current reality, it is reasonable to expect that these false results are the exception rather than the rule. The majority of scientific discoveries in the biomedical sphere should be sufficiently rigorous and robust to allow other investigators to build on that work and move the field forward. Hopefully the changes instituted as a result of this debate will help further strengthen our discovery processes.

C. Glenn Begley is Chief Scientific Officer and Senior Vice President for Research and Development at TetraLogic Pharmaceuticals in Malvern, PA. Follow @

Single Page Currently on Page: 1 2 previous page

By posting a comment, you agree to our terms and conditions.

  • Ken

    This is a really nicely written and balanced article. I believe a key issue, though, is that of the incentives and disincentives for PIs around the issue of reproducibility and how these are weighed/balanced against one another: (1) prestige; (2) reputation; (3) expense relative to resources. One, perhaps simple-minded, notion would be to assign a score/index that is published along with a paper with respect to its statistical rigor/design. In principle, all papers should at least meet a minimum threshold. However, I would argue that even within the “universe” of accepted papers, some are far more rigorous than others, and there would be a nice, “crisp” incentive if on the title page of the paper there were a “rigor ranking,” e.g. “AAA” “AAB” etc…This separate evaluation would provide: (1) an incentive to be as rigorous as possible and perhaps make it worth using more resources to do so; (2) conversely, spending fewer resources to get a still publishable study, but perhaps with a somewhat lower “rigor score.” I’m sure there are multiple problems with this framework, but perhaps it would at least give a push in the right direction…

  • Hank

    Indeed well written. I would have liked to learn more though about what is actually being done to mitigate the problem and ideas what could be done like the suggestions of in previous comment. For sure the academic institutes need to get their act together as well as boards and SABs of companies.

  • A Suhrbier

    Scientists are pressured to undertake and publish research that’s popular among scientists, not research that is useful in the real world. See link to online article I published recently

    http://www.abc.net.au/science/articles/2013/11/27/3899981.htm

  • K. Francis

    Well written and really brings attention to something we should all be reminded of daily. There is no single solution but perhaps emphasizing the issue at the undergraduate or high school level may help awareness later in one’s career.