As once the great George Parshall noted, and rightly so, doing good research and bad research costs about the same amount of money. Once research funds, in one way or another, become available to an individual, there is little control over how wisely and productively the money is being spent. In academia, quality control comes into play later, after the grant finances have been used up to yield the end product, i.e., research results for dissemination in the form of a scholarly report. The name of that quality control is peer review.
Peer review has long been in deep crisis, see, for example, this Forbes article published in 2015. As Geoffrey Kabat, the author of the article, points out, "The dissemination of wrong or plain meaningless findings in scientific journals is costly."
Very true indeed. A recent example of how damaging disseminated misleading results can be is the Cassava Simufilam scandal and FDA’s controversial approval of Biogen’s Aduhelm. The developments of both medications to treat Alzheimer's disease take roots in the currently highly questioned peer-reviewed Nature article published back in 2006. This paper that reported likely fabricated data has entailed, among other consequences, the following.
- Allocation of a tremendous amount of research funds to Alzheimer’s research based on the bogus 2006 publication. Just the American National Institutes of Health (NIH) has devoted $1.6 billion of research funding to such research projects in 2022 alone, nearly half of the federal funds devoted to Alzheimer’s as a whole;
- The unprecedented 14.5% rise in Medicare premiums due to Aduhelm, priced at $56,000 a year; and
- As aptly noted by Bloomberg, "The Alzheimer’s drug approval mess has left millions of people in limbo."
One might argue that publishing a misleading report in an area of substantially less immediate importance is unlikely to have consequences that are nearly as dire as those described above. Shen's flawed publication is improbable to cause even remotely comparable losses. True. However, every disseminated bad piece of work pollutes the scientific literature and has a negative impact on the research culture and mentality, things that are hard to put a dollar sign to yet are paramount for future advances of science.
Peer review has long been in deep crisis, see, for example, this Forbes article published in 2015. As Geoffrey Kabat, the author of the article, points out, "The dissemination of wrong or plain meaningless findings in scientific journals is costly."
Very true indeed. A recent example of how damaging disseminated misleading results can be is the Cassava Simufilam scandal and FDA’s controversial approval of Biogen’s Aduhelm. The developments of both medications to treat Alzheimer's disease take roots in the currently highly questioned peer-reviewed Nature article published back in 2006. This paper that reported likely fabricated data has entailed, among other consequences, the following.
- Allocation of a tremendous amount of research funds to Alzheimer’s research based on the bogus 2006 publication. Just the American National Institutes of Health (NIH) has devoted $1.6 billion of research funding to such research projects in 2022 alone, nearly half of the federal funds devoted to Alzheimer’s as a whole;
- The unprecedented 14.5% rise in Medicare premiums due to Aduhelm, priced at $56,000 a year; and
- As aptly noted by Bloomberg, "The Alzheimer’s drug approval mess has left millions of people in limbo."
One might argue that publishing a misleading report in an area of substantially less immediate importance is unlikely to have consequences that are nearly as dire as those described above. Shen's flawed publication is improbable to cause even remotely comparable losses. True. However, every disseminated bad piece of work pollutes the scientific literature and has a negative impact on the research culture and mentality, things that are hard to put a dollar sign to yet are paramount for future advances of science.
Furthermore, the ultimate goal of scientific research is not only to avoid or minimize losses but also to generate gains and benefits to the society. Here, one might further argue that Shen's work is just a mechanistic study, not an invention that can have a strong direct impact on the development of a valuable commercial product. This argument, however, is hardly convincing. The groundbreaking DuPont hydrocyanation process for making nylon 66 would not have been commercialized without truly innovative, meticulous mechanistic studies of top-notch scientists. Among their fundamental findings that have served as a big part of the foundation of modern homogeneous catalysis as a whole, are the seminal Tolman electronic parameters and ligand cone angles that have been guiding organometallic, inorganic, and catalysis research for over 50 years and are discussed in every textbook on these subjects.
One might also say that even the most scrupulous reviewer cannot detect research frauds of certain sort, such as masterfully falsified images and doctored spectra, faked elemental analysis data, etc. Likewise, there is no way for a reviewer to spot irreproducible experiments, as it is not reviewers' job to check submitted procedures. (I have done that as a reviewer on a few occasions, though.) True, there is no reason whatsoever to suspect that those who reviewed and recommended for publication the aforementioned notorious 2006 Nature report did not do their job properly, as recognizing that some of the images in the manuscript were manipulated was merely impossible.
However, what if a reviewer overlooks some obvious flaws with a manuscript (s)he was asked and agreed to evaluate? Such as clearly unrealistic X-ray bond distances, wrong kinetic modeling, inadequately presented spectra, incorrect use of a widely known methodology leading to questionable interpretations, omitted citations, dubious statements, unsupported conclusions, etc. All of these easily recognizable problems were present in the Shen manuscript. Nevertheless, the paper passed the peer review process, was positively evaluated by the handling editor, and finally published.
It remains unknown, at least to me, if the reviewers did not take pains to more-or-less carefully read Shen's manuscript or were merely incompetent, or both. Neither do I know if the handling editor -who selected the reviewers- read the paper before accepting it for publication. What is clear, though, is that the reviewers and the editor did not do their job.
One might also say that even the most scrupulous reviewer cannot detect research frauds of certain sort, such as masterfully falsified images and doctored spectra, faked elemental analysis data, etc. Likewise, there is no way for a reviewer to spot irreproducible experiments, as it is not reviewers' job to check submitted procedures. (I have done that as a reviewer on a few occasions, though.) True, there is no reason whatsoever to suspect that those who reviewed and recommended for publication the aforementioned notorious 2006 Nature report did not do their job properly, as recognizing that some of the images in the manuscript were manipulated was merely impossible.
However, what if a reviewer overlooks some obvious flaws with a manuscript (s)he was asked and agreed to evaluate? Such as clearly unrealistic X-ray bond distances, wrong kinetic modeling, inadequately presented spectra, incorrect use of a widely known methodology leading to questionable interpretations, omitted citations, dubious statements, unsupported conclusions, etc. All of these easily recognizable problems were present in the Shen manuscript. Nevertheless, the paper passed the peer review process, was positively evaluated by the handling editor, and finally published.
It remains unknown, at least to me, if the reviewers did not take pains to more-or-less carefully read Shen's manuscript or were merely incompetent, or both. Neither do I know if the handling editor -who selected the reviewers- read the paper before accepting it for publication. What is clear, though, is that the reviewers and the editor did not do their job.
Now the question arises as to how quality control could be enforced in academic research and publishing
Here is an idea for how scholarly journal editors and reviewers could be made more accountable for mishandling manuscripts. If a bad paper is retracted, the names of the handling editor and reviewers whose negligence resulted in the publishing of that paper should be disclosed in the retraction notice.
