Peer review – net benefit or not?
When I started this post, it was to be about search engines, the problem being that a search is never specific enough, and just about anything can be mixed up in the answer. To illustrate the problem, I thought I would Google "Peer review criticisms", and by doing so, I got 3,570,000 hits. Surely they can't all be relevant. However, curiosity got the better of me, and after an admittedly ridiculously inadequate look at the literature, I decided to post about peer review itself. The question to be addressed is, is peer review helpful to unraveling the secrets of nature, or is it part of the problem? So here are some comments that I picked up.
However, before I start, a different question might be, does peer review do any good? One possibility is that it sends the half-baked back to the author for further baking. It may also be claimed to help emerging scientists to present their ideas in a way that would lead to better understanding of what they are doing. If so, this could be very helpful, however, that only applies to papers sent back for revision. There is also the question of whether papers being sent back for revision truly need it? That someone does not write in the style of the reviewer is beside the point. It also eliminates the "crank" stuff, but herein lies the problem: what happens if an upcoming Einstein is labeled a crank? The current argument is, if it is that good, it will find a place eventually. Perhaps, but will it then be read? However, let us return to the literature.
First, a quote from a source that shall remain anonymous: "Peer review seldom detects fraud, or even mistakes. It is biased against women and against less famous institutions. Its benefits are statistically insignificant and its risks – academic log-rolling, suppression of unfashionable ideas, and the irresistible opportunity to put a spoke in a rival's wheel – are seldom examined." To that I would add, it is most certainly biased against individuals that do not have a University or major institutional address. The individual scientist does not exist, according to the reviewing system. If you do not have a suitable address, you must be a crank. No need to waste time reading the paper. Herein lies another real problem: papers can be rejected by the Editor without peer review, or even without any evidence of having been read, at least past the title.
Now, from www.evolutionnews.org/2012/02/problems_with_p056241.html
This article made six points:
- Good science does not have to be published in the peer-reviewed literature. The examples cited include rejections, and I shall deal with those later.
- The peer-review system wrongly rejects scientifically valid papers, while it wrongly accepts scientifically flawed papers. Personally, I feel that it is too much to expect a peer reviewer to find fraud and you cannot expect it to uncover faulty experimental procedures that are not specified. In this sense, journals are increasingly encouraging methods to be explained as a reference to somewhere else, which in turn is a reference to somewhere else again, and so on. However, it also makes the valid point that peer-reviewing is both time-consuming and expensive, and often excludes people for no good reason.
- Scientific peer-reviewers are not perfectly objective. Again, the performance of the reviewers is questioned, but a very interesting point followed: journals have a very strong economic interest in preserving the current system, and scientists go along with it because it helps them maintain their position. In my mind, this is not a good reason to persist with it.
- The "peer-review card" is often played to silence scientific dissent. This, to my mind is a serious criticism, although more of the scientific elite than the reviewing system.
- Peer-review is often biased against non-majority viewpoints." Denyse O'Leary is quoted as saying: "The overwhelming flaw in the traditional peer review system is that it listed so heavily toward consensus that it showed little tolerance for genuinely new findings and interpretations."
- Not being recognized in peer-reviewed literature does not imply a lack of scientific merit. The simple fact is, in logic, the contrary position is a fallacy in the ad verecundiam class.
In a later post I shall continue on this theme, but before I do, what are your thoughts on this matter?
Posted by
Ian Miller on Aug 16, 2015 9:25 PM Europe/London
