Is it time to end it?
Yes. It’s as worse-than-worthless as the climate models, in that it provides far too many false positives. And as we saw with Climategate, it’s too easy for it to become “pal review.”
13 thoughts on “Peer Review”
Comments are closed.
If I were ready to be slaughtered, I think I’d be a scared cow. Who wouldn’t be?
I was about to comment on the article, till I saw it was 208 days old. I agree with much that was said, and I was appalled that someone in the comments spent so many words defending what is completely broken. Peer review is at best a way to cut down the size of the editor’s slush pile. At worst, it is a gruesomely efficient way to make sure that no thought that would sully academia’s political orthodoxy might be given a hearing.
The true gold standard of Science has been known for 350 years. Experiment and Replication! It’s just regarded in academia now as being too inconvenient to one’s position within academic hierarchies to bother with.
Oh, yes, end it! Stop people from freely running their journals however they want!
But how? I say the answer is arXiv.org. Specifically, change the terms of service on arXiv so that anyone who submits a paper to arXiv.org agrees to forcefully interfere with their colleagues’ peer reviews.
“Oh, yes, end it! Stop people from freely running their journals however they want! ”
Uhhh, …Bob? The whole point of the article is that this is already happening. The article’s fault is not opposing peer review as “the gold standard”, but in not mentioning replication. That is the true means of making sure knowledge is really advancing, instead of chasing its own tail.
From 1650-1950, Science built a good reputation for getting closer and closer to reliable knowledge. In the meantime, academia as a whole developed a reputation outside academia of whoring their opinions to whatever government gave them the most money and power, from Henry VIII’s divorce onwards.
It is becoming more and more obvious that getting shut of peer review is *one* of the things needed to cleanse science of the political influence that has crept in. In the 1920s it began with the “Social Sciences”. It moved onwards into Psychology and Psychiatry.
In 1943 Kanner published work showing definitively that autism had physical effects on the growth of the skull in early life and a history of family descent, ….in short, autism was caused by genetic inheritance. Then Psychiatrists from Germany showed up after the war, touting what Bruno Bettleheim came to call, “The Refrigerator Mother theory” of Autism. Academia rallied to them, and made clear what would happen to researchers who talked about genetics influencing human behavior. Kanner recanted by 1948, and kept his position. Those who refused the new diagnosis too often did not keep theirs, with rumors that they were racist rumbling in the background.
It was *all* done through peer review, whether of funding or publishing and the unwillingness to publish something by fear that doing so might get an editor and his reviewers called “racist”, because it gave credence to the idea that genetic inheritance might cause behavioral differences. By 1969 I finally showed enough old papers to a psychiatrist that he admitted privately my “wiff of autism” was probably genetic. Would he help? No, he had a family to feed.
Bettleheim died in 1991. People started admitting that results he supposedly obtained were a fraud on people who had never been autistic in the first place. People started re-reading the old research from before 1948, and replicating it, and editors would publish it. Then came Wakefield’s vaccine fraud, and the 40 years we’d lost to peer-reviewed pretense are still costing us the lives of children who their mothers won’t immunize.
While this was happening “Atari Democrats” realized the power of the federal monopsony on science funding. One in particular got behind the issue of climate. he made sure as Chair of congressional oversight committees that peer reviewers knew his views, and what he could do for their institutions, or not, as he chose.
When he became vice-president he got the traditional VP baliwick of Office of Science and technology policy to oversee. Now he was setting administration Science policy, and affecting who got funded even more. Yes, peer-review became the tool of those who are academics first, and scientists a long way second.
Getting rid of peer review alone is not enough, but is a necessary part of reasserting the scientific method. We must also break the monopsony over the funding of science by governmental agencies. Only when the government can be defied by results obtained through people who can then remain in their scientific profession will we restore as much faith in the scientific method as existed in 1950.
Another example of govt. money corruption. Once taxpayers are on the hook it only grows. Only Trump is talking about eliminating govt. departments (although he of course has back peddled a bit.)
Hmm are you are advocating Skeptical Review? Only contrarians need apply.
Maybe these journals should do something like decadal science surveys but have an endowment that allows them to hire contractors to reproduce a certain number of influential papers.
” …but have an endowment that allows them to hire contractors to reproduce a certain number of influential papers.”
Good idea! Better would be that every research grant include an escrow set-aside amount for a replication by the publisher’s contractor, of the experiment, whether it is successfully replicated or not. Replication *should* cost less, because every paper should include enough description of what was done to proceed without initial mistakes and experimental equipment design.
You might be a leftist if you defend peer review – except as it was practiced by the Catholic Church before the the Bible became open source.
I got a paper to review earlier this year. I’d written a paper using a vaguely similar experimental technique, though the particular science was not my area of expertise by any means. What they were trying to do was easy enough to understand, however, so the I tried to use the authors’ equations, to get a feel for how they worked. Got the answers they did, but then that didn’t match experimental results from a paper I’d hunted down for background reading on the subject. Found the authors had missed an important bit of science that rendered their experimental results mostly irrelevant to their hypothesis, among other things, and rejected the paper.
I spent, perhaps,15-20 hours of time on the paper. Peer review can work, but most scientists probably don’t want to put in the necessary time; and that doesn’t even touch on the ethical problems of pal review.
Part of the problem with peer review is that you simply can’t easily replicate massive studies. I can’t just drop my own work and redo somebody else’s drug trials, for example. And if the authors say P < 10^-6, I don't have a lot of recourse but to take them at their word.
Peer review can't work with dishonest papers, either. If an author fudges data, it could be almost impossible to ferret that out.
Because of all these things, and others, peer review has become sort of a "sanity check," trying to spot major blind spots. For that purpose, if done well, it can still work. But passing peer review doesn't mean the results of a study is "true and accurate." The standard of truth is replication, but it's not easy (read: it's virtually impossible) to get money to redo a study that came out last month.
IMHO, the best influence that the current monopsony of government funding could have would be for a decree from the WH that *all* science grants include X amount for replication, whether by publishers or anyone else that does it first.
Excuse me, …I should have said that …”the rule would be “X” *percentage* of the grant, whatever its amount was, …
In economics it is not uncommon for a peer-reviewed paper to appear two years after it was accepted. By then everyone who cares will already have read it as an online working paper. Academia is broken.
Done correctly peer review will be looking at:
– is the procedure described in sufficient clarity to (at least in principle) allow replication?
– is the procedure reasonable, or is there a specific shortcoming? Could a factor neglected by the researcher account for the results?
– do the conclusions follow from the data? Some of the post reviews published on WUWT highlight some gross errors in statistical analysis that a proper peer review should have caught.
Scientific orthodoxy should not enter into the process. Whether the data is reasonably correct is impossible to determine except by independent replication.