General Science, Media Criticism, Political Commentary, Science And Society, Social Commentary

Is Science Self Correcting?

13 Comments

Scientists wish, but it’s not. At least in the short term:

Academic scientists readily acknowledge that they often get things wrong. But they also hold fast to the idea that these errors get corrected over time as other scientists try to take the work further. Evidence that many more dodgy results are published than are subsequently corrected or withdrawn calls that much-vaunted capacity for self-correction into question. There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think.

Various factors contribute to the problem. Statistical mistakes are widespread. The peer reviewers who evaluate papers before journals commit to publishing them are much worse at spotting mistakes than they or others appreciate. Professional pressure, competition and ambition push scientists to publish more quickly than would be wise. A career structure which lays great stress on publishing copious papers exacerbates all these problems. “There is no cost to getting things wrong,” says Brian Nosek, a psychologist at the University of Virginia who has taken an interest in his discipline’s persistent errors. “The cost is not getting them published.”

Yup. And peer review is not much of a quality control, when it becomes “pal review.”

This partially explains why there’s so much crap science in climate research. Probably for nutrition as well.

Read the whole thing. Undue faith in the current process of evaluating and correcting junk science will be appropriately reduced.

Oh, and then there’s this:

Statisticians have ways to deal with such problems. But most scientists are not statisticians.

Professor Hockey Stick certainly isn’t. Which is why it was so easy for people who do understand statistics to publicly pull his Nobel-winning pants down. And of course, Paul Krugman isn’t, either.

[Update a couple minutes later]

OK, one more excerpt, just to demonstrate why you should RTWT:

The idea that there are a lot of uncorrected flaws in published studies may seem hard to square with the fact that almost all of them will have been through peer-review. This sort of scrutiny by disinterested experts—acting out of a sense of professional obligation, rather than for pay—is often said to make the scientific literature particularly reliable. In practice it is poor at detecting many types of error.

John Bohannon, a biologist at Harvard, recently submitted a pseudonymous paper on the effects of a chemical derived from lichen on cancer cells to 304 journals describing themselves as using peer review. An unusual move; but it was an unusual paper, concocted wholesale and stuffed with clangers in study design, analysis and interpretation of results. Receiving this dog’s dinner from a fictitious researcher at a made up university, 157 of the journals accepted it for publication.

Dr Bohannon’s sting was directed at the lower tier of academic journals. But in a classic 1998 study Fiona Godlee, editor of the prestigious British Medical Journal, sent an article containing eight deliberate mistakes in study design, analysis and interpretation to more than 200 of the BMJ’s regular reviewers. Not one picked out all the mistakes. On average, they reported fewer than two; some did not spot any.

And yet some people think that we should base multi-trillion-dollar policy decisions on this crap.

13 thoughts on “Is Science Self Correcting?”

    1. Uhhh… in markets self interest is what does the self correcting. Often what is self corrected in markets is the lack of humility or honesty.

      It’s not compassion or some abstract idea of justice that drive customers to write bad reviews.. and others to read them.

      1. Absolutely right. So the key is using self interest rather than rewarding it when it is used wrong.

  2. I think these “stings” miss the point. Instead of seeing if the journals will publish nonsense, they should be seeing if the journals will publish retractions, corrections or debunking.

    There was a time when “tearing down” established work was a normal way for grad students to enter a field.. problem is, you actually have to be skilled to do that.

    1. usually, when I find the biggest errors in papers or textbooks, it’s when I try to recreate the work.

      When you look at something clean sheet, and build your own analytical code, or reductions
      and you get broadly different results from the “Big Names”, well that’s when it gets interesting
      because either one of you has made a code/math error or someone has a starting assumption
      that’s just balls.

      I laughed so hard when Reinhart and Rogoff were overturned by some grad student at UW.
      All the big shots in DC were quoting them, like they were the word of the lord, and yet,
      they had fundamental errors in their spreadsheets.

  3. Who said the scientific process was limited to peer review? The errors in Mann’s methodology were discovered and corrected (though those corrections made little difference to the conclusions).
    A cynic could argue that Newton got his laws of motion wrong and his errors weren’t corrected for nearly 250 years.

  4. Mr. Andrew W:

    Do you seriously believe what you just wrote?

    Newton’s laws in the range of very small fractions of “c” have been verified to large numbers of decimal places as has Einsteins elaboration on those laws of motion. There isn’t anywhere near that kind of precision and accuracy in the temperature record — you are meaning what you say about Newton being found wrong in only the loosest manner of analogy? You are aware of that, no, because it is sometimes hard to tell if people are dead serious or are in leg-pulling mode?

    The Mann (or whoever takes credit) “hockey stick” temperature record, in a way, had been advanced as something approaching classical and relativistic mechanics experiments in accuracy. Here you have this dead-flat temperature curve (the shaft of the Hockey Stick) covering what some people thought to be the Medieval Warm Period and the Little Ice age, refuting those historical accounts as being local and anecdotal, and then in the 20’th century, woosh, you have this abrupt upturn in temperature, the blade of the Hockey Stick.

    So then you have the merging of two entirely different kinds of measurement. Pre-20th century, this is largely tree-ring data, and there are big, fat error bands that you could drive a Medieval Warm Period and a Little Ice Age through, but somehow after averaging and smoothing and PCA’ing (principle components analysis), you end up with a dead-flat smoothed curve smack down the middle of those fat error bands. And then you graft (glue? Duck Tape?) 20th century thermometer records as the blade at the end of the shaft of the Hockey Stick.

    You come up with this “smoking gun” (a Hockey Stick, actually, let’s not mix our metaphors) where there is no mistaking, there is a clear anthropogenically forced CO2 induced temperature “signal” that you have to be some kind of right-wing, State of Texas schoolbook-vetting nutter to not see.

    Only the publishers of the Hockey Stick are being coy. You are merging two entirely different data sets, one of them with big fat error bars to come up with this. You then try to extend the tree ring temperature estimate into the 20th century, and it goes down and not up, so you “hide the decline” as they say, only hiding the decline doesn’t mean what the wing-nutters think it does because the tree ring data have fat error bars and are not meant to be used without the heavy smoothing so they are worthless is saying anything about the 20’th century anyway, and nothing was hidden in the published papers as it was all footnoted if you wingers would bother to read the fine print? Did I miss anything?

    And then those two Canadians who lack any climate science credentials get into a micturation competition over the raw data used to generate the shaft of the Hockey Stick, which is only grudgingly offered by Mann and others (do I have that right?), and the Canadians claim that if you do the PCA their way, you get back the Medieval Warm Period and the Little Ice Age, but this analysis is suspect because just shut up?

    Even if Climate Change is for real inasmuch as there are real temperature changes brought on by CO2 emissions but those temperature changes are small (for now) compared with natural variations such as occured over the last 1000 years, this doesn’t mean that we are not at risk of cooking if we keep pumping out the CO2.

    But the whole point of the Hockey Stick is, not, the current temperature change is a precursor to a large change if we don’t change course, the current temperature change is supposed to be an undeniable sign that the Global Warming is happening right now.

    But what if Global Warming is real but the Hockey Stick is whatever adjectives has gotten Rand into legal trouble? What if O.J. is guilty but the LAPD planted evidence? Both circumstances undermine science and undermine justice.

    1. Thank you for your reply.

      I was being a little cynical in using the term “cynic”. The point I was trying to make was that Humans are rationalizing rather than rational, and with a strong enough incentive they’ll believe whatever they want to believe, no matter how strong the evidence that contradicts those beliefs, hence the reference to Newton’s Laws.

      The LIA and MWP were both evident in Mann et al 1999.

      I didn’t know Rand was in “legal trouble” for using adjectives to describe the hockey stick.

      I have no interest in the other points you raise concerning right-wing, State of Texas schoolbook-vetting nutters, whether O.J. is guilty and if the LAPD planted evidence.

      1. The point I was trying to make was that Humans are rationalizing rather than rational, and with a strong enough incentive they’ll believe whatever they want to believe, no matter how strong the evidence that contradicts those beliefs, hence the reference to Newton’s Laws.

        In just one other word, “rationalizing”.

        It’s pretty disingenuous to compare the uncertainty and complexity of climatology research, even something relatively simple like the behavior of global mean temperature over human history, to Newtonian mechanics which can easily be verified to several significant digits on a tabletop.

  5. “The LIA and MWP were both evident in Mann et al 1999.”

    Link? Reference?

    Seriously, the presentations I have seen of these data show a dead-flat past-1000-years. So you are saying that Mann et al corrected their work and their correction shows the LIA and MWP? You mention Mann et al 1999 — that is a “while back” — are you instead saying that the LIA and MWP are evident in their original disclosure?

    Speculations on whether the jury got it wrong and O.J. really “did it” or whether the LAPD planted evidence are an analogy. You can choose to have interest in such matters or not, but they are an analogy, just like the correspondence between Newtonian and Einsteinian mechanics is offered as an analogy to refinements of models in climate science.

    There are many facets and form of evidence offered in support of the anthropogenic CO2-driven warming hypothesis. For sake of discussion, let’s just say that there is a CO2-induced warming “signal” in the temperature data, but that signal is small compared to natural variations — the LIA and MWP as you and others use acronyms.

    This does not necessarily mean that continued, relentless, and exponential growth in carbon fuels usage won’t have a planet-changing effect with serious consequences for humans and for the rest of the natural world. But if the current temperature “signal” is a small effect compared to natural variations and other disturbances, and if either the scientific community or the science popularizers say otherwise, such really poisons the well.

    It is indeed like the police planting evidence. A person could be guilty, but an overzealous prosecution, if not by fraudulent conduct but by spin and overstating their case, if such annoys the jurors or provides legal grounds for a defense and gets a guilty person a not-guilty verdict, this behavior frustrates justice. CO2-induced warming could be a fact, but if the scientific community and its apologists and popularizers try to advance their case by claiming what the evidence does not support, there can be an equally bad outcome.

      1. I don’t pretend to be a climate scientist like Mann, and I don’t really participate
        in the debate, given it appears to be a consensus of the associations,

        but I think if you look at Fig 1 and Fig 3 you can see what appears to be a warm period
        in the 1300’s and a cold period in the 1500s. The graphs aren’t well scaled but, the filtered ones
        appear to show the MWP and LIA.

  6. This isn’t that surprising. Professors are always complaining about grading papers. I don’t think most of these peer reviers have any desire to grade their peers homework, especially at the level it was presented, and so don’t put the required effort in. The people who need to be peer reviewed are often boring people with tedious work and treat writing something up as masterbation with their vocabulary. It takes a special type of person who enjoys reading that stuff and be contrarian enough to look for faults.

Comments are closed.