Megan McArdle has an excellent piece on the nature of the discipline and its perverse incentives:
The system was rewarding a very, very specific thing: novel but intuitively plausible results that told neat stories about human behavior. Stars in that field are people who consistently identify, and then prove, interesting but believable results.
The problem is that reality is usually pretty messy, especially in social psychology, where you tend to be looking for fairly subtle effects. Even a genius will be wrong a lot of the time: he will invest in hypotheses that sound convincing but aren’t actually true, or come up with data that is too messy to tell you much one way or another. Sadly, the prestige journals aren’t looking to publish “We tested this interesting hypothesis, and boy, the data are just a mess!” They want a story, the clearer, the better.
Academics these days operate under enormous pressure to churn out high volumes of these publications. Hitting those targets again and again is the key to tenure, the full professorship, hopefully the lucrative lectures. Competition is fierce for all of those things, and it’s easy to get knocked out at every step. If getting good results is somewhat random, then all those professors are very vulnerable to a string of bad luck. The temptation to make your own luck is thus very high.
Again, I do not excuse those who resort to cheating. But as the consumer of these publications, we should be worried, because this system essentially selects for bad data handling. The more you manipulate your data (and there are lots of ways to massage your data so that it shows what you’d like, even without knowing you’re doing it), the more likely you are to come up with a publishable result. Peer review acts as something of a check on this, of course. But your peers don’t know if, for example, you decided to report only the one time your experiment worked, instead of the seven times it didn’t.
It would be much better if we rewarded replication: if journals were filled not only with papers describing novel effects, but also with papers by researchers who replicated someone else’s novel effects. But replicating an effect that someone else has found has nowhere near the prestige–or the publication value–of something entirely new. Which means, of course, that it’s relatively easy to make up numbers and be sure that no one else will try to check.
Most cases are not as extreme as Stapel. But if we reward only those who generate interesting results, rather than interesting hypotheses, we are asking for trouble. It is hard to fake good questions, but if the good questions must also have good answers . . . well, good answers are easy. And it seems that this is what the social psychology profession is rewarding.
What I found fascinating about this is that you can substitute the phrase “climate science” for “social psychology” and (say) “Mann” for “Stapel,” and it makes just as much sense.
This is probably worth a PJMedia piece.
[Update a few minutes later]
One other phrase that would have to change: “that told
neatpolitically appealing stories about human behaviorhumanity’s impact on the environment.”