Actually, maybe that disappearing "knowledge" is OK

A couple of weeks ago I posted an entry about the disappearance of online academic journals and how that's a bad thing.  Well, this article made me rethink that a little bit.

The author, Alvaro de Menard (who seems knowledgeable, but on whom I could find no background information, so caveat emptor), apparently participated in Replication Markets, which is a prediction market focused on the replicability of scientific research.  This is not something I was familiar with, but the idea of a prediction market is basically to use the model of economic markets to predict other things.  The idea is that the participants "bet" on specific outcomes and that incentives are aligned in such a way that they gain if the get it right, lose if they get it wrong, and maintain the status quo if they don't bet.

In this case, the market was about predicting whether or not the findings of social science studies could be replicated.  As you probably know, half the point of formalized scientific studies is that other researchers should be able to repeat the study and replicate the results.  If the result can be replicated consistently, that's good evidence that the effect you're observing is real.  You probably also know that science in general, and social science in particular, has been in the midst of a replication crisis for some time, meaning that for a disturbingly large number of studies, the results cannot be replicated.  The exact percentage varies, depending on what research area you're looking at, but it looks like the overall rate is around 50%.

It's a long article, but I highly recommend reading de Menard's account.  The volume of papers he looked at gives him a very interesting perspective on the replication crisis.  He skimmed over 2500 social science papers and assessed whether they were likely to replicate.  He says that he only spent about 2.5 minutes on each paper, but that his results were in line with the consensus of the other assessors in the project and with the results of actual replication attempts.

The picture painted by this essay is actually pretty bleak.  Some areas are not as bad as you might think, but others are much worse.  To me, the worst part is that the problem is systemic.  It's not just the pressure to "publish or perish".  Even of the studies that do replicate, many are not what you'd call "good" - they might be poorly designed (which is not the same thing as not replicable) or just reach conclusions that were pretty obvious in the first place.  As de Menard argues, everyone's incentives are set up in a perverse way that fails to promote quality research.  The focus is on things like statistical significance (which is routinely gamed via p-hacking), citation counts (which doesn't seem to correlate with replicability), and journal rankings.  It's all about producing and publishing "impactful" studies that tick all the right boxes.  If the results turn out to be true, so much the better, but that's not really the main point.  But the saddest part is that it seems like everybody knows this, but nobody is really in a position to change it.

So...yeah.  Maybe it's actually not such a tragedy that all those journals went dark after all.  I mean, on one level I think that is kinda still is a bad thing when potentially useful information disappears.  But on the other hand, there probably wasn't an awful lot of knowledge in the majority of those studies.  In fact, most of them were probably either useless or misleading.  And is it really a loss when useless or misleading information disappears?  I don't know.  Maybe it has some usefulness in terms of historical context.  Or maybe it's just occupying space in libraries, servers, and our brains for no good reason.

You can reply to this entry by leaving a comment below. This entry accepts Pingbacks from other blogs. You can follow comments on this entry by subscribing to the RSS feed.

Add your comments #

A comment body is required. No HTML code allowed. URLs starting with http:// or ftp:// will be automatically converted to hyperlinks.