The Mendicant Bug

Luis von Ahn has an insightful post lamenting the fact that we are holding onto a paper-world philosophy of academic publishing in a digital age. He kicks out the fledgling idea that a “wiki, karma, and a voting method like reddit” hybrid might supplant our current method. I’m always a little confused by the reluctance to change publishing models in academia. Granted, I have never struggled to get tenure at a university, nor is it remotely likely that that will ever be something I do. But still, computer scientists of all people, should be willing to change and adopt a more sensible model. It turns out we’re just people after all.

What might a wikarmeddit version of academic publishing look like? A good place to start might be Stack Overflow. They are a self-proclaimed combination of wiki, blog, reddit, forum, and have karma. Perfect, right?

The benefits of peer review by the herd are great, but not without pitfalls. First of all, you can be herd-reviewed by morons. Moron 1 might think everything Researcher A publishes is GOLD and gives the thumbs-up no matter how badly the research was done. Ditto on the flipside, with Moron 2 hating everything Researcher A does. I’m not really being fair. The number of real morons who bother with this sort of thing is probably low, but the number of non-experts is a different matter.

On the other hand, open sourcing the research results like this allows all sorts of insights that you wouldn’t see from peer review. First of all, has a reviewer ever tried implementing an algorithm described in a paper? If you are a reviewer who has — I salute you. I doubt it’s very common. But when I come across a paper that is interesting for a problem I’m working on, I do try to implement it. If it gives me fits, I either abandon the method or try to contact the authors. This is simplified in a StackOverflow academic review setting, where the herd is giving this sort of feedback to the authors as a part of the review process. You can see how this level of communication would be beneficial. Inane non-expert commenters will either be filtered out (if they are truly inane) or they will shed light on confusing parts of your research presentation, allowing dissemination of your research to an even wider audience. This last thing is often given lip-service by the scientific community, but rarely have I seen actual attempts to do so.

So the next question is do we reinvent the wheel? Stack Overflow already has a community of smart people in place. Why don’t we just start using it?  Maybe SO could include some functionality for more research oriented questions.  All research can be viewed a set of questions.  Is this a good way of attacking this problem?  Is there a better way of doing it?  Is the methodology sound?  Isn’t my method the shiz?

Note: I’m fairly certain I’m not the first person making this call. I’m pretty sure I heard someone else recently make this point (maybe it was John Cook?) but i can’t find the reference.  Please comment.