The Future of Peer Review?

slashdot recently ran an article on a concept for distributed peer review, GPeerReview, hosted at Google (code.google.com). It still looks rather half-baked (the author decides whether or not to post the reviews?!) but the basic idea seems like a possible foundation for something that could be actually useful.

The tool is presently just a command-line program (Linux and Windows are the only binary distributions they list right now) that links together and digitally signs two documents - your review, and a file containing the paper you are reviewing. So your review is specifically tied to one particular version of the paper, an interesting feature. There doesn't seem to be any mechanism for handling versioning, and it seems to require a local copy of the paper rather than using a citation or a link of some sort.

The tool also claims to be able to "crawl" lists of publications looking at these signed reviews and telling you something about the network. But it specifies no mechanism for organizing such things - from the FAQ:

Q: How will all the papers people self-publish be organized?
A: That is also a problem we are not trying to solve. This problem is better solved by someone else. Someone will probably create a site that aims to index these publications. If not, there's always search engines.

Hmmm. Sounds like an opportunity for a scientific society product, perhaps? There are some other interesting ideas suggested here:

When a very traditional conference accepts a paper, it may simply use GPeerReview to sign a statement such as "This paper was accepted for publication in Name of conference URL in year." Thus the reviews remain confidential and the reviewers remain anonymous, but the acceptance can still be verified by checking the signature.

Something like that could be quite useful - of course, the fact that a paper is physically included in the conference proceedings takes care of this too, but this is essentially an alternate way of "publishing" such proceedings in a more distributed fashion - but again that falls back on the question of organization.

One of the slashdot commenters favorably quoted mathematician Michael Nielsen, who has had some rather strong things to say about peer review recently. Nielsen has some good ideas too though, about the future of science and open access. Something to return to in posts here later I guess...