Peer Review and the Sphere of Legitimate Debate

Over at Lucia's blog on an essentially open thread, there was some back and forth on peer review, the merits and barriers to formal publication vs blogs, and so forth. My old friend Joel Shore responded to a comment on science journals expanding faster than the speed of light with a possible source, and mentioned the size of the Physical Review journals doubling every decade.

Since I work there, I thought I'd respond with more up to date data, and also added some thoughts on peer review related to some commentary on a simple diagrammatic explanation of journalistic practice, recently posted by Jay Rosen. My comments follow.

For what it’s worth, Physical Review publication statistics are available online here (2007 numbers):

http://forms.aps.org/general/annstats07.pdf

Table 1 has the numbers, Figure 1 shows it visually. The period of most rapid sustained growth was probably 1980-1995 where the number of papers published really was doubling every 10 years or so; a lot of that growth was from outside the United States where international physics programs, particularly in Asia (Japan and Korea, etc.), were picking up steam and looked to our journals to publish. From 1994 to 1997 things seemed to steady out at very close to 10,000 articles published per year - international economic conditions, particularly the collapse of the Eastern European scientific enterprise, are presumably the explanation for the stall. The recent resumption of growth has again been largely international - physics articles from China have been going up by over 10% per year in recent years. But the actual doubling time expected going forward is more like the long-run number of 20 years (3 to 4% growth per year), rather than the 10 year doubling that briefly seemed to hold.

As to the merits of peer review as the Physical Review journals run it - it is a bit like a court system, but with physical argument and logic as the subject rather than points of law. An editor sits as judge, and several knowledgeable peers of the author act essentially as lawyers - they are asked to read the article carefully and decide whether there is any case against its publication or not. These referees generally take their role very seriously (after all, they sit as authors on the receiving end of such reviews as well). Experienced authors, knowing the rules, also take great pains to write as clearly and precisely as possible, to avoid misunderstandings that can ensnare an article for months. The editor makes the final decision, in some cases even taking the author’s side against referees whose critiques the editor deems irrelevant. The reputation of the journal rests on the results of the editor’s decisions - the quality of the articles that make it through - and in particular the decisions on high profile “extraordinary” papers. Just as with our court system, there are occasional wrong convictions or miscreants who go free, but by and large the world is a far better place thanks to these institutionalized processes than it would be (and was) without them. The progression of science accelerated greatly with creation of such institutionalized systems of review a few hundred years back.

As with any communications medium, the vast majority of communication in scientific journals is about issues that are “subject to legitimate debate”. Things that are already firmly established aren’t terribly interesting unless perhaps you are a textbook publisher or historian; reproducibility is one key element of science, but there is little publication of such reproduced results unless some new aspect, such as higher precision or different physical constraints, makes it of some legitimate interest. On the other hand, anything that contradicts firmly established results established in preceding decades has a very high burden to being considered believable by editors and referees - there must be either a logical synthesis of the new and the old in some fashion, or significant evidence for why the old results were wrong. “Extraordinary claims require extraordinary proof”, as the saying goes. So the vast majority of what is published is in that middle ground between the well-known and the extraordinary, and being acutely aware of those boundaries is what makes a scientifically trained editor and a peer referee so valuable.

There is nothing comparable to the institutionalized peer review system anywhere in the blog world.