A paper recently appeared in the open access peer-reviewed journal PeerJ titled Lung cancer incidence decreases with elevation: evidence for oxygen as an inhaled carcinogen. It makes the hard-to-believe claim that the oxygen we breathe is a cause of lung cancer. This is pretty far from my own expertise. Can I trust this result? Reading the paper I find it seems to have done the sort of statistical tests I would expect, checking for a long list of other possible factors including such things as radon and UV exposure (as well as of course cigarette smoking). Maybe they missed an important factor (cosmic ray exposure? levels of some other carcinogens that vary with elevation?) but the analysis looks pretty reasonable to me.
I know nothing about the authors or their previous publication history. Their institutions are reasonably well-known (University of Pennsylvania, UC San Francisco). In the paper they declare "no funding for this work" which is a little suspicious - most good science is done with some sort of external funding. What about the journal? PeerJ is a new entrant (launched in February 2013), focused on Biological and Medical sciences, with a very different business model from traditional journals. But it claims to be applying rigorous peer review. I have some first-hand knowledge of that as a co-author (among many) on a paper being considered for publication in that journal - we had a pretty thorough first round of review by external referees.
What about external commentary on this oxygen-causes-cancer paper? There was quite wide positive coverage presumably following a university or journal press release, for example this report from EurekAlert!. The only negative response I could find was this one from Fiona Osgun at Cancer Research UK which concludes "this paper is an interesting read certainly, but definitely doesn’t tell us that oxygen causes lung cancer" - her primary complaint seems to be they likely didn't take smoking fully into account due to issues with the years for which data was acquired in the study.
Who to trust here? Unfortunately it's very unclear at the moment. To me this doesn't look like either the journal or authors were "behaving badly" - at worst they might have made some honest mistake that will be understood as this research is followed up on. At best - well, maybe we now understand another source of cancer risk. But maybe I'm way off base - like I said, this isn't a field I'm at all familiar with!
A second recent paper in a similar vein is Solar activity at birth predicted infant survival and women's fertility in historical Norway published in the quite-prestigious 350-year-old Proceedings of the Royal Society B (impact factor 5.683). This is again a statistical study of correlations, this time between solar activity (sunspot counts) at date of birth and various metrics of survival and fertility. Again the authors seem to have accounted for a variety of possibly confounding factors - the science looks reasonable to me (far from an expert in this field). The authors themselves are from the "Norwegian University of Science and Technology" in Trondheim which seems like a respectable place. The journal is clearly among the most well-regarded in the world. There's no sign of bad behavior by either author or journal here.
So what about reactions? The paper received pretty wide-spread media coverage, again essentially parroting a (positive) press release, with occasional quotes from other scientists expressing doubts about the mechanism. The only lengthy critical response I could find was this one by Richard Telford, and his critique of the UV mechanism appears quite devastating. Norwegians have very little UV exposure in the first place, and cloud and regional variability is much more significant than the solar cycle effect. His suggested explanations for the paper (assuming the authors and journal did not behave badly):
The first is that some property of the data causes the statistical methods (which look good) to fail unexpectedly. The second is chance – one of the 5% of results that appear to be statistically significant at the p = 0.05 level when the null hypothesis is true.
There is a third possibility - that some other mechanism than UV variation is mediated by the solar cycle and has an impact on life expectancy etc in Norway (and perhaps elsewhere).
So do I trust this one? Given the strength of the one critique, right now I feel most likely the authors have made some sort of honest mistake. But again I'm way outside my own expertise, it is really hard to know what's right.
In both cases this state of uncertainty isn't bad - it's quite normal for science at the cutting edge to be ambiguous and uncertain. Scientific results that stand the test of time require confirmation by replication of studies like these under different conditions, testing the explanatory hypothesis via all the experimental and theoretical implications one can. When multiple lines of evidence all support the same picture of reality, then one can be fairly certain it is right. When we just have one line of evidence, in cases like this, it's fine and appropriate to be skeptical. Over time the truth should be sorted out.
Jay Rosen recently posted a "banking theory" of trust (in his particular emphasis, the trust given to major news operations). The basic idea is that "trust" is not a Boolean switch - either you trust a source or you don't - but exists in gradations that vary in time. A given source has (from the perspective of a reader) reserves of trust that increase through actions promoting trust (openness, providing useful, verifiable truths) and that are drawn down through other actions (particularly making unverifiable claims that say "trust us").
The question of who to trust has always been with us, but with today's fast pace and interconnected world its salience is greater than ever. Individuals and institutions may earn our trust for a period of time, but a change in responsibilities for a person, new management for an organization, or a simple unveiling of long-standing bias can rapidly erode that trust and leave us confused and desperately seeking new guides in the world.
I've written before about the importance of trust, particularly in communicating about science. One of my personal refrains has been that "trust is elusive" and "trust is fragile" - hard won and easily lost. In the context of Jay Rosen's "banking theory", it takes great effort to build up the reserves but even a small error can make a substantial dent in those reserves, and a perceived pattern of carelessness or indifference, or indications of bias or other human defects, can quickly wipe them out.
So considering Science as an institution, with the openness and transparency engendered by the internet we're in a period of significant change. With all these changes, can the public still trust "Science"? Evidently there are many out there who don't - last night's US Senate votes on climate change were a little disheartening...
The rise of predatory publishers is one side of the problem - to the outsider it may be hard to tell whether a supposedly peer-reviewed scientific journal deserves any trust at all. A new entrant like PeerJ may be publishing really valuable ground-breaking research (like maybe the oxygen paper here) - if the papers it publishes overall prove good, it will be banking up that trust Rosen talks about, and gain the respect (and metrics) it deserves - but this will likely take years. On the other hand, a long history and good metrics such as "impact factor" (like the Royal Society publications) is a good indicator of a big bank of built-up trust - but a lot may be squandered by publishing results that turn out wrong (as I suspect with the sunspots).
But it's not just scientific journals whether traditional, new, or predatory, that are out there communicating science these days. We have blogs, social media, and the traditional media where science journalism has taken on a range of new online forms. Scientists can communicate more directly to the wider world through their own websites, or through university press offices which seem these days to like turning boring science into over-hyped "clickbait".
There are some in favor of much more radical change - Michael Eisen, for example, thinks peer review is broken and post publication peer review is the solution. A recent article in Chemistry World points out some successes but also some challenges for post publication peer review. The general idea is that scientific journals are not needed, scientists can just post their work to the public and let things be sorted out there.
I'm generally sympathetic to this kind of approach - pretty much all the advantages Michael Eisen and others mention are true. My problem with post publication peer review: most published scientific articles do and will receive absolutely no comment whatsoever after publication. That lack of comment gives you no signal at all about the trustworthiness of the research. At least when published in a journal, the journal is staking its name and brand, the trust we have in it, on those articles it includes. Without the journal what signal do we have for the vast bulk of scientific research?
Even when post publication peer review happens, the back and forth in commentary can leave bystanders bewildered. For the two recent articles I mentioned above, only the lung cancer-elevation study has received any comments on either PubPeer or PubMedCommons - and that comment was merely a link to the Cancer Research UK article I mentioned. The authors responded, with quite cogent argument and analysis; they also re-emphasize the transparency of their work - the data and code is openly available. There is no further rebuttal at PubPeer or elsewhere that I can find. Did this post publication peer review increase my confidence in the work? Slightly. I would want to hear a handful of other perspectives from experts to be strongly confident in the work. And that's just one paper that received quite a lot of publicity (the article has been viewed 3271 times according to PeerJ).
So what are real solutions for improving the situation in Science generally, to give us sources of trustworthy information regarding our understanding about the world? I'll be exploring some ideas in future posts - but first I want to cover some more egregious cases of clearly bad science (physics, which I do know something about) coming out in quite respectable journals - much worse than the "sunspots" case I've discussed here. That'll be the next post...