In previous posts I have discouraged discussion of Michael Mann's work since I had not investigated it at all myself - but inevitably it came up anyway. There were a couple of interesting comments from Steve Mosher and AMac that I am highlighting in this post. If commenters here agree that the Tiljander case is the closest thing anybody has come up with to show consistent misbehavior by climate scientists (following the basic "fraud-like" criteria I set out) then I commit to looking into it myself and trying to understand why scientists and bloggers seem to be disagreeing about it. AMac's denial of "fraud" while calling it an "honest mistake" seems odd to me - if it's really an "honest mistake" it should be acknowledged, not repeated.
Or if folks here think the Tiljander case is not a real problem but some other hockey stick "trick" or feature is definitely fraudulent, I'll look there. Tell me what your best case is!
Just because scientists are human - that is biased, inconsistent, lazy, argumentative, make mistakes, argue, play "politics", etc. etc. does not make some piece of science a fraud. Scientists in their natural state are fiercely competitive with one another - recognition for solving some problem or being first to discover some new truth about the world is all that matters. Tearing down somebody else's work, if you're right, is always grounds for praise. As long as there is some collection of predictions about the world from a piece of science and measurements to verify those predictions, then no matter what the biases or mistakes of the scientists involved, as long as they are not being deliberately fraudulent, the truth will prevail. Of course, without that check and balance from nature, even without fraud, science can get wildly speculative (*cough* string theory *cough*).
Human frailties can mar any piece of scientific work, and this shouldn't surprise anybody. The worry is that some pieces of work that people have come to respect and rely on have been, in some manner, fabricated and are themselves wrong. But fraud is hard to perpetuate in science - it almost always turns up later when others try to do the same experiment or analysis over again and consistently get some different result. On the other hand, if there has not been any actual fraud, what's the problem? The science is still right, even if the scientists behaved abominably (and I've personally witnessed some pretty abominable stuff from people who received great honors...). That's sort of the beauty of the objectivity that the intrinsic competition and reference to nature of science forces on you: personalities really don't matter, only the truth does - it's only the thought that counts as I wrote some time ago.
But - there have been cries of fraud. Let's try to get to the bottom of them. Here are 5 objective criteria for clear continuing fraud that I posted here in a comment the other day:
(1) A result (graph, table, number) presented in a peer-reviewed or IPCC article that was false - i.e. said to be one thing, but was actually something else. Incomplete presentation is not sufficient - I want to see something that was actually false (such as this AR4 case would have been if it had worked out). Truncation doesn't count unless they claimed to be presenting a whole series and clearly actively concealed the truncation. End-point smoothing doesn't count (for example the Briffa 2001/NCDC graph) unless they specified how they were handling the endpoints and did it differently. Etc.
(2) Where the falsity made a material difference to the overall message of the graph, table or number. That is, the visual or mental impact of the difference is obvious to a cursory look at the presentation, and doesn't require detailed magnification of the curve or looking at the last decimal point to see any difference.
(3) Where the problem, identified by blogger or scientist or whoever, has been presented in a clear manner demonstrating the "wrong" and "right" versions for all to see
(4) Where the original scientific group responsible has not responded with acknowledgment of the error and corrected the record as far as possible, and committed not to make the same mistake again
(5) Where the original group has in fact repeated the error more than once, after public disclosure of the problem.
I'm reposting here two lengthy responses to this outline, and encourage further discussion of these in the comments below:
...Short answer: No, But.
This freestanding comment is a Reply to MikeN's "Interesting, I think" (Sun, 6/27/2010 - 00:36) and Arthur Smith's "On the "'my standard' question" (Sat, 06/26/2010 - 18:26). This seeming side-issue may illuminate some of the points being discussed with the termination of the Briffa series in 1960.
Arthur listed 5 criteria in "On the 'my standard' question". Paraphrasing,
(1) A false result is presented in a peer-reviewed article or IPCC report.
(2) The falsity made a material difference to the overall message of a graph, table or number.
(3) The "wrong" and "right" versions of the identified problem have been presented in a clear manner.
(4) The authors have not acknowledged and corrected the error, or committed to not repeat the mistake.
(5) The authors have repeated the error, after public disclosure of the problem.
Two definitions for "Fraud":
a: deceit, trickery; specifically: intentional perversion of truth in order to induce another to part with something of value or to surrender a legal right
b: an act of deceiving or misrepresenting: trick
We're in tricky [sic] territory already: accusers can mean (or claim to mean) that they're discussing "misrepresentation", but the charge of evil intent is present or nearby. Lack of care and precision in statements made by scientists and advocacy bloggers is one of the major polarizing factors the AGW dispute, IMO. Steve covered this ground nicely in Climategate: Not Fraud, But 'Noble Cause Corruption' (also note the cries for blood in the comments).
It's tractable to evaluate what somebody wrote in a journal article, much less so to ascertain what was in their heart at the time of writing. To me, this says most "fraud" charges will be either wrong or unprovable. They'll always be red flags to a bull (bug or feature?).
As described in the Methods and SI of Mann08 (links here), Prof. Mann and co-authors set out to catalog and use non-dendro proxies that contain temperature information. They assembled candidates and looked at behavior over the time of the instrumental record, 1850-1995. During this time of warming, the calculated mean temperature anomaly in most CRUtem cells (5 deg longitude x 5 deg latitude) rose. Proxies with parameters that also rose passed screening and progressed to the validation step (see the paper). The four measures (varve thickness, lightsum, X-Ray Density, and darksum) taken by Mia Tiljander from the lakebed varved sediments of Lake Korttajarvi, Finland also passed validation, and thus were used in the two types of paleotemperature reconstructions (EIV and CPS) that make up the paper's results. The authors recognized potential problems with the Tiljander proxies, but used them anyway. Because of their length (extending much earlier than 200 AD) and the strength of their "blade" signal (Willis Eschenbach essay), the proxies are important parts of the reconstructions.
The evidence is overwhelming that Prof. Mann and co-authors were mistaken in their belief that the Tiljander proxies could be calibrated to CRUtem temperature anomaly series, 1850-1995. The XRD proxy discussed here. The issue was recently raised again by A-List climate scientist and RealClimate.org blogger at Collide-a-Scape, The Main Hindrance to Dialogue (and Detente). Gavin and Prof. Mann's other allies are unable to address the matters of substance that underlie this controversy; see my comment #132 in that thread.
Arthur's 5 Criteria and Mann08/Tiljander
(0) Mann08's use of the Tiljander proxies is not fraud, IMO. All evidence points to an honest mistake.
(1) False result presented in a peer-reviewed article? Yes.
(2) Falsity made a material difference to the overall message of [graphs]? Hotly contested. Mann08 has many added methodological problems, making it difficult to know (see comment #132 and critical posts linked here). IMO, this demonstrated failure of key Mann08 methods (screening and validation) calls the entire paper into question.
(3) Clear presentations of "wrong" and "right" versions of the identified problem? Hotly contested. Gavin believes that the twice-corrected, non-peer-reviewed Fig S8a shows that errors with Tiljander (if any) don't matter. I rebut that in comment #132 and in this essay.
(4) The authors have not acknowledged and corrected the error, or committed to not repeat the mistake. Yes. In their Reply published in PNAS in 2009, Mann et al. called the claims of improper use of the Tiljander proxies "bizarre."
(5) The authors have repeated the error, after public disclosure of the problem. Yes. Mann et al. (Science, 2009) again employed the Tiljander proxies Lightsum and XRD in their inverted orientations (ClimateAudit post); see lines 1063 and 1065 in "1209proxynames.xls" downloadable in zipped form from sciencemag.org (behind paywall).
Summary, and Lessons for the Briffa Series Truncations
The key issue is not fraud. Nor is it that authors of peer-reviewed articles make mistakes. Everybody--scientists, book authors, and climate-bloggers included--makes mistakes.
Instead, the important question is: Does climate science adhere to Best Practices? Appropriately, scientists and bloggers scrutinize articles that cast doubt on the Consensus view of AGW, as shown by Tim Lambert in the 2004 radians-not-degrees case. What about papers that support the Consensus view? Are such errors in those papers picked up? Do the authors correct those papers, too?
Best Practices don't mainly concern the detection of glaring, easily-understood errors like a radian/degree mixup or an upside-down proxy. There are a host of issues -- as there are with drug research, structural engineering, mission-critical software validation, and a large number of other areas. I won't enumerate them -- beyond a plea for the correct and rigorous use of statistical tools. Recent threads at Collide-a-scape are full of suggestions and insights on this question, from AGW Consensus advocate scientist Judith Curry, and many others.
The key to the Tiljander case is the defective response of the scientific establishment and the AGW-advocacy-blogging community. I think it teaches that paleoclimatology is a young science that has yet to establish Best Practices (as the concept is understood by other specialties, by regulators, or by the scientifically-literate lay public). To the extent that Best Practices should be obvious -- e.g. prompt acknowledgement and correction of glaring errors -- scientists' and institutions' responses merit a "D" or an "F" to this point.
Broadly speaking, I think scientifically-literate Lukewarmers and skeptics accept the analysis of the last few paragraphs. In contrast, opinion-setters in the climate science community and among AGW-Consensus-advocacy bloggers emphatically reject it.
IMO, these differing perceptions explain much of the gulf between the opening positions of Steve Mosher and Arthur Smith on the general question of the justification for the 1960 truncation(s) of the Briffa series, and on the specific question of Steve's error in ascribing the padding of the AR4 truncation to a splice with the instrumental record.
I think this is a really important comment. It lets me describe the central thesis of the book and our view of things.
What the mails detail is the creation of a bunker mentality. this mentality is best illustrated by some of the mails written by Mann. Essentially it is a vision of a battle between climate scientists and skeptics. Us and them. I put aside the question of whether this
mentality was justified or not. The important thing is that this mentality existed. Jones in an interview after climategate confirms the existence of this mentality. I do not think there is any evidence that contradicts this observation. The mentality existed. It is reflected in the language and the actions. What I try to focus on is how this mentality shapes or informs certain behaviors. We struggled a great deal with the language to describe the behavoir. Fraud was too strong a description. I would say and did say that the mentality eroded scientific ethics and scientific practices. it lead to behaviors that do not represent "best practices." These behaviors should not be encouraged or excused. They should be fixed.
When we try to make this case we face two challenges. We face a challenge from those who want to scream fraud and we face a challenge from those who want to defend every action these individuals took. Finding that middle road between "they are frauds" and "they did no wrong." was difficult to say the least. In the end its that middle ground that we want claim. The mails do not change the science ( said that many times in the book), but the behaviors we see are not the best practices. We deserve better science, especially with the stakes involved. If our only standard is the standard you propose, then I don't think we get the best science. I'll just list the areas in which I think the bunker mentality lead people to do things they would not ordinarily do. And things we would not ordinarily excuse.
A. Journals. There are a a few examples where the mails show the small group engaging in behaviors or contemplating behaviors that dont represent best practices.
1. Suggesting that "files" should be kept on journal editors that make editorial decisions you dont agree with
2. Influencing reviewers of papers.
3. Inventing a new category ( "provisionally accepted") for one paper so that it can be used by the IPCC.
B. Data archiving and access.
1. Sharing data that is confidential with some researchers while not sharing it with others. If its confidential, its confidential. If its
not, then its not.
2. Failing to archive data.
C. Code sharing.
1. Withholding code when you know that the code differs from the method described in the paper and correspondents
cannot replicate your results because of this discrepency. And you know they cannot replicate BECAUSE of this failure
of the paper to describe the code completely.
D. Administrative failures.
1. Failure to discharge one's administrative duties. see FOIA.
E. Failure to faithfully describe the total uncertainties in an analysis.
As you can see, and as we argue, none of these touches the core science. What we do argue is this. The practices we can
see in the mails do not constitute the best practices. I've argued at conservative sites that this behavior did not rise to the
level of fraud. And I took my lumps for failing to overcharge the case. On the other hand, those who believe in AGW (as we do), are unwilling to acknowledge any failings. We were heartened by Judith Curries call for a better science moving forward. We think
that the behaviors exhibited do not represent the best science. We think we can and should do better. The gravity of the issue demands it. So on one side we hear the charges of fraud . That's extreme. On the other side we hear a mis direction from the core issue. When we point out that best practices would require code and data sharing,for example, the answer is
" the science is sound." we dont disagree. What we say is that the best path forward is transparency and openness. Acknowledge that the decisions made were not the best and pledge to change things going forward.
Concern E is the heart of the matter WRT chap 6 of WG1. On our view Briffa was put under pressure to overstate the case.
That's not fraud. It's not perpetuating false statements. If you study the mails WRT to the authoring of that chapter you will come away with the impression that Briffa was under pressure to overstate the certainty. That doesnt make AGW false. It cannot. It is however a worrisome situation.
Here is Rind advising the writing team.
"pp. 8-18: The biggest problem with what appears here is in the handling of the greater
variability found in some reconstructions, and the whole discussion of the 'hockey stick'.
The tone is defensive, and worse, it both minimizes and avoids the problems. We should
clearly say (e.g., page 12 middle paragraph) that there are substantial uncertainties that
remain concerning the degree of variability - warming prior to 12K BP, and cooling during
the LIA, due primarily to the use of paleo-indicators of uncertain applicability, and the
lack of global (especially tropical) data. Attempting to avoid such statements will just
cause more problems.
In addition, some of the comments are probably wrong - the warm-season bias (p.12) should
if anything produce less variability, since warm seasons (at least in GCMs) feature smaller
climate changes than cold seasons. The discussion of uncertainties in tree ring
reconstructions should be direct, not referred to other references - it's important for
this document. How the long-term growth is factored in/out should be mentioned as a prime
problem. The lack of tropical data - a few corals prior to 1700 - has got to be discussed.
The primary criticism of McIntyre and McKitrick, which has gotten a lot of play on the
Internet, is that Mann et al. transformed each tree ring prior to calculating PCs by
subtracting the 1902-1980 mean, rather than using the length of the full time series (e.g.,
1400-1980), as is generally done. M&M claim that when they used that procedure with a red
noise spectrum, it always resulted in a 'hockey stick'. Is this true? If so, it constitutes
a devastating criticism of the approach; if not, it should be refuted. While IPCC cannot be
expected to respond to every criticism a priori, this one has gotten such publicity it
would be foolhardy to avoid it.
In addition, there are other valid criticisms to the PC approach....."
The PARTICULARS of this are unimportant. What matters is Rind's advice about treating uncertainties in a forthright manner.
All of our criticism of Briffa can be summed up in one sentence. He didn't do the most forthright description of the uncertainties.
That's it. whether it was his treatment of McIntyre's paper, or failing to disclose the truncated data in the clearest manner, that is the take home point we want to stress.
However, I'm not sure I understand why Steve is reluctant to cry "fraud" - or AMac for that matter. As I noted at the start, if "Tiljander" (whatever that means, I have not looked into it at all myself) makes a difference and they're still doing it wrong without acknowledging the error, then either perhaps the error hasn't been explained in a clear enough manner (showing how it makes any difference), or that's real fraud. Same with Steve Mosher's complaint about what Briffa was "forced" into (whether or not the emails provide enough context for these conclusions I can't say either - again I haven't looked into it myself). But if what Steve says here is true, then there was a numerical quantitative parameter - uncertainty - that was mis-stated in the IPCC reports. A proper analysis would show a different number. If Steve is right somebody should be able to do that proper analysis and get the right number, and show how it makes a difference. Persistence in using the wrong number after it's been shown there's a correct one would again be an instance of continued repeated fraud.
So - is it there, or not? What's the best case? Comments welcome on Mann in particular here, thanks!