Steven Mosher: even Fuller of it

[UPDATE - June 24, 2010: the following text has been slightly modified following some discussion at ClimateAudit and in particular a detailed explanation from Steven Mosher of what he did wrong. Changes are indicated by strikethrough for deletions and bold for additions].

When people are obviously wrong it doesn't take much time or effort to dismiss their claims. When Joe Barton apologizes to BP we know he's spouting nonsense. When Gerhard Kramm gets simple integrals and averages confused it doesn't take much effort to convince anybody other than Kramm where he went wrong. When Tom Fuller blusters about quantitative meta analysis, Gish Gallops, and alternate universes you can tell he has trouble with logical coherence.

But the tricky cases are those who are much more subtle in their nonsense. Making stuff up is easy. Making stuff up that on the face of it looks somewhat plausible does take a bit more skill. Figuring out that the "plausible" stuff is just as much nonsense as the obviously wrong takes considerably more work, and some of these actors tend to make a lot of work for those of us trying to defend real science. One of the most skilled in creating plausible nonsense is Christopher Monckton. Prof. John Abrahams is the latest of us to take on Monckton's fabrications, and collectively thousands of hours have surely been spent tracking down the ways in which Monckton has misrepresented science.

Brian Angliss has recently put a lot of effort into tracking down the basis of some of the claims regarding "climategate", in particular looking at the implications of malfeasance on the part of the scientists whose emails were stolen. Many of these the conclusions Angliss examined were claimed at the website ClimateAudit, and in particular in a book published by Steven Mosher and Tom Fuller. There followed an extensive thread of comment including from Fuller and Mosher, and a response from Steve McIntyre at ClimateAudit that clarified some of the claims prompting Angliss to revise his article to attempt to correct his own mistakes.

The first discussion point in Angliss' review of the claims and in the ClimateAudit back and forth with Mosher and Fuller is the meaning of the "trick" to "hide the decline" phrase found in the stolen emails. This has been adversely interpreted in a couple of different ways but the actual meaning has been clearly identified as the process of creating graphs that do include tree-ring-based temperature "proxy" data only up to 1960, or 1980, a point where they start to diverge from temperatures measured by instrumental thermometers. There is nothing scientifically nefarious or "wrong" about this - the "divergence problem" has been extensively discussed in the scientific literature including in the text of the most recent IPCC report. If you have reason to believe a particular collection of tree ring data is a good measure of temperature before 1960 but for some still uncertain reason not after that point, then it's perfectly legitimate to create a graph using the data you think is reliable, particularly if these choices are all clearly explained in the surrounding text or caption.


Figure 2.21 from IPCC TAR WG1


Figure 6.10b from IPCC AR4 WG1

What's definitely not legitimate is presenting a graph that is specifically stated to be showing one thing, but actually showing another. That might happen just by accident if somebody messed up in creating the graph. But the ClimateAudit discussion and Mosher/Fuller book appeared to claim that in one figure in the 3rd IPCC report (TAR WG1 figure 2.21, 2001) and in one figure in the 4th report (AR4 figure 6.10b, 2007) there was a real instance where "the scientists had actually substituted or replaced the tree ring proxy data with instrument data" deliberately, for the purpose of "hiding the decline". As Angliss cited, McIntyre definitely uses the word "substitution" (but Angliss was apparently wrong that McIntyre did this in the IPCC context), and Fuller highlighted a portion of the Mosher/Fuller book using the word "replaced". McIntyre later clarified that his claim was not related to these IPCC figures but rather something else. However, Steven Mosher in comment #7 on Brian's article at June 8, 2010 at 12:34 pm stated very clearly that he knew what the trick was and that this substitution/replacement was used for the IPCC figures:

you wrote:

"Looking closely at the graph shows that the tree ring data was neither replaced nor substituted. The zoomed-in version of IPCC TAR WG1 Figure 2.21 at right shows that the instrument data starts around 1900 (red line, red arrow added) while the tree ring data ends at around 1960 (green line, green arrow added). If the tree ring data after 1960 were simply substituted or replaced as McIntyre and Fuller claim, then the instrument data would have been appended to the end of the tree ring data or the instrument data would be shown in green in order to maximize the potential for misinterpretation. Neither is the case."

The TAR is the third Report. We are talking about the FAR. figure 6.10. But I can make the same point with the TAR was with the FAR. You clearly don’t know how the trick works. Let me explain. The tree ring data POST 1960 is truncated. That is step 1. That step is covered in the text of chapter 6 ( more on that later ) The next step is to SMOOTH the data for the graphical presentation. The smoothing algorithm is a 30 year smooth. Whats that mean? For example,
if you have data from year 1 to year 100, your first data point is year 15. Its value is the combination of the 15 PRIOR YEARS and the 15 Following years ( for illustration only to give you an idea how centered filters work) your LAST year is year 85. This year is the combination of the prior 15 years of the record and the last 15 years. year 86 has no value because there are not 15 following years. So with a record that goes to 1960 your SMOOTH with a 30 year window should only display up to 1945. The problem of end point padding ( what do you draw from year 1945-1960) has extensive literature. So for example, there is extending the means of adjacent values at both ends of the smooth. ( the proceedure used in Ar4 ch06) In the case of Briffa’s curve, this procedure was not used. It was used for all the other curves, but in Briffa’s case it was not used. To fill out the filter, to supply data for 1945-1960, the INSTRUMENT SERIES was used.
This has been confirmed by replication. So still, after all this time people do not understand the trick because they have not attended to the math.

1. the series is truncated at 1960.
2. a smoothing filter ( typically 30 years) is applied.
3. To compute the final years of the smooth ( half the filter width) the temperature series is used.

That procedure is the trick. in a nutshell. If you want directions read Jones’ mail.

So Steven Mosher here claims that the "trick" was to use the instrumental data for "end point padding" in the 1960-truncated Briffa (2001) series used in IPCC AR4 Figure 6.10b (and presumably in the similar series in the TAR figure 2.21 Brian Angliss looked at). So that, despite claims to the contrary, in the IPCC reports Mosher claims they really did substitute/replace tree ring with instrumental data. And in a way that was concealed to the public - in particular, the caption of figure 6.10b specifically states what the end-point padding was:

“All series have been smoothed with a Gaussian-weighted filter to remove fluctuations on time scales less than 30 years; smoothed values are obtained up to both ends of each record by extending the records with the mean of the adjacent existing values.”

Similarly in TAR figure 2.21 the end-point padding is stated as:

“All series were smoothed with a 40-year Hamming-weights lowpass filter, with boundary constraints imposed by padding the series with its mean values during the first and last 25 years.”

Mosher is claiming a very specific procedure was used for smoothing that differs from that stated in these figure captions. I asked what the basis was for this claim, but no particular email from the scientists emerged to explicitly support Mosher's claim, and the closest thing to any analysis of the problem were pointers to this thread at ClimateAudit where, if the above endpoint padding procedure was examined, it's certainly not clear from the discussion.

One of the commenters pointed to the difference between the Briffa 2001 curve in the AR4 Figure 6.10b figure and in this NCDC page on the reconstructions:


Briffa 2001 reconstruction with others from NCDC

And indeed you see the Briffa curve (light blue) drops down a bit precipitously in the NCDC figure close to its endpoint in 1960, while the IPCC AR4 figure doesn't drop nearly so far - here's a closeup of the IPCC version:


Figure 6.10b from IPCC AR4 WG1 - closeup on the endpoints

So why are these different? While the differences in the individual curves common to both figures seem rather minor visually, there definitely looks like a problem with handling of smoothing near the Briffa 2001 end point in 1960. But note that the NCDC figure actually doesn't specifically state what endpoint padding was used for its graphs - it only says "All series have been smoothed with a 50-year Gaussian-weighted filter". Perhaps Mosher is right, that the NCDC figure uses the nearby-mean endpoint padding that the IPCC figure claimed to use, while the IPCC figure uses the instrumental data for padding, contrary to its specific claim about padding with the mean? If Mosher is right, that means the scientists really did conceal what they were doing here, and the figure caption for figure 6.10b (and presumably for the TAR figure as well) was a lie.

A commenter (Glen Raphael, #112) at Angliss' post thought proof of Mosher's point was that nobody had debunked it yet:

Perhaps an even stronger bit of evidence is that we haven’t seen Mosher’s claims “debunked” by any of the usual suspects. If his account were incorrect and there were some innocuous alternative way to generate the same graphics, don’t you think we’d have heard about it by now? Wouldn’t a rebuttal have shown up in gloating posts or comments at Deltoid, DC, RealClimate, Tamino, or all of the above? I think it’s safe to say *if* these claims were false they’d be easy for somebody with access to the data to debunk and it’s also safe to say that if they *could* be debunked that would have been done

Well, maybe nobody has actually seen Mosher state what he's talking about so clearly before. But now that he has, yes, it should be easy to debunk. Let's take a look.

The raw data for the NCDC graph is available for download via the page linked above. And here's what it looks like:


Raw (unsmoothed) NCDC data (R code here)

Now let's apply a Gaussian smoothing filter with nearby-mean endpoint padding as both the NCDC and IPCC AR4 figures claimed - though I don't know exactly the parameters or equation they used, I found a functional form that seems to roughly reproduce the main features of the curves in those figures:


NCDC data with Gaussian smooth (10-year) + padding with mean of first/last 15 years (R code here)

The Briffa 2001 curve (blue, ending in 1960) looks remarkably like the curve in the IPCC AR4 Figure 6.10b above. Let's look at close-ups near the end-point:


Magnified view of smoothed NCDC curve, and IPCC AR4 Fig 6.10b (R code here)

The Briffa 2001 curve is blue on the left (from NCDC data) and light blue on the right (IPCC). While the right-hand light blue curve is a little hard to see under all the others, it seems to peak at very close to zero, slope down and then tail off to flat right around -0.15 in both cases. Comparing to the Briffa 2000 curve (green on the left, darker green on the right) the Briffa 2001 endpoint is just a little above the bottom of the valley of the Briffa 2000 curve in both cases. I.e. a pretty good match.

But then, what explains the NCDC figure which had the Briffa 2001 data definitely heading rapidly downwards at the end? As I mentioned, the NCDC page doesn't say how endpoints were handled in that figure, so we'll need to do a bit of guessing.

The first natural possibility is that the endpoints were padded with the very last (or first) point in the series - rather than taking the mean of 15 or 25 points near the end, using the very last point alone. Doing that with the raw NCDC data gives this figure:


NCDC data with Gaussian smooth (10-year) + padding with first/last year data (R code here)

Oops! That looks even less like the NCDC graph, and not like the IPCC graph either - here's a close-up on the end:


Magnified view of smoothed NCDC curve with endpoint padding, compared with nearby mean padding (R code here)

The Briffa 2001 data now has a new valley at about -0.1 and then curves up - the reason for this is that the 1960 endpoint has a value of 0.076, much higher than the typical (negative) values in earlier years. So that one endpoint pulls the whole curve up when you pad and smooth in this fashion.

So clearly the NCDC graph didn't use the IPCC padding or simple endpoint padding for the Briffa 2001 data. One likely alternative was that they used the original Briffa 2001 data extended beyond 1960 for padding, despite the fact that the scientists had concluded it was no longer a valid temperature proxy after 1960. The NCDC data doesn't include the post-1960 Briffa 2001 data, and I couldn't find it in a quick search, but looking elsewhere it appears the data quickly falls to somewhere around -0.4 degrees temperature anomaly. So I took the simple expedient of padding the Briffa 2001 curve with -0.4's:


NCDC data with Gaussian smooth (10-year) + padding (Briffa 2001 only) with -0.4 (R code here)

and here's a comparison of the endpoint region:


Magnified view of smoothed NCDC curve with -0.4 padding (for Briffa 2001), compared with nearby mean padding (R code here)

This indeed looks very much like the original NCDC figure. So the unstated padding in that figure that brought the Briffa 2001 curve down so much was very likely use of the original Briffa 2001 curve beyond 1960, while chopping off the smoothed curve in 1960. That's perhaps justifiable, but a little inconsistent with the statements concerning the source of that data.

So it's pretty clear that the difference in endpoint smoothing between the NCDC and IPCC figures does not require grafting the instrumental data onto the tree-ring data as Steven Mosher claimed. But what do the graphs look like if you do that grafting?


NCDC data with Gaussian smooth (10-year) + padding (Briffa 2001 only) with instrumental data (R code here)

and comparing the endpoints region again:


Magnified view of smoothed NCDC curve with instrumental padding (for Briffa 2001), compared with nearby mean padding (R code here)

While the difference is small, you can see that the instrumental-padded curve flattens out more quickly than the mean-padded curve, and never goes much below -0.1 in temperature anomaly, while the mean-padded curve does go lower. Only the mean-padded curve matches the behavior of the IPCC AR4 WG1 Figure 6.10b illustrated above.

So, conclusively, despite Mosher's claims of certainty on what the scientists did, he was wrong.

But it sure takes a lot of effort to prove that one claim wrong, doesn't it?

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

The NCDC data doesn't include

The NCDC data doesn't include the post-1960 Briffa 2001 data, and I couldn't find it in a quick search

Try here: http://www.eastangliaemails.com/emails.php?eid=146&filename=939154709.txt

Ok, that works - and it looks

Ok, that works - and it looks pretty much idential to the '-0.4' continuation I plotted above:

(R source)

I'm taking a look through

I'm taking a look through what you've done here. Thanks for the R code. I'll compare this to Steve McIntyre's code as well.

I'm a bit confused as to the Briffa 2000 vs 2001. The NCDC chart has 2000, as do TAR and AR4. Then you added 2001.

No, the NCDC and AR4 charts

No, the NCDC and AR4 charts have *both* Briffa 2000 and "Briffa et al 2001". The only series truncated in 1960 in both charts is the "Briffa et al 2001", Briffa 2000 goes to 1993 in the data file. In the NCDC chart the "Briffa et al 2001" curve is labeled "This study" - because the page at NCDC is titled "Low-frequency Temperature Variations from a Northern Tree Ring Density Network, Journal of Geophysical Research, 106 D3 (16-Feb-2001) pp. 2929-2941, K. R. Briffa*, T. J. Osborn, F.H. Schweingruber, I.C. Harris, P. D. Jones, S.G. Shiyatov, and E.A. Vaganov. " - so that NCDC page is essentially a summary of "Briffa et al 2001" itself.

The TAR chart does have a "Briffa 2000" label truncated in 1960; I don't know the relation between "Briffa 2000" in TAR and "Briffa 2001" in AR4/NCDC, but the latter is where I could find data, so I went with that.

Thank you very much for the

Thank you very much for the clarification. Personally I had wondered about this for a while but didn't know if it would be addressed. (I have no reason to think that Brian is a great better at statistics than I am -- and on the basis of the above text your familiarity with statistical methods seems fairly advanced.)

A fair amount of gloating on Mosher's and Fuller's part -- but given their history no reason to think that the gloating had any basis in actual reality.

Tim, I wasn't going to

Tim, I wasn't going to address the claims that Mosher made with respect to how the AR4 graphs were generated in anything resembling a timely manner, and I'm quite glad that Arthur did so. I will do so eventually, but it will probably be weeks to months before it happens.

There is nothing here that

There is nothing here that contradicts Steve McIntyre at ClimateAudit, who discussed Phil Jones, and Michael Mann in Nature and GRL. The only discrepancy I see so far is that McIntyre links to a NHemMean dataset while you use jgr3, may be the difference in charts. Looks like you have caught Steve Mosher in an error. I took it at face value and explained it to you not realizing your background. Appears I have induced you to waste your time. I'd have been more careful if it was Thomas Fuller making the statements, who is not as well read. I have Steve's book and will see what he wrote about there.

No, I didn't look into any

No, I didn't look into any specific claims by McIntyre. He seems to be more careful to be accurate, albeit very frequently ambiguous or confusing.

I'd like to hear Mosher's explanation for why he was so certain on something that is evidently quite wrong. Did McIntyre's posts confuse him too?

Steve McIntyre in his

Steve McIntyre in his Heartland presentation DOES say that the TAR graph was done as Steven Mosher wrote. I can still find no post on ClimateAudit to that effect. Did you try to replicate the TAR graph as well?

Not yet - if somebody has a

Not yet - if somebody has a source for the original data for that figure, I could certainly look into it. It's definitely somewhat different from the AR4 figure (for which this NCDC data certainly applies). For example, around 1930-1940 *all* of the TAR lines go noticeably above 0, but that's not true in the AR4/NCDC case.

Q. who wrote "Over at Climate

Q. who wrote "Over at Climate Audit Steve McIntyre has started a new series on the “the trick.” To this day the trick is still largely misunderstood by nearly everyone discussing it, except Steve and a few of his readers. The trick deniers, are back at it using some old stupid pet tricks –moving the pea under the thimble. They clearly do not have a command of the mathematical operations underlying the trick or a command of all the versions of the trick."

Those whom the gods wish to destroy, they first make mad .....

One post at ClimateAudit did

One post at ClimateAudit did conclude that the TAR graph version of Briffa is more hockey-stickish than others and temperature grafting is the culprit. It is not listed in the body of the post, and changing to the new server has killed some of their links, so I missed it.

http://climateaudit.org/2007/05/11/the-maestro-of-mystery/

There is a post by UC describing adding temperatures.

for a very vague definition

for a very vague definition of "describing". What precisely did UC do? There's no graphs (any more) in support, either. And I don't get what McIntyre was up to in the main post there, either - trying to reproduce figure 2-21, yes, but only a couple of the series by Mann and Jones, not looking at Briffa's data at all. Perhaps he implemented the smoothing and end-point padding wrong, or perhaps he was using the wrong data series (do we definitively have a source for all the series in that TAR figure, anywhere?)

Turns out I completely

Turns out I completely misread Steve's post. When he wrote at S&R of talking about TAR, then WMO, then back to TAR, I assumed he meant in one paragraph that starts 'one version of the trick,' and figured it was a poorly written paragraph. Rereading it, he is clearly claiming bad smooth on TAR, so he was being unclear in his clarifications with Brian Angliss.

I think these are the sources for TAR. Not sure which column in the last one yet.
ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/mann1999/...
ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/jones1998...
ftp://ftp.ncdc.noaa.gov/pub/data/paleo/treering/reconstructions/n_hem_te...

Hmm, apparently that's not

Hmm, apparently that's not the Briffa source either. Here's a rundown of Steve McIntyre's attempts at locating the data. Not sure if this was the last word.

http://www.climateaudit.info/data/climate2003/pages/blog/briffa.mxd.htm

The book has the same errors

The book has the same errors repeated as well. The WMO chart simply doesn't do what they say, it clearly goes past 1960 with instrumental data. CRU has released an original without the trick as well.

I'm not even going to bother testing your code. AR4 had Briffa as a lead author who is much more responsible.
TAR had Michael Mann, and he is capable of anything. I will try and see if Mosher got at least one claim right.

In the 2007 thread “the

In the 2007 thread “the maestro is in da house”, Steve determined (see post update) that the smoothing used in MBH99 was also used in the TAR. The post wound up with Steve describing various attempts to match Mann’s reconstruction without success.

The actual method went undiscovered until UC commented 2 years later back in May 09 (in my comment at S&R I had stated that UC replicated a few years ago but I had overlooked the date change jump from 07 to 09 as I was reading through the comments). Experimenting with one of Mann’s smoothing algorithms, UC replicated MBH99 (and by extension – the IPCC TAR graph) by adding the instrumental series and padding with 0’s. See post and comments: http://climateaudit.org/2007/05/11/the-maestro-of-mystery/

It was UC who later suggested (in Novbember 09) when reading the emails that Jones was likely referring to MBH99 in the “trick” email. Having replicated the plot mere months earlier it was a natural connection to make.

According to IPCC TAR, the smoothing for the spaghetti graph was as follows:

All series were smoothed with a 40-year Hamming-weights lowpass filter, with boundary constraints imposed by padding the series with its mean values during the first and last 25 years.

According to UC this cannot be the method used.

MBH99 data: ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/mann1999/...

Jones et al 98 data: ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/jones1998...

UC’s code: http://climateaudit.org/2010/04/01/code-the-trick/

UC's code compared "padding

UC's code compared "padding with instrumental data" vs. "padding with zeros". That cannot be right - TAR didn't say they padded with zeros, they padded with the "mean values during the first and last 25 years". The last 25 years of MBH99 (1955-1980) averaged well above zero. Whatever UC was doing, he was not replicating the method claimed in the TAR caption itself.

You say: "up to 1960, or

You say: "up to 1960, or 1980, a point where they start to diverge from temperatures measured by instrumental thermometers."

I'm bothered by that 'start to diverge'

Slightly reframing, from 1960 or 1980 it is clear the tree rings do not support reliable temperature measurments - w know this by comparing with actual thermometer readings. What is the basis for the assumption that the rings are reliable for the years we don't have thermometer records ?

How do we now divergence started in 1960, rather than always being a feature of the tree-rings ?

1- By comparing them against

1- By comparing them against other trees ring series, which do not exhibit modern divergence.
2- By comparing them against non-tree proxies.
3- By noticing that, at least for the "foreseeable past", no major divergence similar to the modern one appears.

well here is what temps look

well here is what temps look like ( per Arthur) after 1960

0.15, 0.05, 0.02, -0.23, -0.35, -0.06, -0.01, -0.18, -0.2, -0.02, -0.14, -0.3

It's an interesting question whether tree rings that go down 'diverge' from this portion (1960-70) of the record.
I suppose we can go back to briffa's orginal study and look at the criteria he employed to truncate. Is there is an objective criteria for cutting off a divergent series? established in the literature?

"If you have reason to

"If you have reason to believe a particular collection of tree ring data is a good measure of temperature before 1960 but for some still uncertain reason not after that point, then it's perfectly legitimate to create a graph using the data you think is reliable, particularly if these choices are all clearly explained in the surrounding text or caption"

This sounds reasonable, but there are three time periods that need to be considered:

Firstly, the period where there is good evidence that the tree ring data and temperature are correlated - this is the calibration period when you have actual temperature data and actual tree ring data. Second, the divergence period where the data shows that tree ring data and temperature no longer correlate. However there is a third very important period the time before the calibration period where no temperature data is in existence. This is in fact the major part of the dendroclimatic record in terms of years.

Your explanation of the "trick" would indeed permit you to delete the divergence period data, with appropriate textual explanation, but it would also require you to delete the precalibration period data as well since without an understanding of why the divergence is ocurring you cannot postulate that the same effect was not happening before the calibration period.

Several people have referred to the divergent data as being 'incorrect' or 'erroneous', this is not the case, the data are what they are. What the divergence eperiod demonstrates is that the hypothetical relationship between tree rings and temeperature is only partially understood and consequently the relationship cannot yet be used to estimate temperature when instrumental data was unavailable.

This comment, as well as the

This comment, as well as the one above by Robbo, display some confusion as to what the divergence problem actually is. It's simply not the case that after 1960 "...the data shows that tree ring data and temperature no longer correlate." What actually happens after 1960 is that some tree ring series at high latitudes diverge from both measured temperatures and other tree rings, which continue to track observed temps nicely. So we know there isn't divergence prior to the instrumental record simply because the tree ring data aren't divergent during that period.

I'm sorry but your logic is

I'm sorry but your logic is at fault. We now know that "some trees" are divergent. Thus we know that only some trees follow temperature sufficiently well to make them indicators of paleo temperature. The problem is that we do not know why this happens nor do we know which trees will be divergent and which not. So you have no justification for using any tree ring data prior to the instrumental period as indicators of paleo temperature. We can probably hypothesise that some of these tree rings will indeed be proxy thermometers but we have no way of knowing which ones

Of course we have a way of

Of course we have a way of knowing which ones, namely a simple inter-comparison between multiple proxies. Multiple consilient proxy reconstructions provide strong evidence that none of the proxies are substantially off. Trying to throw out an entire category of data because of issues with a small subset may be standard creationist fare, but it's not how science actually works.

Your explanation of the

Your explanation of the "trick" would indeed permit you to delete the divergence period data, with appropriate textual explanation, but it would also require you to delete the precalibration period data as well since without an understanding of why the divergence is ocurring you cannot postulate that the same effect was not happening before the calibration period.

The divergence phenomenon is not ubiquitous, even among boreal tree-line series. This allows us to compare regionally similar divergent and non-divergent series and such comparisons reveal no significant pre-calibration period divergence, in some cases going back thousands of years. They can also be compared to other proxies, such as lake bed sediments. Again, there is strong consilience. Indeed, McIntyre has inadvertently demonstrated this, at least for the last few hundred years or so, in his kerfuffle over the Yamal/Khadyta River series.

Of course, this is merely strong empirical evidence that divergence is not an historical problem, and scientific curiosity, not being satisfied with mere empirical evidence, does seek robust physical explanations for phenomena, so research is ongoing. Also, dendrologists, being cautious researchers, have tended to exclude divergent series from reconstructions in recent years.

Luminous. I have no issue

Luminous.

I have no issue with the decision to exclude divergent data. It's ONE decision that an analyst can take. In making this decision, however, it's important to document the uncertainties involved. Where exactly does one truncate a series? That is a decision that carries undocumented uncertainty, what level of divergence is "required" corelation below 40%? 38%, 45%, 20%. What is the sensitivity to this decision? In any case, I've repeatedly argued that Briffa's decision to truncate was a logical justifiable decision. The issue ( biggest issue) for me was the presentation and the discussion of this decision. It was far more straightforward to SHOW the decline ( its full magitude) and explain WHY this divergence does matter, than to truncate the data and merely say " divergent data after 1960 was truncated." I wanna see the picture and if it looks interesting, then I wanna look at the underlying data. For example, how does the tree ring data diverge from the "raw" temperatures in Siberia. That temperature record has all sorts of interesting little quirks ( in the scribal records, the belly of GHCN) But if the divergent data is NOT archived, then its hard to go back and look at other analytical choices other than truncation. Briffa wants to truncate? fine. But when he archives the data, he should archive the full dataset. And when he presents the data he should present the impact of his choice to truncate.

Finally, a question for you

Is it best practices to archive data that is divergent or not.

Arthur, Firstly, you have not

Arthur,

Firstly, you have not "conclusively proven" that the Mosher/Fuller was wrong, you have yet to identify many of the procedures that the climate scientists used. All you've proven so far are that climate science methods are poorly documented and difficult to figure out. How about a bit of caution with your claims before you've really figured it out?

UC worked this out in great detail over at ClimateAudit some time ago (MikeN provided the link). UC did an incredible amount of work reverse engineering how the climate scientists produced their plots (and did a lot of work demonstrating the rather poor qualities of the filters and end-point padding that the likes of Mann used). Unfortunately, UC's blog (which was an auditblog) disappeared when auditblogs in general was wound up.

But let's make it clear here: UC did a great job reverse engineering this problem, which you have just started to embark on. You complain that UC's comment is not descriptive enough. Did it never occur to you that it isn't actually UC's responsibility to correctly document what climate scientists did in the IPCC TAR? You complain that it is hard work to resolve these disputes. I agree with you. But the reason it is hard work is because the climate scientists who produced the plot in the first place (1) are quite inconsistent with their methods (2) use non-standard, ad hoc methods on each graph and (3) fail to explain what they did.

If the climate scientists simply did a proper job of explaining how they generate the graphs, stick to simple, common methods then you wouldn't have this problem.

The responsibility does not lie with the auditors, but the originators of the data. I'm astonished you can't see this. If you want to contact UC, I put up his (old) public e-mail address up on lucias blog this post, comment 24578. I'm not sure if that e-mail is still working, but Lucia managed to contact him I think so she may be able to help. I suspect UC will have MATLAB code available to generate the plots.

But please try to remember. The root cause of this is the inadequate explanation on how the graphs were generated in the first place. That isn't McIntyres, Moshers or Fullers fault. It is interesting to watch your frustration here - this is the exact frustration the likes of McIntyre, Jean S and UC have suffered trying to figure out exactly what climate scientists have been doing for years.

Mosher made a statement

Mosher made a statement specifically about Figure 6.10 in AR4. I here have proved that his statement about that figure cannot be correct.

Replicating what the IPCC authors claimed regarding the AR4 figure was trivial, once I had the actual data (from the NCDC page) and had written a small amount of code to do the filter as described there. The description in the AR4 figure caption was accurate and complete - that was the first graph I put up.

Figuring out the NCDC graph was the tricky part because the endpoint smoothing was not described in that (2001) paper - that was what seemed to have triggered people thinking there was something wrong with the endpoint smoothing.

But overall, the choice of endpoint smoothing in a graph like this is not a major scientific matter - it is a question of data presentation, and extremely trivial in the larger scheme of things. It is only people like Mosher and McIntyre who are making a big deal out of something that, in this case, Mosher turned out to be completely wrong on.

UC apparently did something similar with the TAR figure, but there is now insufficient information to reproduce or understand what he did, unless we can get it directly from UC. it's unfortunate that his work was erased by McIntyre's website troubles; it's very odd, and I find it highly ironic, that Mosher (and McIntyre?) are basing claims on work that McIntyre failed to preserve for others to see and replicate later.

Arthur, please slow down a

Arthur, please slow down a minute. I appreciate you are trying to get to the bottom of a problem here but it is important to understand what is going on. You make a number of incorrect assertions based on my commentary.

Just as an example, take the TAR figure and UC's work. UC's work was not lost to the web because of a problem at McIntyre's website. Auditblogs had nothing whatsoever to do with McIntyre. And UC is not difficult to contact if you want to find out more information, and I expect he still has a copy of his work. I have already provided routes through which you could contact him. Have you even tried yet?

Secondly, the TAR graph should have been easy to recreate from the raw data if the descriptions in the text were sufficient. They weren't, and it took a considerable amount of work to find the problems with it - in your own words, "But it sure takes a lot of effort to prove that one claim wrong, doesn't it?". This is exactly the problem that McIntyre, Jean S and UC have had with climate science for years. It is quite ironic that you should make a similar complaint, and direct that complaint at people who have already done much of the reverse engineering for you.

Please try to understand that these things take time to understand, document and describe. Please understand it is not McIntyre's responsibility to describe the shoddy ways in which these things are done. The descriptions in the report should be sufficient. They are often not.

As for endpoint padding, it is rather important and UC's old site detailed some of the problems (end-point noise amplification) associated with "standard" climate science end point handling. It is a scientific issue, because it can be quantified and analysed. The visual effect in graphs and how the graphics are perceived can be important, particularly with long filters. It certainly "mattered" to the Hadley centre and CRU when the noise-amplifying end point filter they applied to their headline graphic for HadCRU suddenly switched from showing exaggerated warming to exaggerated cooling. (Noise has a tendency to do that). When that happened, they dropped the misleading end point handling faster than you could say "inappropriate time series manipulation". Many at CA knew the flaws of their method long before it became apparent to them, thanks to the work of UC.

I admit I haven't looked in detail at the AR4 aspects yet, and to be consistent I should do so. There is no question that the climate scientists cleaned up their act somewhat for AR4 in comparison with the TAR. But you make many strange statements in this post in the rush to publish and point fingers.

Yes, I emailed UC using the

Yes, I emailed UC using the address you provided this morning.

I agree the data behind IPCC graphs should be more readily available; some is, some isn't. I expect AR5 to do better on this. But that's par for the course for any scientific document up to now. RealClimate had a good discussion on replication some time ago - getting the original data and reproducing curves is a useful check, but more important is checking scientific results completely independently, that's the real validation of the science.

You may not have looked in detail at AR4, but Steven Mosher claimed very specifically, in the quote I provided, that AR4 Figure 6.10 was where this specific thing had been done. I quoted that, and that's what I examined.

Arthur, I've had a quick look

Arthur, I've had a quick look at the data and your calcs and they seem reasonable. I would agree with your observation that it looks like the evidence supports the calculation of Briffa with end points padded with the means.

However, I believe Mosher's description is quite accurate for the TAR.

I would also agree with McIntyre's view that plotting Briffa without truncation (or at least with the NCDC style of smoothing using the entire data set) would be a more accurate presentation of the present state of paleoclimate reconstructions.

UC's graph can still be found on McIntyre's website by the way - replace the "www.climateaudit.org" with "www.climateaudit.info" in the original URL, and the graph is still there.

Hmm, having continued to look

Hmm, having continued to look at this in more detail, I have to say that the 10-year gaussian smooth you have used here looks right for the NCDC plot, but wrong for the IPCC AR4 plot.

When I use a 5-year gaussian smooth over 30 years, I get a plot that looks much more like the AR4 plot than is achieved with the 10-year smooth.

It does affect the final plots, although I'm still inclined to agree that it looks likely that endpoint padding with the mean was used in AR4.

Yes, that could well be - the

Yes, that could well be - the curves in the AR4 plot do seem to vary a bit more than the removal of "fluctuations on time scales less than 30 years" statement would justify - obviously they didn't entirely specify their smoothing or endpoint mean parameters, so I guessed on that. But those choices, as long as they're reasonable, are going to change only details, not overall shape...

Spence, I think he has proven

Spence, I think he has proven Mosher wrong pretty well, assuming his methods and data are valid.

The accusation was that Briffa's data were appended with instrumental data, smoothed, then cut off at 1960 to hide the decline. He has shown that when he does the stated processing he gets a close match to the actual, and Mosher's claimed method does not. Just looking at the chart in question, it's pretty clear there would be no attempt to hide the decline here, as it makes no difference to the chart. The visual trick here looks quite different, the instrumental simply takes over at the end to show an uptick.

"If you have reason" is the

"If you have reason" is the critical phrase in your post. I cannot think of any reason why the tree ring data would be a reliable indicator of pre-instrumental temperatures when diverging from modern instrumental temperature records. The divergence since 1960 is prima facie evidence that the correlation over the earlier instrumental period is either an artefact or a result of cherry picking, and no amount of statistical horseplay has any bearing on the matter. The only way the "trick" could conceivably be defensible would be if they presented the data with and without the post 1960 numbers and gave a clear, coherent explanation as to why they thought the recent data should be discarded. That has never happened, for the simple reason that no such explanation exists.
As it stands, the behaviour is simply dishonest and would be grounds for a formal written warning in my business, with summary dismissal for a second offence; in climate science, however, the offence has been repeated on many occasions and defended by almost everyone in the "consensus".

Many explanations exist. The

Many explanations exist. The question is which one, if any, is right. Some of them scientifically justify discarding the recent data, some do not.

However, Mosher made a specific claim of dishonest behavior, which I here proved to be wrong. You are making another claim, one which has been repeated many other places, but it is much vaguer - perhaps if you think there is "grounds for a formal written warning" you should look into it and prepare a detailed formal statement of where anything deceitful was done. So far, all the examinations of specific detailed issues that I'm aware of have shown no dishonesty on the part of the scientists at all.

Show me one explanation that

Show me one explanation that stands up. The only thing that justifies discarding the recent data is failure in measurement or transcription - otherwise the recent data are at least as valid as the dataset on which the reconstruction is based and if they falsify the hypothesis, that the proxies provide an accurate indicator of global temperatures over the last 1000 years, it is dead in the water. Any statistical analysis of a dataset in which the inconvenient parts have been discarded is entirely valueless, and publication of such data in a document intended to inform public policy is, as I say, dishonest. What more do I need to say - the rest is window dressing.

Oh, I don't know, you can't

Oh, I don't know, you can't think of anything about the last 60 years that's different from the previous 1000, that could possibly be an explanation? As far as I'm aware, the scientists have come up with quite a few possibilities there. There's certainly valid scientific disagreement on the question, but it is only fraud if there is deliberate misrepresentation somewhere. So far nobody's shown me proof of any case of such a thing - Steven Mosher's claim was the closest I've seen, and at least one part of it, as I showed here, simply was absolutely wrong. And that's all I'm looking at in this case. Further comments on the subject of whether or not tree rings are a valid proxy or whether they were at all justified in truncating at 1960 or 1980 I'll consider off topic here... thanks.

Arthur, the question isnt

Arthur, the question isnt whether they are valid proxies or not. The question was and remains what is the proper way to show the uncertainty. We can have that discussion without even discussing tree rings. I spent a few pages discussing a variety of options that were available to Briffa. And the primary thrust of the discussion was not the splicing, but rather understanding the rhetorical effect of briffa's choice and the context of his choice. Briffa had choices WRT the deletion. The question in my mind was NOT the justifiablity of his choice to truncate. In the book I describe this as a legitamate choice. My focus was trying to put that choice in context. The context of what overpeck was asking of him and what McIntyre was asking of him.

I don't understand what this

I don't understand what this blog post is trying to prove. The issue is really pretty straightforward I think. As McIntyre has pointed out, Phil Jones under questioning has not stated that there is anything wrong with the Briffa data. No compelling reason has been offered to discard it other than that it diverges from the temperature record.

But of course that is the crux of the problem. If the tree-ring data diverges from the temperature record it brings into question tree-ring reconstructions as a valid proxy which is of course what Jones, et al were trying to hide.

Your readers should refer to Fig 5 of the 2000 Briffa paper:

http://post.queensu.ca/~biol527/Briffa%20and%20Keith%202000.pdf.

It clearly shows a divergence of proxy temperatures from the temperature record by more than two standard deviations. This is what Jones truncated.

Here is a paper that might help make sense of the current uncertainties of tree-ring reconstructions:

http://www.springerlink.com/content/873486u687j56246/fulltext.pdf

My take is that tree-ring proxies are far too uncertain and shouldn't be used until a reliable methodology emerges. But there is no getting around the fact that what Jones did was pure deception and an honest scientist knows better.

There are many issues; this

There are many issues; this blog post is not trying to say anything one way or the other on whether tree ring proxies are good measures of temperature or not. The purpose of this post is to investigate a specific clear allegation that was made by one of the significant players in hyping "climategate", Steven Mosher, about what the scientists did. A claim that, if true, would have been another instance of a false statement in the most recent IPCC report - and perhaps more significant than the others that have been found, since it was in the first (WG1) report.

But Mosher's claim was false; the scientists were absolutely clear and upfront in their presentation of the data in this case, in contrast to the claim Mosher made.

"If the tree-ring data

"If the tree-ring data diverges from the temperature record it brings into question tree-ring reconstructions as a valid proxy which is of course what Jones, et al were trying to hide."

That OF COURSE is not only not what they were trying to hide (they were trying to "hide the DECLINE", remember?) but is also not what most of their critics claim they were trying to hide. In other words, you're just making this up to support your ideological belief that climate scientists are involved in some huge conspiracy to delude people.

Within the SCIENTIFIC discussion, to which you pay no attention, there has been much discussion of the divergence problem and what questions it raises. The question of whether the tree-ring constructions are, prior to the divergence, a valid proxy is not a persistent one among INFORMED people because of the agreement of the reconstruction with numerous other proxies.

Cicero concludes- "My take is

Cicero concludes-

"My take is that tree-ring proxies are far too uncertain and shouldn't be used until a reliable methodology emerges. But there is no getting around the fact that what Jones did was pure deception and an honest scientist knows better."

And I totally agree.

When the emails are referred to as "Stolen" - you kinda get the idea as to what side of the fence the author is sitting on. My take is that someone on the inside appalled at the CRU's stance re valid FoI requests decided to release the whole damn lot anyway!

Jones and Mann et al should be ashamed - what they did is not science - and it has cost science a great deal of credibility in the eyes of the public. How can anyone justify not using tree ring data because it does not show the results required? Dismissing such data and splicing on data you do want calls into question the assumptions and certainly the advocacy of current Climate Change science.

As noted above, the issue of

As noted above, the issue of whether tree ring proxies, or some of them, are valid temperature proxies is an interesting scientific question but it's off topic for this post, and I'll remove further comments in that direction, thanks. The question before us a specific allegation made by Mosher and, apparently, McIntyre regarding endpoint smoothing in some curves in the IPCC reports. Whether or not some of those curves shouldn't have been in the reports at all is a quite different question.

"As noted above, the issue of

"As noted above, the issue of whether tree ring proxies, or some of them, are valid temperature proxies is an interesting scientific question but it's off topic for this post, and I'll remove further comments in that direction, thanks." That is your prerogative and I recognize that blogging is a volunteer effort and I do appreciate your work to examine this issue. But I want to point out a specific error that you are making that illuminates why the trick is problematic.

You state, "[w]hat actually happens after 1960 is that some tree ring series at high latitudes diverge from both measured temperatures and other tree rings, which continue to track observed temps nicely. So we know there isn't divergence prior to the instrumental record simply because the tree ring data aren't divergent during that period." This is false.

(1) The tree ring series are long relative to the calibration period, and (2) the total population of tree ring series from which to choose proxies is large. If you select series based on their correlation to instrument data during the brief calibration period, there's some non-zero probability that the correlation is specious (or, more precisely in this case, purely coincidental). Define the initial number of chosen series as x(1) and the probability that these series coincidentally correlate to instrument data during the calibration period as p(1). After 1960, some of the tree ring series are shown to no longer correlate and are removed from the population. The remaining number of series in the new group of series is x(2)=X(1)-n where n is the number of series removed. The probability p(2) that the new group of proxies (being a mere subset of the original group) coincidental correlate to the instrument record during the calibration period is still non-zero and, in reality is not materially different from p(1) except that p(2) > p(1) since all that you have done is to reduce the population.

[apsmith: making an exception here, but please, this really isn't something I'm familiar with or particularly germane to the question at hand. Whether certain series make sense or not is a question of scientific opinion and judgment, not a matter of fact, despite either side's efforts in that direction.]

The analysis confirms my own

The analysis confirms my own in most respects (although I haven't blogged on it yet).

A couple of corrections, though. The IPCC version of Briffa was rescaled and rebaselined to a common baseline with the other curves. You can see this as an overall "warming" of the Briffa curve throughout TAR 2.21, compared to Briffa's version. You can also see the last little hump is above zero in the IPCC version, but below in Briffa's.

This changes the impact of padding with instrumental version - in fact, it makes it even clearer that the instrumental version was *not* used for this series. However, it turns out that it is possible that "zero" padding was used. Although the difference between end-point mean and "zero" padding at the 1960 end of the graph is virtually indistinguishable, the difference in the earliest part of the Briffa curve can only be explained by zero padding, as far as I can see. The net effect of this (presumed) mistake is to make the start point warmer, along with a spurious decline at the beginning of 15th century. Hardly evidence of a "warmist" plot.

I believe that UC's analysis was based on the MBH99 curve, which Mann himself has acknowledged was padded with corresponding instrumental values (there is a comment and justification to this effect at RC). However as far as I know no one has actually attempted a demonstration of this for the Briffa curve and in fact such a demonstration is not possible.

One last point. It's true that Briffa's own truncated smooth did use post-1960 points. To me that was probably incorrect - especially since the archive only goes up to 1960. If post-1960 is considered spurious and wasn't even used in calibration, then it probably should not have been used for padding either. Be that as it may, when Osborn sent the series to Mann in October 1999, he recommended truncating at 1960. That series didn't make it into the First Draft in any case as it was too late, and there was no backing reference.

The second time Osborn sent the series, following Mann's request for final versions for the Second Draft, he sent the same series *but this time explicitly truncated at 1960*. So smoothing with post-1960 values was an option that was not even available to Mann. This latter point is detailed here:

http://deepclimate.org/2010/05/14/how-to-be-a-climate-science-auditor-pa...

I'll be looking at all of these issues (as well as touching on what McIntyre himself has said on the subject of Briffa end point smoothing) in a future Deep Climate post.

No, you are quite wrong here.

No, you are quite wrong here. Briffa 2001 is not rebaselined in the AR4 plot.

When the 5 year gaussian smooth is used instead of the 10 year smooth, everything lines up without rebaselining, including the transition over the y-axis zero in the late 1930s, and indeed the 5 year smooth brings the instrumental padding down below -0.1 near the end of the line, which means the instrumental padding is rather closer to the actual chart than the 10-year smooth example shown above.

UC correctly reverse engineered what Phil Jones referred to as "Mike's Nature Trick", which was pivotal to subsequent analyses. Jean S completed the analysis of the WMO statement, which is Phil Jone's misapplication of Mike's original trick. On checking more carefully, I'm not sure who did the TAR work. You can see this sequence at this CA post, including Gavin initially denying the original trick, then Mike "clarifying" it (i.e. finally owning up after the cat was out of the bag, contradicting Gavin to boot), then Gavin throwing in the "ten year old paper" line that is always a good one to use when it becomes clear that defending the indefensible isn't such a good strategy after all.

Hopefully, at some point, they will realise being straight and upfront from the start avoids these problems.

I didn't mention AR4. Briffa

I didn't mention AR4. Briffa is rebaselined and rescaled in TAR without question. McIntyre noticed the upward displacement of Briffa recon, although he wasn't sure why it was done.

OK, your third para reads as

OK, your third para reads as if you're comparing to Arthur's plots, which are AR4. But if you're not, fair enough.

As for the TAR, we already know if you follow the recipe given you get a very different graph. For some background, it would be useful to include a background link - here is McIntyre's analysis. McIntyre shows here that the TAR graph really is a piece of work, produced back at a time when I can only assume climate scientists didn't think they were going to be terribly closely scrutinised.

I tried using the data Arthur links to above with a 40-year Hamming window, and the result does not match in shape or position. My chart matches McIntyre's very well. Pretty astonishing for what is supposed to be such an important document - people churning out data in their spare time can get it right, but the IPCC couldn't.