Arguments
Kevin Cowtan Debunks Christopher Booker's Temperature Conspiracy Theory
Posted on 27 January 2015 by Kevin C, dana1981
In The Telegraph, Christopher Booker accused climate scientists of falsifying the global surface temperature data, claiming trends have been "falsified" through a "wholesale corruption of proper science." Booker's argument focuses on adjustments made to raw data from temperature stations in Paraguay. In the video below, Kevin Cowtan examines the data and explains why the adjustments in question are clearly justified and necessary, revealing the baselessness of Booker's conspiracy theory.
The video features a prototype tool for investigating the global temperature record. This tool will be made available with the upcoming MOOC, Making Sense of Climate Science Denial, where we will interactively debunk myths regarding surface temperature records.
Thanks Dana. That was an interesting exercise. Here are a few comments:
Thank you, Kevin. That video was so elegantly simple even I understood it!
Thanks for this. It is excellent and a serious help for those of us who confront the crazies.
AFAIK, most changes in data records are either due to changes in sensor type (like going from mercury in glass to electronic sensors) or changes in hour of recording (including how to calculate daily averages). In a particular country, this (a break in teh record) was then probably due to a nationwide switch made by the national body in charge of those measurements. Have to ask our South American readers here to chime in ...
Over on notalotofpeopleknowthat (Booker's source), Paul has posted another exposé titled All Of Paraguay’s Temperature Record Has Been Tampered With, looking deeper at all the sites in Paraguay.
In the comments someone called Eliza says (among other things):
Which is the period around that first drop in the raw data for several of the Paraguay stations. They don't seem to have noticed the connection over there.
@4:
That's exactly what it looks like. Should be fairly simple to check with the Paraguayan met office.
I wonder if the Washington Post would be willing to lead the investigation into improperly validated reporting of climate change claims like they did regarding the Rolling Stone article about unacceptable activity at UVa.
It would seem reasonable to challenge every media that reported that particular piece of Washington Post leadership on honest reporting to give equal billing to every case of a similarly inadequately validated report of a climate disruption claim.
And an article identifying the media that would not repeat such reports, even though they repeated the Rolling Stone one, could be the culmination of their efforts, exposing which media is actually not trust-worthy.
Telegraph. Not the first time they are in error. Might call them opinionated and incapable of learning of their errors. Some call this 'news'paper 'the torygraph', which quite well fits in my view these sorts of extreme rightwing 'newspapers' are nothing more than public, but encoded, message boards for their readership consisting of torys, the meaning of which derives from the Middle Irish word tóraidhe; modern Irish tóraí: outlaw, robber or brigand.
Dr. Cowtan's final argument was that the adjustments were too tiny to be suspicious. “Why would they do that?” he asks at the end of his video, meaning why would anyone commit fraud for an inconsequential difference in the result?
However, I digitized the endpoints in the last graph which he showed in his video, and found that NOAA's adjustments had increased globally averaged land surface temperatures by 35%.
35% is not inconsequential.
The graph w/ comments, and a small spreadsheet with the numbers, are here:
http://tinyurl.com/cowtan-v-booker
Here's a half-size version of the annotated graph:
[RH] Adjusted image size.
I didn't say that very well.
"...had increased globally averaged land surface temperatures by 35%"
should have been:
"...had increased the warming seen in globally averaged land surface temperatures by 35%"
DaveBurton,
You have mislabeled your graph. In the video the graph is clearly labeled NOAA Paraguay data. You have changed the label to Globally averaged Land Surface data. Dr. Cowtan points out that such a change in the Paraguay data would only affect Global data by 3%.
How could you make such a large error by mistake? It appears that you are deliberately trying to mislead whoever you show your mislabeled graph. This must be a conspiricy to defraud the public about this data. Who is paying you to make these fradulent claims?
It is well known that change of time of reading and change in temperature monitoring stations can make readings this different. The installation of Stephenson screens as described in 5 above woud cause this change.
It is easy to claim dramatic changes have been made when you mislabel the data.
daveburton @9 and @10, in the video Kevin Cowtan says (3:55) that "... the NOAA adjustments make only about a 3% difference to the amount of warming over the last 50 years" (my emphasis). By comparing the range from 1900-2014, you are not checking his claims, but an entirely different claim that he did not make. You may think this is a minor point, but in fact the key attribution statement for the IPCC is with regard to the period 1950-2010. That is, it is with regard to (approximately) the last half century. There are several reasons for that, one of which is the relative accuracy of the temperature record post 1950. Consequently there are substantive reasons why the IPCC and Kevin Cowtan concentrate on the most recent 50-60 years.
Further, on checking your spreadsheet it becomes very apparent that you have made your determination simply by comparing end points. Because of the high short term variability of temperatures, such comparisons can easilly mislead by showing atypical values. To avoid this, the correct method is to determine the respective trends, and compare them directly, or if you insist on comparing end points, compare the endpoints of the trendlines. This is particularly the case as it is future temperature trends we are concerned with. Comparing the endpoints for 1950-2014 (14% difference by your method), 1965-2014 (10% difference by your method) shows the effect of increasing accuracy overtime resulting in less difference between adjusted and unadjusted data, and highlights the problem with using endpoints.
Finally, although Michael Sweet is incorrect to describe the graph as Paraguay data, it is the land only data, of which Kevin Cowtan indicates their is a 10% difference after adjustments. That 10% is reduced to 3% by the inclusion of SST data, and we could expect a similar scale of reduction in your figure if it compared land/ocean temperature data rather than land only data. That is, your 35% would reduce to approximatly 11% difference over a 115 year interval, or less than 1% difference per decade.
Zeke Hausfather has an excellent post on adjustments to temperature, though at an unfortunate site.
michael sweet wrote, You have mislabeled your graph. In the video the graph is clearly labeled NOAA Paraguay data. You have changed the label to Globally averaged Land Surface data."
No, Michael, that graph is global. Listen to Dr. Cowtan describe it, at 3:30:
https://www.youtube.com/watch?v=qRFz8merXEA#t=210
He says, "We can calculate a global land temperature record using..."
Tom Curtis wrote, "in the video Kevin Cowtan says (3:55) that "... the NOAA adjustments make only about a 3% difference to the amount of warming over the last 50 years" (my emphasis)."
You are correct, Tom, but his graph was for the last 114 years, and his conclusion ("why would they [bother to] do that") only makes sense if the sum of NOAA's adjustments were inconsequential. It doesn't make sense if NOAA's adjustments were only inconsequential for one cherry-picked time interval, and very consequential for the whole record.
For instance, if you compare the adjusted to unadjusted warming for the first 70 years of the graph (1900 to 1970), instead of the last 50 years, you'll find that the adjustments increased the net warming by more than 900% (from 0.025 °C to 0.257 °C).
Of course, you're probably saying to yourself, "but that's cherry-picking!" — and you're right. But no more so than just examining the last 50 years is cherry-picking.
Tom continued, "you have made your determination simply by comparing end points. Because of the high short term variability of temperatures, such comparisons can easilly mislead by showing atypical values."
You're right about the temperatures, Tom, but that's not what's being compared (by Dr. Cowtan, and by me). What's being compared is the temperature adjustments (i.e., the difference between adjusted and unadjusted temperatures), and they do not exhibit much short-term variability.
Tom continued, "it is the land only data, of which Kevin Cowtan indicates their is a 10% difference after adjustments."
But that's only for the cherry-picked last 50 years.
Tom continued, "That 10% is reduced to 3% by the inclusion of SST data."
SST is not air temperature, and, unfortunately, the SST data quality through most of the 20th century is even sketchier and subject to more adjustments than the land air temperatures.
Tom continued, [if you multiply by 30% because only 30% of the globe is land] "your 35% would reduce to approximatly 11% difference over a 115 year interval, or less than 1% difference per decade."
You can't do that, Tom. It's mathematically incorrect. The percentage difference isn't a sum that you can divide by 11.5 to get the per-decade adjustment-generated percentage difference in warming, it's an average.
No, Dave Burton, it is not okay to compute trends by merely connecting end points. You need to compute a regression line. Seriously, a 900% change?!
Dave Burton, you are incorrect that the temperature anomalies do not exhibit much short-term variability. For example, see the escalator graph.
daveburton @14:
1)
I indicated a couple of very substantial reasons as to why we should concentrate on the period since 1950. Because there are substantive reasons for the choice of time period, it is not a cherry pick. Suggesting that it is without adressing those reasons amounts to simple name calling as an argument. However, leave that aside.
The simple fact of the matter is that the majority of early twentieth century warming is natural in origin. Anthropogenic factors account for approximately a third of that warming, or according to one recent analysis, effectively none of it. This can be seen by the relative slopes of temperature and CO2 concentration in the early, and late twentieth century:
More substantively, comparing model retrodictions with global temperatures shows the models substantially underpredict the global trend in the early twentieth century:
(Note, the pink shaded area is the model prediction for anthropogenic plus natural forcings.)
That discrepancy can be accounted for almost completely by reference to short term variability not included in model forcings, primarilly ENSO, although the unusual warmth in the 1940s is still not accounted for.
So, with this knowledge in hand, your argument then becomes that the people at NOAA increased temperature trends in the early twentieth century when such increases run contrary to the AGW narative, but did not significantly increase trends in the late twentieth century where such increases would have supported the narrative.
Expanding the analysis beyond the late twentieth century does not make the theory of dishonest adjustments any more plausible. It just means you need to understand more background information to understand the relevance of what is happening.
2)
Here are the difference between raw and final USHCN temperatures:
Very clearly, you are wrong to claim "...they do not exhibit much short-term variability". There is substantial short term variablity, particularly in the early twentieth century, and particularly for 1900 itself. That is, of course, for the contiguous United States. Unfortunately I do not know of a similar chart for the global land record, but there is no reason to think it would also lack variability from year to year in adjustments. So while the adjustments show less short term variability than do the actual temperatures, never-the-less, they show substantial short term variability and comparisons of differences should be made using trend values.
3)
First, it would be nice if there was a more or less continuous, global record of air surface temperatures at 2 meters above sea level. Unfortunately no such record exists, so you make use of the records you in fact have rather than pretend to complete ignorance. Those records are, however, strongly relevant. As anybody knows, who has lived both inland and near the sea, the thermal mass of the surface waters mean they absolutely dominate the variability in nearby air masses. That is so even 50 km inland, let alone a mere 2 meters above the surface of the water. Although there will be some difference in the actual values between SST and 2 meter air temperature immediately above the surface, they will not be large and they will be consistent so that trends in the former would have been reflected in trends in the later.
Given this, I regard your exclusion of SST from the record of interest as a mere cop out. It is particularly the case given that all Global Mean Surface Temperature records use SST data over oceans, for want of any better record to use.
Second, the SST temperature data is in fact very extensive, although it does show some coverage lapses (as does the land record), particularly in the Southern Ocean.
Third, while the SST data is also subject to adjustments, they are not subject to "the same adjustments":
(Source)
As you can see, while the net effect of adjustments on land has been to cool the early twentieth century relative to the late twentieth century, the net effect on SST has been the opposite. That is, it warms the early twentieth century relative to the late twentieth century. That is, it reduces the twentieth century temperature trend.
So, if we look at the actual data, it does not support your dismissal, which is shown to be glib, and misleading. Contrary to the impression you convey, combining land and SST data will reduce the global trend from 1900 more than would be the case by simply diluting the 30% land coverage with data with no trend adjustment.
4)
Then it is a good thing that that is not what I did. Rather, I projected the difference in adjustment over the last 50 years back over the record to 1900. That, of course, only gives a ball park figure, and given the additional data noted at point three above, underestimates the reduction in the difference for global figures. It is near enough, however, for this discussion.
Tom Dayton @16, daveburton was incorrect (see my point 2 @17 above), but neither he nor I were talking about the simple anomaly on that point. Rather, we are talking about the adjustments, the difference between raw and homogenized data, which also shows substantial short term variability. Just not as much as the actual anomaly, and much less over the last 50 years.
Tom Dayton, (re #15) the 900% example was intended as an illustration of the problem with using a cherry-picked subset of the full time series. It is an example of what not to do. But it is computed accurately. Seriously. It really is >900%.
However, (re #16) I did not say that the temperature anomalies do not exhibit much short-term variability. I said that NOAA's adjustments do not exhibit much short term variability. Which is right.
I concede, however, that when the adjustments are compared to the unadjusted temperature trend, e.g. by computing the adjustment amount as a percentage of the trend, the result is senstitive to both numbers. However, I did not pick an anomolously low right-hand endpoint to minimize the early 20th century warming. 1970 actually represented a minor peak in temperature: it was warmer than 1967, 1968, 1969, 1971 & 1972.
Tom Curtis, just to be perfectly clear, I said nothing about "dishonest adjustments."
It is very possible to to make incorrect adjustments without conscious dishonesty, though simple error, or unrecognized bias. There's a very human tendancy for scientists to scrutinize data more critically if it is contrary to their preconceptions, than if it confirms them. That tendency can easily introduce bias in the reported results, despite everyone involved having the very best of intentions.
NOAA used to have, on their web site, a description of how they adjusted measured temperature data based on what seemed to me like a very crude proxy for urban heat island effect: satellite observations of nighttime illumination. That material is gone from their site, now, and they've blocked archive.org from archiving such materials, and I didn't save a copy (at least I don't think so), so I don't have the details. But the procedure seemed highly dubious, to me, when I read about it a few years ago.
That doesn't mean that those adjustments were dishonest, however.
My point wasn't about motives. I was simply addressing the misconception that NOAA's adjustments made a negligible difference in the reported result. They don't. They make a big difference.
Dave Burton, you have revealed NOAA to be profoundly incompetent indeed, because their sinister attempt to hide the explanation of homogenization was so poor that a quick internet search revealed it in several place such as the CDIAC site. Congratulations.
daveburton @20, first and most importantly, your comments about the phrase "dishonest adjustments" are entirely beside the point. You completely fail to adress the substantive points I made in my comment @17, failing in particular to adress the clear evidence that there is substantial year to year variability in adjusted data, particularly in the early twentieth century. You further fail to adress the fact that the adjustments to land data, and to SST data in the early twentieth century have opposite effects on the trend. Your failure to adress these and other issues makes your comments about the phrase "dishonest adjustments" look like a simple, and deliberate distraction from the fact that your points have been refuted in detail, and that (apparently), you have no substantive response.
Second, you claim to "have said nothing about 'dishonest adjustments'". Never-the-less you are commenting on a critique of Booker's article, and Booker definitely suggests the adjustments are dishonest, calling them "scary chicanery". You, yourself summarize Booker's article by saying:
That is, according to you Booker is saying the adjustments were made with the intent to "... inflate the reported global warming, and fan the flames of climate alarmism", ie, that it was done from dishonest motives. As I did not indicate that it was your theory, my language was entirely justified and your attempted distraction is itself ungrounded.
To address concerns about comparing data only for the endpoints (1900 & 2014), I went back and used WebPlotDigitizer to digitize a number of intermediate points. I digitized all years ending in "0" so that decadal trends could be calculated starting with year 1900, which is the earliest year in Dr. Cowtan's graph. I also digitized years ending in "4" so that decadal trends could be calculated ending with year 2014, which is the latest year for which data exists. (I didn't include the other years because digitizing all that data is very tedious!)
Using WebPlotDigitizer, I digitized the temperatures for each plot (unadjusted temperatures and adjusted temperatures) for the following years:
1900, 1904, 1910, 1914, 1920, 1924, 1930, 1934, 1940, 1944, 1950, 1954, 1960, 1964, 1970, 1974, 1980, 1984, 1990, 1994, 2000, 2004, 2010, 2014.
(Note: I started over, so some digitized values are very slightly different from the values I previously obtained.)
I loaded all the data into a spreadsheet in Excel, with these column headers:
year unadj_temp adj_temp
I then added ten calculated columns:
adj-unadj = the difference between adjusted and unadjusted temperature.
50yr_unadj_diff = the difference in unadjusted temperatures between the designated year and fifty years earlier. If positive, it indicates warming compared to fifty years earlier; if negative, it indicates cooling.
50yr_adj_diff = the difference in adjusted temperatures between the designated year and fifty years earlier.
50yr_%warm_from_adj = the percentage of the warming which is due to the adjustments, over 50-year intervals, starting with 1900-1950, and ending with 1964-2014. (Percentage is shown as "100%" if the unadjusted data indicated cooling.) The results were:
100.0%, 22.2%, 39.6%, 100.0%, 54.4%, 100.0%, 48.1%, 96.8%, 21.6%, 30.9%, 20.7%, 17.0%, 7.7%, 7.5%.
Note that, over the last 50 years (1964-2014), only 7.5% of the warming is due to adjustments (i.e., the adjustments increased warming by 8.1%). However, of all the 50-year intervals, that interval shows the lowest percentage of warming due to adjustments.
70yr_unadj_diff = the difference in unadjusted temperatures between the designated year and seventy years earlier.
70yr_adj_diff = the difference in adjusted temperatures between the designated year and seventy years earlier.
70yr_%war_from_adj = the percentage of the warming which is due to the adjustments, over 70-year intervals, starting with 1900-1970, and ending with 1944-2014:
89.0%, 43.0%, 28.6%, 75.8%, 23.7%, 27.4%, 35.6%, 27.2%, 16.2%, 15.9%.
114yr_unadj_diff = the difference in unadjusted temperatures between 2014 and 1900 (0.86 °C).
114yr_adj_diff = the difference in adjusted temperatures between 2014 and 1900 (1.16 °C).
114yr_%war_from_adj = the percentage of the warming which is due to the adjustments (26%).
114yr_increase_by_adj = the percentage by which the adjustments increased the reported warming (35%).
I added all these files to my little web page about this argument between Booker and Cowtan:
http://tinyurl.com/cowtan-v-booker
The screenshot files (with and without digitization points added), digitized data files, spreadsheet, WebPlotDigitizer calibration & data file, and notes are all available in a convenient .zip archive, in case you want to check my work or do additional calculations.
One last note: to address Tom Curtis's concern, based on the USHCN adjustments, that the endpoints migh have anomolous adjustments, I used Excel to plot adj-unadj for the 24 years which I digitized. If you download the spreadsheet and load it into Excel, you'll see that it looks like it's almost linearly increasing, though because I digitized only 20% of the years it's obviously incomplete.
This is the Excel-generated plot of adj-unadj (adjusted minus unadjusted temperatures) for the 24 years which I digitized.
Because I digitized only 20% of the years it's obviously incomplete, and because the years I digitized aren't quite evenly spaced the horizontal spacing of the plotted points isn't quite right, and because it's manually digitized from a screencap of a graph it's not completely accurate, but it's a better reflection of NOAA's global land surface temperature adjustments than "the difference between raw and final USHCN temperatures" (U.S. only) which Tom Curtis posted:
[RH] Adjusted image size. Please try to keep images to 500px width.
If one is going to compare adjusted temperature anomalies to unadjusted temperature anomalies, ought not the unadjsuted temperature anomalies be calculated relative to the unadjusted mean temperature? Otherwise you are calculating an anomaly by comparing an unadjusted temperature to an adjusted mean.
Or am I missing something?
daveburton @24, thankyou for the efforts at digitizing. I notice that while the difference over 115 years between 1900 and 2014 is, by your digitization, 0.3 C, or 35% of the 0.86 C unadjusted increase; the 111 year difference (second value on the graph) is approximately 0.23 C. That represents a 23% reduction in the adjustment. The following value shows an increase in adjustment so the decrease is not monotonic. Surely you are not going to argue that a 23% difference (30% using your preferred method of calculating percentages) is insignificant. If you are not going to make that argument, however, my point about the sensitivity to end points if you do not use the linear trend to calculate the difference is proven. Indeed, it appears that you have added about 30% to the reported adjustment by, what is afterall, an arbitrary choice of endpoint.
My other points remain unaddressed.
From the posted U-Tube, Kevin Cowtan tells us:-
@23 daveburton tells us he calculates this last 50-year land temperature' figure as 7.5%. It appears there is a level of agreement for the last 50 years and we can move on.
daveburton @23 stresses the 35% figure for the 115-year record. This is far higher than for the last 50-year period. Indeed @23 we are told that a whopping 89% of the global land warming 1900-70 appears only within the adjusted data.
What I therefore find bizarre in this accusation that the adjustments are poorly founded (with heavy hints of conspiracy wafting about to boot) is that climatology has busted a gut trying to explain the early twentieth century temperature record. If the adjustments were as dodgy as suggested by daveburton, why has nobody struggling to explain the adjusted land temperature figures broken ranks and used an alternative adjustment methodology?
I would hazard a guess at least one reason. The early twentieth century SST data would look like they came from a different planet and be very difficult to explain if the NOAA adjustments (and all the others which reach the same conclusion) were truly in gross error.
RH/moderator, thank you for fixing it, and please accept my apology; I'll trim/shrink to 500px in the future.
davidsanger, I'm sorry, I don't understand your question.
Tom Curtis, I don't think it is reasonable for you to characterize starting at the beginning and ending at the end as "arbitrary." I do agree that it would be better to do a regression fit, rather than just using the endpoints, but to do that properly that would require all 228 data points, and digitizing just 48 of them was tedious enough.
If you want to digitize the rest, please do so. You can start where I left off; I've posted a link to all the relevant files.
When you're done, I don't think you should use a linear fit, though. It looks curved to me. I think a quadratic fit would better capture the trend.
To me it looks like the endpoints on the "adjustments" graph are roughly on what would be the quadratic-fit curve. So I think if you do all that work you'll get about the same result. (I've written some Perl code for doing quadratic fits, though; I'd be happy to share it, if you want it.)
The unadjusted temperature curve (for the denominator when calculating the percentage by which adjustments increased reporting) is more complex. It has about three inflection points (around 1925, 1955 & 1995), so a 5th degree polynomial would be needed to fit it, or a spline.
Tom, you also say, "My other points remain unaddressed." I guess you're talking about your argument that it is reasonable to discuss only the warming added by adjustments made to temperature measurements taken during the last 50 years, rather than the full 114. But I thought I did rebut that adequately, when I showed that that was the period for which the adjustments contributed the smallest percentage of warming.
For the ten 70-year periods, the amount by which the adjustments increased the global land surface temperature warming varied from 18.9% to over 700%, and for the full 114 years it was 35%.
Remember the question being addressed in this discussion: was Dr. Cowtan right or wrong when he argued that suspicion about NOAA's adjustments could be dismissed because they resulted in an inconsequential difference in the result? So why, then, if you want to answer that question, would you examine only the period for which those adjustments made the least difference?
That's like a student arguing that, because he only got 7.5% wrong on one of his quizes, he should get an "A" in the course, even though on most of the quizes he got over 30% wrong.
MA Rodger, I agree with your first three sentences. But then you rebutted an "accusation" that you think I made: an "accusation that the adjustments are poorly founded (with heavy hints of conspiracy wafting about to boot)."
Where do you think I made that accusation? Please quote it.
When you rebut something that someone hasn't actually said, it's called a "strawman." As I said to Tom Curtis @20, "My point wasn't about motives. I was simply addressing the misconception that NOAA's adjustments made a negligible difference in the reported result. They don't. They make a big difference."
I acknowledge that I have complained (mostly elsewhere) rather bitterly about the habit of the currators of the temperature data at NASA & NOAA of blocking the archiving of data and documents, by various means. I think that's suspicious-looking. But, in truth, I don't know whether the adjustments are right or wrong, and that wasn't my point. My point was simply that the adjustments are far from negligible, so trying to defend them on grounds of supposed negligibility doesn't work.
[JH] You are now skating on the thin ice of excessive repetition which is prohibited by the SkS Comments Policy.
daveburton @28.
I'm so sorry. I didn't realise we were playing troll rules. Forgive me while I chase any potential strawman from the scene.
You do not accuse NOAA of error, directly. You do say you are not sure if NOAA are correct or in error. So to be pedantic, you either accuse them of potentially being in error or you have no knowledge in the matter. Yet evidently you do consider the NOAA adjustment of raw temperature data to be an issue, something people are apparently "tryng to defend." This brings much emphasis to the likelihood of you knowing what you are about and also that you strongly countenance an NOAA error to exist. You may hide behind a figleaf of remaining doubt assumed by you for sake of argumentation, but your position is evident. Your position accuses NOAA of error.
You do say you see evidence to support suspicions of wrong doing, of the NOAA being "up to no good" and, as you link to such a comment from here, the location is of no consequence.
I stand by every syllable of my quote that you present @28.
As to your "point," I am not sure what you are about.
The alleged "misconception that NOAA's adjustments made a negligible difference," is surely unfounded. Kevin Cowtan is quoted by me @27 and it is plain to all but you that any "negligable difference" being ascribed to these NOAA adjustements applies to "the last 50 years." Why then do you persist with this "misconception" nonsense by arguing that the "difference" over the last 115 years is not "negligible"? Is this because you want to conflate a non-negligible adjustment with your accusations of error and wrong-doing?
To better understand what David Burton is attemtping to accomplish on this comment thread, go to: http://www.sealevel.info/Cowtan_unintentionally_vindicates_Booker.html
davidsanger
"If one is going to compare adjusted temperature anomalies to unadjusted temperature anomalies, ought not the unadjsuted temperature anomalies be calculated relative to the unadjusted mean temperature? Otherwise you are calculating an anomaly by comparing an unadjusted temperature to an adjusted mean.
Or am I missing something?"
Yep.
Anomalies aren't calculated relative to a mean, at least not a global mean. For each station the anomaly for any reading from that station is calculated relative to a baseline average for that station. Then these anomalies are averaged (with an area weighting) to give the global mean.
Averaging stations to get a mean then calculating anomalies as differences in the mean is highly vulnerable to issues with station availability. local station biases etc. By basing the analysis on comparisons of a station against it's own base line, many local biases cancel out. You get a more robust result.
Glenn, I think the question remains whether the anomaly for the station is calculated against the adjusted or unadjusted mean for that that station. Adjustment will surely change that mean value.
I understand that the anomalies are based on the temperature record for a specific station, Glenn@30 , but my point, as @scaddep@31 noted, was that if you look at unadjusted anomalies then you need to recalculate the baseline using unadjusted temperatures.
Looking at Dr. Cowtan's source data it is indeed evident that is what he has done.
The effect of adjustment on GHCN V3 trends is gicven by Lawrimore et al, 2011, Table 4. For the century, land only, it typically increases trend from about 0.7 to about 0.9 °C/Cen. Of course, for global that reduces to a difference of about 0.05, which is about what I got here.
First, and in response to various comments, the correct way to compare the effect of adjustments, if you are gong to use the endpoint method, is to take (Tfa-Tia)-(Tfu-Tiu), whereTfa is the final tempertaure in the adjusted series, Tia is the i-th temperature in the adjusted series, and likewise in the second parenthesis for the unadjusted series. If you include terms from the adjusted and unadjusted series in the same brackets, a difference in the baseline period could be interpreted as a difference in the actual adjustments. This, I believe, is the point raised by davidsanger, and responded to by Glenn Tamblyn. However, as daveburton used the correct method, ignoring the fallacies of that using endpoints is itself an error, I do not see the relevance.
Second, in response to daveburton @28, I have now digitized the first 21 and the final values in the adjusted and unadjusted series. Applying your method of determining the difference in adjustment, using each of the first 21 years as start points and 2014 as endpoints, I obtained adjustment percentages ranging from 18.85 - 35.9%. The mean value was 25.5% with a standard deviation of 4.98%. You will notice that using 1900 as a start year gives a value 1.97 standard deviations above the mean. In fact, for the first 21 years, 1900 gives the second largest value after 1915. I trust you will concede that an increase almost two standard deviations, or 38.4% over the mean value, is sufficient to consider the 1900 value an outlier. If you do not, I see little further point in this discussion.
Interestingly, the variance in the percentage value is much greater than the variance in the adjustment itself. For the actual adjustment, 1900 shares the maximum adjustment with 1901, at 0.3 which is 1.71, but only 12.7% greater than the mean value (0.266 C). The reason for the increased variance is that by taking a percentage, you introduce the difference between the anomaly values into the term, thereby making the year to year variance in temperature a significant factor, contrary to both of our claims above.
Please note that the temperture trend is essentially zero over the first 21 years (0.0022 C/decade adjusted; -0.005 C/decade unadjusted), so while year to year variability is, the trend is not a significant factor when analyzing the adjusted percentages.
This brings me to the third point, specifically, there is a crucial difference between "arbitrary" and "cherry picked". 1900 is an arbitrary start year because the fact that it happens to be a centenial year (and hence was chosen as the first year of the data) is a fact relating to human customs, and has no physical significance. Had we used a base eight mathematical system, or had a different religous history in Europe, some other year would have been chosen. In like manner, 1880 is an arbitrary year with regard to the temperature record, being chosen as the first year of the temperature record (for NCDC and GISS) because of historical facts about when and where people started keeping temperature records. Because these years are arbitrary, we need to take great care to avoid endpoint effects which can significantly distort an analysis (as I think I have shown it has distorted yours).
That is quite different from cherry picking, in which an arbitrary endpoint is chosen because it distorts the analysis. So, while 1900 is an arbitrary start point, it is not a cherry pick (and nor have I accused you of cherry picking).
Note further that while 1950 or 50 years ago are also arbitrary endpoints, the choice of an endpoint approximately in that range is not because it is approximately at that time anthropogenic forcings began increasing at a very rapid rate (whereas before hand they had increased slowly, or not at all according to some determinations). Further, that also corresponds to a high point in the AMO so that temperatures increases since them have been decreased by the AMO if affected significantly at all, ensuring that nearly all of the increase since then has been anthropogenic. Further, in terms of understanding the science, 1950 is the year the IPCC chose as the start point of its attribution claim so that if we want to understand how much adjustments affect that claim, we need to use that as the start point. These are the various non-arbitrary reasons for using the shorter period that you have not addressed.
Finally, I intend to fully digitize both the GHCN3 and HadSST3 adjustments on a publicly available spreadsheet, as I think the results will be interesting independently of this discussion. That may, however, take a couple of weeks so I hope you will be patient waiting for the full data that verifies my claims above.
Nick Stokes @33, glad to have a true expert on the topic particpate. I have my doubts about the 0.05 C figure for global values, primarilly because the SST data is also adjusted, but in the opposite direction. Therefore it does not merely dilute, but cancells the effect of the land station adjustments (in agregate). As a result, depending on the figures and the SST record used, the global adjusted figure will be less than 0.05 C, and may even be negative. Do you have any comments on that?
Tom@35 You don't need to digitize the data. It is available at Dr. Cowtan's GHCN calculator.
Just select "station" and "add all" and the either "adjusted"or "unadjusted".
For anomalies check the box "aligns stations on common baseline". It does compute each anomaly seriers independently.
For temperature leave the box unchecked. Be sure the "area weighting" box is checked.
Then press calculate and it will add the series to the display. It can only show two series at the same time, but you can easily download the data as textfiles.
I ran a simple least squares on the adjusted anomaly for the past 50 years (1965-2014) and got a trend of +0.0230 degrees a year; the unadjusted anomaly showed a trend of +0.0214 degrees a year.
No doubt as the tool is refined and more people have access others can check these and other results.
Thanks, Tom@35. Victor V made the same point about SST here. I guess you are referring to Fig 2 in KevinC's post. Unfortunately, it's harder for SST to pin down (let alone find numbers for) a "raw" dataset, and so is hard to do a corresponding analysis.
Which all illustrates the silliness of the whole adjustments accusation. Adjustment is just the process of trying to a good set of data for global analysis. Relating it to where you start from just depends on whether you started from a good place or not. With both SST and MSU, say, it's really hard (and pointless) to pin down a unique starting point.
David: The GHCN tool is unfinished, frequently broken, and not ready for release, I'm not sure how you got the link. I've removed it now.
It's mainly meant as a teaching tool. The rather more sophisticated SkS tool can do a better job, and can do land-ocean series.
Tom: The HadSST3 raw data are available as grids from Hadley. I can give you a temperature series if you want, or you will shortly be able calculate it in the SkS tool in CRU mode (note it at this moment it doesn't read the latest Hadley SST files, I've sent a fix which will hopefully be online later today).
Here is a comparison of the series with and without the SST adjustment. It is indeed in the opposite direction and much larger than the land adjustment:
Here are the land-only results from Lawrimore Figure 16:
I may be able to get the original data for this figure if you want it.
Table 4 is this data (via Nick, adjusted to make the units match):
Kevin,
For your next video you might consider showing the sea temperature adjustments as a contrast to the land ones. If you say "Does it make sense to adjust the land temperature to increase the slope when you adjust the sea temeprature to lower the slope more at the same time?" I am sure you will phrase it better than me.
Sugggesting the scientists are fudging the land data does not stand up to evaluation when the sea temperature is brought into the argument.
davidsanger @36, thankyou for the very useful, if temporary help. Using the data from Kevin's program to which you linked, I can update my results from my post @34. So, the trend for the first 21 years are (to the first three decimal places) 0.001 C/decade for the adjusted, and zero for the unadjusted data. The mean percentage increase in the adjusted data as a percentage of the unadjusted increase is 24.3%, with a Standard Deviation of 4.79%. 1900 shows a percentage change in from adjustment of 33.99%, or 2.03 standard deviations (and 39.9%) above the mean. Using 1900 as the start point, therefore does make a significant difference to the analysis, biasing the outcome towards overstating the difference adjustments make. That slight difference is one of only two changes needed in my stated analysis. The other is that with the correct* data, the 1900 start year gives the largers adjustment difference (33.99%), now coming out ahead of 1915 at 33.75%.
More interestingly, having now the full dataset I checked the percentage change in trend for various trend lengths. Like you I got 0.23 C/decade adjusted (0.214 C/decade unadjusted) for the 50 year period mentioned by Kevin in the video. That amounts to an adjusted trend that is 107.57% of unadjusted trend (a 7.57% increase). Ergo Kevin overstated rather than understated the increase for that period. I also calculated the trends from 1900 and 1950 as:
1900-2014: 0.105 C/decade adjusted, 0.077 C/decade unadjusted (36.72% increase)
1950-2014: 0.177 C/decade adjusted, 0.153 C/decade unadjusted (15.52% increase)
Those trends break down to a approximately a 12.25% increase in the 1900-2014 global trend assuming (incorrectly) no adjustments in the SST data, and approximately a 5.2% increase in the global 1950-2014 trend on the same assumption.
* Subject to the assumption that the software was not broken in a way that introduced errors when I used it.
Kevin C @38, thankyou for responding. I have a few questions.
1) In the SST graph, I assume the All adjustments line represents a SST series including some land data from coastal or island stations. Is that correct? If not, what is meant by the "No land adustments" line?
2) With regard to the SkS tool, do I need to download a file of the gridded data to use it? Further, can you point me to instructions on how to use it, as it is not intuitive to me?
3) My intention, with the data, is to produce a jurry rigged global temperature series by taking 0.4 x land plus 0.6 times SST data as the time series, and plotting on a single graph the effects of land, SST and combined adjustments. I am aware that method is very crude, and my intent is to show the approximate impact only. I would greatly prefer if somebody who could to a more sophisticated job would prepare the graph using a method and dataset actually matching one of the three traditional temperature series (GISS, HadCRUT4 or NCDC). I understand Zeke Hausfather is thinking of doing something similar for BEST. Any chance you would be that "somebody"?
I think such a graph would be very usefull in persuading the persuadable about the bona fides of the people constructing the temperature records (at which point further rubbish such as Booker's latest will help isolate the unpersuadable). With that graph, Steve Mosher's recent comment will have real rhetorical teeth:
Zeke Hausfather has followed through:
It looks like he has used GHCNv3 and HadSST3 as I was going to, which has the convenience of access to data, but does not correspond directly to any of the major temperature series.
He also shows the data seperately:
But that does not show anything not shown already above (other than the extension to 1880).
Repeating Mosher again:
Tom: In the green box under the temperature tool there is a link to the blog post, which technically answers your question but only if you make a number of correct inferences about what is going on. So I'll try and provide a direct answer.
Land and ocean map series are reconstructed separately, and then combined. That's what all the major record providers do, because it's the right thing to do. While this is clear from even rudimentary analysis, I'm not aware of a careful treatment of the question in the literature however. It's on my to do list, but I've been a bit sidetracked recently.
As a result, if you provide only land or only ocean data, you get a pure land or ocean temperature series.
There is one subtlety however - the fill butttons. In grid box (i.e. CRU) mode, turning these off means that coastal cells get downweighted accoding to the amount or land (or ocean for an SST reconstruction) in that cell, which is the right thing to do. If you turn them on, then coastal cells count at full weight when using just land or just ocean, which overweights the coastal cells.
Off the top of my head I can't tell you for sure which the Hadley SST series do, but I think it is the latter. So the safest course is to calculate both adjusted and unadjusted series with your preferred option.
When doing a land-ocean reconstruction, the fill buttons play less of a role, because a cell with both land and ocea data is always constructed from a weighted combination according to the land fraction in the cell.
The data you need are the GHCN tavg .inv and .dat files for land, and the HadSST gridded data for the oceans. Look for HadSST.3.1.1.0.median.zip and HadSST.3.1.1.0.unadjusted.zip. In each case don't forget to unzip them.
The fix to work with recent SST files doesn't seem to have made it into SkS yet, so for the short term I've made a working copy available here. This will be removed once the SkS version is fixed.
If you continue to have problems, say so and I'll do a screencast on it. As a software author it is never possible to know if your software is useable except through feedback from people trying to use it.
Getting back to original aim of this blog entry, Christopher Booker has come back to the issue of data management (Adjusting climate data is the biggest science scandal ever http://www.telegraph.co.uk/news/earth/environment/globalwarming/11395516/The-fiddling-with-temperature-data-is-the-biggest-science-scandal-ever.html ) and further suggestions by Homewood of incorrect adjustment, this time in the Arctic. Is there a clear rebuttal of Booker's latest claims available?
Again, you can do it with the SkS tool. Use just the land data, and deselect all but two rightmost latitude band buttons (60-75 and 75-90N).
If you just use the 75-90N zone, it indeed looks as though the Arctic hasn't warmed. But if you look at the coverage graphs you'll see why: Coverage is down to a single station before 1940, and no more than 7 in recent decades. This sort of meaningless plot has been going round for a long time, see for example this graph.
With a larger set of stations you get a meaningful record.
Ideally we'd like to calculate a record for just the Arctic, but allowing stations from just outside the Arctic to contribute for locations where they are closer than any Arctic station. You can do that using the KNMI climate explorer, but that doesn't give access to the unadjusted data. Select GISS or Cowtan & Way, and then give a latitude range on the next screen. You can also compare to the ERA-interim reanalysis on the monthly indices page.
FrancisMcN @44, have you heard the old fable about the boy who cried wolf?
I ask because the people who have been crying wolf about the temperture record have been at it for a long time. They have been repeatedly proven wrong. They have been proven wrong in detailed cases where time after time the adjustments they claim are unwarrented have been shown to be in fact completely warrented. They have certainly been proven wrong in the overall picture, where it has been shown repeatedly that they cherry pick instances of adjustments which warm the local record (when there are almost as many on land that cool the local record); and where it has just been shown that the overall effect of all adjustments of land and SST records combined reduces the warming trend in the twentieth century:
Given this, you have to wonder why anybody still pays attention to the Bookers of this world. The reason, of course, is that "The boy who cried wolf" is an old fable. In these modern time, proving that you have repeatedly misinformed the public just gets you repeat columns in the Murdoch press. Some people, and some media are more intent on the political effect of what they say than the truthfulness (and in this case it is very clear cut who they are).
Tom Curtis at 46 - no wolf seen here just a desire to have the most authoritative and directly applicable rebuttal to the outpourings of a tiresome character who unfortunately seems to get a lot of attention in my part of rural Devon (the 21K+ comments to the on-line article also suggest a fair following!).
Kevin's speedy response was very welcome as was his original item and I will do my best to delve into the data as he advises but I fear that is not going to be a easily deliverable way of convincing the sort of Telegraph reader that takes Booker's items at face value. I was wondering if there are some good examples of data adjustment where the same principles have resulted in reduced temperature in recent years that could be quoted to show that the adjustment cuts both ways? Perhaps I'll just have to start looking but I would be grateful for any suggestions!
I just had a brief look at Homewood's blogpost on which the Booker article is based. Homewood claims to find 19 out of 23 Arctic records adjusted. It turns out, however, that he only looked at north Atlantic and northern European stations (51o W to 87o E, or from the West Coast of Greenland to north of the Urals). We learn this from a revision to his post brought about by Nick Stokes noting three stations that showed cooling. What concerns me is that in the area that Homewood definitely claims to have analyzed, only two (Vestmannayja and Reykjavik) show a strong warming adjustment. The remaining 17 of the 19 adjusted stations show weak warming adjustments at best, and in Trommaskato, Marlye Karmaku, and Turuhansk show cooling adjustments. Others may also show cooling adjustments as I have not gone over them in detail. The two stations showing a strong warming adjustment are both in Iceland. However, Akuyeri and Stykkisholmur, the other to Icelandic stations, show virtually zero, and only a slight change in trend from adjustments respectively. It follows that the two Icelandic stations with large adjustments, whose temperature histories do not agree with each other, nor with the two Icelandic stations in close agreement, did require adjustment.
In all, Homewood has concealed in his discussion the fact that several of the adjustments to stations of which he talks resulted in a reduced warming trend; that the only two requiring large adjustments were discordant with each other and the other two stations on the same small island, and that overall the adjustments in less than half of the Arctic has only a small effect on the overall trend in that region.
I ask again, why are we paying attention to the boys who cry wolf?
[JH] Like it or not, the buzz created by Brooker's Feb 7 Telegraph propaganda piece, The fiddling with temperature data is the biggest science scandal ever, is starting to gain attention in the MSM. Personally, I believe this particular propaganda piece is designed to divert attention from, and cast aspersions on, the ongoing UN climate talks in Geneva.
FrancisMcM @47, I am not charging you of crying wolf. Rather, that is a charge that I am directing at Booker and Homewood. With regard to stations that show cooling adjustment, the three found by Nick Stokes and the three I found among Homewood's lists of (purportedly from context, warming) adjustments gives you a start. If you want to look further, Steve Mosher from the BEST team cites the entire continent of Africa, which has a warmer trend in the undadjusted than in the adjusted data (although various stations within Africa will differ). Then there are the SST which show a very strong cooling adjustment, sufficient that overall the global surface temperature record trend is reduced by adjustments, not increased.
None of that, however, will persuad those who are not persuaded by the simple fact that the adjustment algorithms, which are applied automatically, are blind to the direction of adjustment or time in the century. That is, they have no inherent bias so that any bias in the adjusted data is entirely a product of the data itself.
[JH] It would be helpful to our readers if you were to provide a definition of "algorthim" and reference some background materials about it. Thanks.
The case presented by the Homewood blog really rests on 19 station data plots from NOAA. The whole set of such plots are available from this directory although to be useful the station ID numbers are needed. NASA have a good tool for finding those.
I'm sure FrancisMcM @47 will find some stations with the trend in warming reduced rather than increased by adjustment. The first Arctic station I picked Barrow in Northern Alaska showed reduced warming following adjustment.
Of course, it is difficult to imagine that this Homewood character is doing anything more than cherry-picking stations. After all UAH show a similar warming to NASA GISS in the Arctic (graphic here (usually two clicks to 'download your attachment' shows Arctic GISS annual & UAH TLT winter & summer temperature record) and that is obtained without surface station adjustments. Then perhaps Homewood & Booker will be next turn their attentions to the satellite adjustements of Roy Spencer.