Arguments
Resources and links documenting Tol's 24 errors
Posted on 6 June 2014 by John Cook
Yesterday, we published a list of 24 errors in Tol's critique of our consensus paper Quantifying the consensus on anthropogenic global warming in the scientific literature. The short URL for our 24 errors report, handy for tweeting and posting in comments, is:
In addition, the Global Change Institute at The University of Queensland issued a statement summarising our response to Tol (2014) including a scholarly version of the 24-errors report (e.g., the same content but without all the bright, shiny boxes).
Today, a link to our reply to Tol made it onto the homepage of Reddit Science, causing our website traffic to surge to 20 times its normal level (so apologies for the sluggish server performance earlier today).
One of the eyebrow raising elements to Tol (2014) is that his analysis still finds an overwhelming consensus on human-caused global warming. This is significant given a recent George Mason University survey found only 12% of Americans know more than 90% of climate scientists agree on human-caused global warming.
Consequently, it's worth reminding people of Tol's views on consensus, expressed in Tol (2014). I've added a freely shareable graphic to our resource of consensus graphics, featuring an excerpt from Tol (2014). Everyone is encouraged to retweet or republish the graphic which is freely available under creative commons:
Other comments on Tol (2014)
There have been a number of articles published in response to Tol (2014) that offer some interesting perspectives on the various critical errors in Tol's paper.
Greg Laden at Science Blogs characterises Tol's paper concisely:
Tol is practicing a special kind of science denialism here, sometimes called “seeding doubt” or as I prefer it, “casting seeds of doubt on infertile ground.” In other contexts this is called “concern trolling” or the “You’re not helping” gambit.
Following this line of thought, Econnexus wonders about Tol's apparent concern that our consensus paper is polarizing:
What's the point of "further polariz[ing] the climate debate" by publicly picking holes in the methodology of a paper when you broadly agree with its conclusions and you yourself imply that you are concerned about said polarization, particularly when your critique has itself been criticised by a number of independent referees?
Small Epiphanies by Graham Wayne wonders how a paper that purports to be irrelevant to energy policy was accepted by a journal about energy policy:
“Energy Policy is an international peer-reviewed journal addressing the policy implications of energy supply and use from their economic, social, planning and environmental aspects…” insists the journal. Why would they publish a paper so clearly outside their expertise? More curious is that before Energy Policy agreed to publish it, ERL and several other journals rejected previous drafts. The reasons for the rejection were as copious as they were blunt.
And Then There's Physics slow claps Richard Tol's destructive comment:
The Consensus Project was, essentially, a citizen science project. I know some of those involved work at universities, but most were simply volunteers who helped to rate abstracts so as to establish what fraction of those that took a position with respect to AGW, endorsed AGW. The resulting paper was one of the papers of 2013 that received the most online attention. In any other field this would be seen as a remarkable success. A citizen science project publishes a paper that has more impact than almost any other paper published that year. Not in climate science, though. In climate science you have to attack and attempt to destroy anything that goes against the narrative that you’d like to control. Personally, I find it depressing that this is the manner in which some choose to conduct themselves and – in some sense – this simply seems like another illustration of how poor the general dialogue is in the climate science debate.
Critical Angle by Andy Skuce discusses how Andy's own private correspondence was earlier stolen by an unknown hacker and subsequently his own words were quoted out-of-context by Richard Tol:
Putting aside, for now, the shabby ethics of citing stolen material, this interpretation of what I said is false. The quote was part of a discussion that a couple of us working on the project were having about our impressions that some of the abstracts in the sample were repeated. We were not sure if those impressions were real or whether they arose from having read hundreds of abstracts over several days and not remembering each one exactly. We had no way of checking the database for ourselves and we alerted John Cook so that he could investigate further. The notion that I was concerned about the effect of “fatigue ” on the quality of my work is quite false and this is obvious to anyone who reads the quote in context.
I will be asking the editor of Energy Policy to correct this error and to remove any references to stolen private discussions in Professor Tol’s paper. It is surprising and disappointing that the reviewers and editorial staff of the journal did not spot this.
Real Sceptic examines Tol's suggestions of rater fatigue and how the actual expert Tol cites says the opposite to Tol's claims:
...this whole point about fatigue is nonsense. What Tol is referring to is survey fatigue, the tendency of people to quit or get less accurate when they fill in long surveys or a lot of surveys. But this was a team of raters who were free to rate abstracts at their leisure, they could start whenever they wanted, continue as fast or as slow as they wanted, and could take a break when they wanted. There wasn’t a deadline for submission or for finishing the ratings. This is a similar method as used by Oreskes 2004 which Tol refers to as one of the “excellent surveys of the relevant literature.”You also expect raters to become more proficient in these type of situations. The author of the book that Tol cited to make his fatigue point confirmed that this is the case for this set up. If this wasn’t the case there wouldn’t be a 97% consensus from the abstract ratings and then also a 97% consensus from the authors rating their own papers.
Hot Whopper attempts to do a better job than us at explaining Tol's error in conjuring 300 non-existent rejection abstracts out of thin air:
The key point is that where there were differences between the raters and a third person had the final say, most papers that were changed from one category to another went from:
- explicit endorsement of AGW to implicit endorsement of AGW or
- implicit endorsement of AGW to no position on AGW or vice versa.
Contrary to what Richard postulated, there was not a huge shift from AGW endorsement or "no position" to rejection of AGW.
SkepticalScience has a couple of charts illustrating this in detail with numbers. Here's a simplified illustration as an animation to show what Richard's blunder resulted in.In other words, where there was a shift in the category of an abstract, it just shifted slightly in the categorisation (mainly from explicit to implicit endorsement, or implicit to no position or vice versa) - and if anything the researchers tended to err on the conservative side if there was a marginal decision. There were hardly any abstracts that were reclassified from "no position" to rejection let alone from endorsement to rejection. Dana's diagram shows that out of 595 reclassified abstracts, only five papers would have gone from "no position" to "rejection of AGW" and only one paper from endorse to reject.
Rabett Run reports a paper by an Anonymous Scientist (why anonymous? Google "Tol Ackerman") that cuts to the heart of one of the critical flaws in Tol (2014):
...the assumption that the remaining differences are equally distributed among categories is more than problematic. It is WRONG and the data showing that it is WRONG have been made public.
I think we'll hear a lot more about this paper by the Anonymous Scientist. What the paper finds is that even if you reverse the changes we made during our reconciliation process, Tol's method still decreases the consensus. Tol has created the perfect climate denial machine: it reduces the consensus even if you reverse the data fed into it:
Most of the reduction in the consensus percentage is not due to the trend in shifts applied during reconciliation, but rather an artifact of the method itself.
The Global Change Institute at The University of Queensland issued a statement on our response to the Tol paper:
The University is committed to the publication of thoroughly researched, internationally peer-reviewed academic articles, without fear or favour resulting from that publication. As a result, the co-authors of the paper in Environmental Research Letters have examined the claims in the latest article published in Energy Policy and have identified 24 issues which can be viewed in an online response here.
These errors invalidate the criticisms of the Energy Policy article and lead to a strengthening of the original conclusions in the Cook et al. publication.
The University is committed to the publication of thoroughly researched, internationally peer-reviewed academic articles, without fear or favour resulting from that publication. As a result, the co-authors of the paper in Environmental Research Letters have examined the claims in the latest article published in Energy Policy and have identified 24 issues which can be viewed in an online response here.
These errors invalidate the criticisms of the Energy Policy article and lead to a strengthening of the original conclusions in the Cook et al. publication.
- See more at: http://www.gci.uq.edu.au/news/uq-researcher-responds-to-criticism-of-research#sthash.HVJdHziM.dpufLastly, I think the final word should go to Environmental Research Letters (ERL) whose expert reviewers rejected Tol's paper around a year ago. You can read the full list of 24 errors identified by the expert reviewers at ERL. Those errors remain in the paper accepted by Energy Policy. Here is a quote from the ERL review:
This section is not supported by the data presented and is also not professional and appropriate for a peer-reviewed publication. Furthermore, aspersions of secrecy and holding data back seem largely unjustified, as a quick google search reveals that much of the data is available online (https://skepticalscience.com/tcp.php?t=home), including interactive ways to replicate their research. This is far more open and transparent than the vast majority of scientific papers published. In fact, given how much of the paper’s findings were replicated and checked in the analyses here, I would say the author has no grounds to cast aspersions of data-hiding and secrecy.
So when can we expect an Auditor to write up their scathing audity of Tol 2014? Especially since the method has a systematic bias which produces the same result even if the data is fed in the opposite way, since that allegation seems to be one of the Auditor's favorite bones to pick.
Someone mentioned on the previous thread that Dr Tol considered that admitting an error in his criticism of Cook et al 2013 would leave his reputation in tatters (or equivalent).
As far as I can see, Dr Tol's perseverance in his criticisms, despite their many flaws, his ethically-dubious (if not outright unethical) use of hacked correspondence, and his apparent confabulation on the subject of rater fatigue, are altogether doing an excellent job of destroying his reputation in a way that admitting error could not.
In fact, admitting that his compulsive attack on Cook et al 2013 is a great error and retracting it would go some way to restoring his reputation.
If Tol accepts the reality, reliability and reasonableness of the consensus (i.e. as in the quote on p.3 of the '24 Errors' PDF), can someone please tell me what he was trying to prove by his critique of C13? It seems to me that, in attempting to criticise the methods - and cast aspertions on the motives - of C13, he has shot himself in both feet.
Martin Lack @3, Tol's considered opinion on AGW appears to be:
1) It is demonstrably real and happening now;
2) The risks of future climate change as a result of AGW are sufficient to warrant a price on Carbon; and
3) In order to impliment (2), he makes common political cause with anybody opposed to any action on AGW (including carbon prices).
Given the irrationality of (3) given (1) and (2), you may be tempted to question it. In evidence, I give the fact that he is part of the Copenhagen consensus (Bjorn Lomborg), an adviser for the Global Warming Policy Foundation, now a go to guy for testimony in Congress for climate denying Republicans, and Tol (2014).
[snip] But he's a generally smart person, so he knows the consensus is real and accurate, that the economics of the situation demands a carbon price, etc.
However, I think he saw all the attention Cook at al. (2013) received, and he wanted a piece. I think it's as simple as that.
[KC] ad hominem snipped
"Tolgate" may be an appropriate label for Tol's vendetta against John Cook and the team of SkS volunteers who expended a lot of blood, sweat, and tears in the production of Cook et al (2013).
Dana, I read Tol differently. His vendetta against Cook et al (2013) began by his making intemperate comments based on a blog post by poptech, and then digging his heals in rather than admit error. However, his initial acceptance of poptech's blog post is not explained by that, nor by a desire for publicity. Nor is his long term cooperation with Lomborg, nor his association with the GWPF, nor his absurd recent comments about the IPCC, nor the consistent bias from his various "gremlins" towards findings that require less action on AGW. I do not disagree that he is a glory hound, but that alone is inadequate to explain his actions.
Tom Curtis - Tol started complaining about Cook et al the day it was published, for example tweeting here and especially here on May 18 2013 - stating "@ezraklein for starters, because that opening 97% is a load of nonsense @maliniw90th". That was well before poptech made his blog post with cherrypicked objections from authors who hadn't responded to the Cook et al queries regarding self-evaluation, and who didn't seem to understand the difference between papers and abstracts.
Tol has spent the intervening time searching for a reason, _any_ reason, to support his initial reaction. And whenever one set of objections were shown to be nonsense, moving onto another and another and...
He's done a terrible job of it.
My personal opinion (just that) regarding his vendetta is an ideological objection on his part to governmental approaches to dealing with AGW, one not based on the science, coupled with an (ahem) abrasive approach to those he disagrees with.
Good information. For what it's worth, I've reposted this along with some comments. :- )
http://whatsupwiththatwatts.blogspot.com/2014/06/richard-tols-big-mistake-or-malicious.html
Towards the end of the post, the link to the "full list of 24 errors identified by the experts reviewers at ERL" also provides links to the Tol's paper as sent to ERL and a later draft. Although there will be changes in the paper evemtually published in Energy Policy, it is very helpful to understanding the nature of Tol (2014). I was not happy discussing a paper in so much detail when, bar the odd quote, it is entirely hidden behind a paywall.
Now seeing the nature of the beast, I find it incredible. From its incompatible title to its strange list of Acknowledgements (a list that includes Dana Nuccitelli, "Willard" and in the draft version "wottsupwiththatblog.") I have to say, this is not a publishable paper.
As far as the 24 errors are concerned, Tol may take an embattled win on one or more of them and a few would end up as score draws, but the majority of the criticisms of Tol (2014) seem well founded and also include fundamental problems for Tol (2014).
But the most fundamental question for Tol is "What is Tol (2014) trying to say"?
Tol (2014) is not a "re-analysis" because that would present an alternative to the 97% as its main finding. Ditto if this paper was mainly concerned with identifying errors within Cook et al (2013). If the thrust of the paper were the inadequacy of method, it would then be less bothered with the specific result - but a whole whole lot more bothered that Tol (2014) demonstrates on how to achieve an adequate method. Tol's paper is polemical but for no justifiable academic reason, and this is magnified by the content of the Abstract.
But why end there. Tol goes well beyond the substance of his paper when he tells us that, although there is "very little reason to doubt that the consensus is indeed correct," for Cook et al (2013) "theirs is not a consensus on the causes of climate change." And anyway "consensus is irrelevant in science," and "has no academic value," because it "serves a political purpose, rather than a scientific one," at least in this case. Yet in this case, this particular consensus, "it is well-known" already and "it does not matter whether the exact number is 90% or 99.9%." And further, the alleged errors within Cook et al (2013) "may strengthen the belief that all is not well in climate research." So don't do this consensus enumerating stuff. Stick with the reviews of the literature itself. They already establish where science has got to on climate change."The IPCC fulfils this role."
That, I fear, is all argument for argument's sake. It is not useful to man nor beast. But I do see in it the sort of argument that somebody in denial would likely come out with.
MA Rodgers is spot on! Within the world of academia, Tol's draft paper has verey little value and does not merit being published in any reputable scientific journal.
As evidenced by his responses to valid criticisms of his paper, Tol does not appear to care about how his paper fares in academia.
What then is the value of Tol's paper?
That question can only be answered within the context of the seemingly never-ending propaganda war being waged by the Climate Denial Spin Machine.
The value of Tol's paper is that it provides fodder for climate deiers to use in their rentlentless vendetta against Cook el (2013).
I just noticed that the "24 errors" PDF reproduces the number 17904 three times in succession in table 1, column 4, each time corresponding to a different percentage. Presumably the second to occurences are errors that you may want to correct.