Richard Tol accidentally confirms the 97% global warming consensus

“There is no doubt in my mind that the literature on climate change overwhelmingly supports the hypothesis that climate change is caused by humans. I have very little reason to doubt that the consensus is indeed correct.”

These are the words of economist and Global Warming Policy Foundation advisor Richard Tol in a new paper published in Energy Policy. Despite accepting that the expert consensus on human-caused global warming is real and correct, Tol has nevertheless spent the past year trying to critique the study the Skeptical Science team published last year, finding a 97% consensus in the peer-reviewed climate literature.

The crux of Tol's paper is that he would have conducted a survey of the climate literature in a slightly different way than our approach. He's certainly welcome to do just that – as soon as we published our paper, we also launched a webpage to make it as easy as possible for anyone to read the same scientific abstracts that we looked at and test the consensus for themselves.

Tol chose instead to look for faults in our study's methods in what he described as a "destructive" approach. Ultimately he concluded that because those who were categorizing the abstracts based on their position on the cause of global warming were human, our ratings were imperfect (this is certainly true), and that accounting for these imperfections brings the consensus value down to about 91%. That's where Tol made his big mistake.

Tol's big mistake

To minimize our uncertainties, we had at least two people categorize each scientific abstract. Where those two raters disagreed, we had a reconciliation process. The disagreeing raters first checked their ratings again; if the disagreement persisted, a third person acted as the tiebreaker to establish the final rating.

Using the difference between our initial and final ratings, it's possible to estimate the number of papers that still remain in the improper categories after our reconciliation process. Tol put the estimate at about 6.7% of the total, and noted that 55% of our reconciliations from initial to final ratings were 'towards stronger rejection', while 45% were 'towards stronger endorsement' of human-caused global warming.

Tol then made a basic and critical error. His methodology resulted in assumptions that, for example, 55% of the remaining incorrectly rated 'no position' category papers should actually be rejections, while 45% should be endorsements. He didn't check to see how the reconciliations changed the initial and final ratings for each category, and this assumption led him to incorrectly conclude the consensus is actually 91%. Still a high percentage, but nonetheless in error.

In reality, as our response to Tol's critique (accepted by Energy Policy but not yet published) shows, there simply aren't very many peer-reviewed papers that minimize or reject human-caused global warming. Most of the papers that were reconciled 'towards stronger rejection' went from explicit to implicit endorsement, or from implicit endorsement to no position. For abstracts initially rated as 'no position,' 98% of the changes were to endorsement categories; only 2% were changed to rejections.

That makes sense when you think about it, because less than 3% of all climate papers reject or minimize human-caused global warming. There's no reason to expect 55% of incorrectly rated 'no position' papers to reject the consensus – in reality there just aren't that many rejection papers. I asked Tol about this point, and he responded,

"I indeed use the marginal distribution of error corrections rather than the conditional ones. There are not enough observations to estimate all the conditionals."

However, the 'no position' category was the largest in our sample, and was the main source of Tol's mistaken calculation, as shown in the figure below. The top frame illustrates the result when calculation is done correctly using the actual reconciliations for each category, while the bottom frame shows the result using Tol's faulty assumptions.

An anonymous individual has also published an elegant analysis showing that Tol's method will decrease the consensus no matter what data are put into it. In other words, his 91% consensus result is an artifact of his flawed methodology.

Pulled from thin air

As the above figure illustrates, by making this mistake, Tol effectively conjured approximately 300 papers rejecting or minimizing human-caused global warming out of thin air, with no evidence that those papers exist in reality. As a result, his consensus estimate falls apart under cursory examination. Ironically, when discussing our study in a US congressional hearing last week, Tol claimed,

"...as far as I can see, this estimate just crumbles when you touch it ... this 97% is essentially pulled from thin air."

When our team corrected for Tol's error, accounting for the ways in which the reconciliation process actually changed the ratings for each category, we found a slight increase in the consensus, from an initial 97.1% to a corrected 97.2%. Accounting for the uncertainties involved, we ultimately found the consensus is robust at 97 ± 1%.

It's also important to remember that our finding of 97 ± 1% consensus in climate research abstracts is consistent with the scientist author paper self-ratings in our study (97.2%), Doran & Zimmerman's 2009 study (97%), the Anderegg et al. 2010 study (97.5%), Oreskes' 2004 study (zero abstracts rejecting human-caused global warming in a sample of 928 papers), 80 National Academies of Science and dozens of scientific organizations from around the world endorsing the consensus (none rejecting it). And of course Richard Tol acknowledges it,

"Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role."

and

"The consensus is of course in the high nineties"

This is an important point, because a new George Mason University report finds that only 12% of Americans are aware that more than 90% of climate scientists have concluded humans are causing global warming. Ironically, even Tol's flawed critique finds the consensus is higher than 88% of Americans believe.

Another Gremlin infestation?

Tol nevertheless tried to find fault in our approach, and as a result submitted a flawed paper. In addition to making several basic errors, Tol cited numerous denialist and GWPF blog posts, including several about material stolen from our team's private discussion forum during a hacking. Similar to the Climategate incident, Tol even took quotes out of context from the hacked discussions to try and invent evidence of 'rater fatigue' among our team where no such evidence exists (we rated papers at our own leisure without any set deadlines).

The flaws in his critique are reminiscent of Tol's recent corrections to errors in his climate economics research, where he explained, "Gremlins intervened in the preparation of my paper." Perhaps the Gremlins struck again.

One might wonder how Tol's critique made it through the peer-review process with so many serious flaws. It took five tries, as the paper was first rejected four times by three other journals. He received some harsh but fair criticism from the Environmental Research Letters reviewers, who listed 24 problems and ways the paper could be improved. When I asked Tol about these critiques, he told me, "I incorporated all comments by ERL that hold water."

However, a side-by-side comparison reveals that Tol's Energy Policy paper still contains nearly all of the shortcomings identified by the Environmental Research Letters reviewers, plus some new ones. Our team's critique of Tol's paper identified several of the same problems as the Environmental Research Letters reviewers and many more – some they didn't catch and some that Tol added to the Energy Policy version – again, 24 in total.

In any case, Tol's critique explicitly acknowledges the expert consensus on human-caused global warming is real and accurate. Correcting his math error reveals that the consensus is robust at 97 ± 1%, consistent with the results of previous consensus studies.

While the consensus may be an inconvenient truth for those who seek to obstruct and delay the implementation of global warming solutions, it's nevertheless an indisputable reality; one we'd be better off if people learned to accept.