Arguments
Here’s what happens when you try to replicate climate contrarian papers
Posted on 25 August 2015 by dana1981
Those who reject the 97% expert consensus on human-caused global warming often invoke Galileo as an example of when the scientific minority overturned the majority view. In reality, climate contrarians have almost nothing in common with Galileo, whose conclusions were based on empirical scientific evidence, supported by many scientific contemporaries, and persecuted by the religious-political establishment. Nevertheless, there’s a slim chance that the 2–3% minority is correct and the 97% climate consensus is wrong.
To evaluate that possibility, a new paper published in the journal of Theoretical and Applied Climatology examines a selection of contrarian climate science research and attempts to replicate their results. The idea is that accurate scientific research should be replicable, and through replication we can also identify any methodological flaws in that research. The study also seeks to answer the question, why do these contrarian papers come to a different conclusion than 97% of the climate science literature?
This new study was authored by Rasmus Benestad, myself (Dana Nuccitelli), Stephan Lewandowsky, Katharine Hayhoe, Hans Olav Hygen, Rob van Dorland, and John Cook. Benestad (who did the lion’s share of the work for this paper) created a tool using the R programming language to replicate the results and methods used in a number of frequently-referenced research papers that reject the expert consensus on human-caused global warming. In using this tool, we discovered some common themes among the contrarian research papers.
Cherry picking was the most common characteristic they shared. We found that many contrarian research papers omitted important contextual information or ignored key data that did not fit the research conclusions. For example, in the discussion of a 2011 paper by Humlum et al. in our supplementary material, we note,
The core of the analysis carried out by [Humlum et al.] involved wavelet-based curve-fitting, with a vague idea that the moon and solar cycles somehow can affect the Earth’s climate. The most severe problem with the paper, however, was that it had discarded a large fraction of data for the Holocene which did not fit their claims.
When we tried to reproduce their model of the lunar and solar influence on the climate, we found that the model only accurately reproduced their temperature data reasonably well for the 4,000-year period they considered. However, for the 6,000 years’ worth of earlier data they threw out, their model couldn’t reproduce the temperature changes. The authors argued that their model could be used to forecast future climate changes, but there’s no reason to trust a model forecast if it can’t accurately reproduce the past.
We found that the ‘curve fitting’ approach also used in the Humlum paper is another common theme in contrarian climate research. ‘Curve fitting’ describes taking several different variables, usually with regular cycles, and stretching them out until the combination fits a given curve (in this case, temperature data). It’s a practice I discuss in my book, about which mathematician John von Neumann once said,
With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.
Good modeling will constrain the possible values of the parameters being used so that they reflect physical realities, but bad ‘curve fitting’ doesn’t limit itself to physical realities. For example, we discuss research by Nicola Scafetta and Craig Loehle, who often publish papers trying to blame global warming on the orbital cycles of Jupiter and Saturn.
This particular argument also displays a clear lack of plausible physics, which was another common theme we identified among contrarian climate research. In another example, Ferenc Miskolczi argued in 2007 and 2010 papers that the greenhouse effect has become saturated, but as I also discuss in my book, the‘saturated greenhouse effect’ myth was debunked in the early 20th century. As we note in the supplementary material to our paper, Miskolczi left out some important known physics in order to revive this century-old myth.
This represents just a small sampling of the contrarian studies and flawed methodologies that we identified in our paper; we examined 38 papers in all. As we note, the same replication approach could be applied to papers that are consistent with the expert consensus on human-caused global warming, and undoubtedly some methodological errors would be uncovered. However, these types of flaws were the norm, not the exception, among the contrarian papers that we examined. As lead author Rasmus Benestad wrote,
we specifically chose a targeted selection to find out why they got different answers, and the easiest way to do so was to select the most visible contrarian papers ... Our hypothesis was that the chosen contrarian paper was valid, and our approach was to try to falsify this hypothesis by repeating the work with a critical eye.
If we could find flaws or weaknesses, then we would be able to explain why the results were different from the mainstream. Otherwise, the differences would be a result of genuine uncertainty.
After all this, the conclusions were surprisingly unsurprising in my mind. The replication revealed a wide range of types of errors, shortcomings, and flaws involving both statistics and physics.
You may have noticed another characteristic of contrarian climate research – there is no cohesive, consistent alternative theory to human-caused global warming. Some blame global warming on the sun, others on orbital cycles of other planets, others on ocean cycles, and so on. There is a 97% expert consensus on a cohesive theory that’s overwhelmingly supported by the scientific evidence, but the 2–3% of papers that reject that consensus are all over the map, even contradicting each other. The one thing they seem to have in common is methodological flaws like cherry picking, curve fitting, ignoring inconvenient data, and disregarding known physics.
Not cherry picking, just nit picking:
"The one thing they seem to have in common is methodological flaws like cherry picking, curve fitting, ignoring inconvenient data, and disregarding known physics."
Are not "cherry picking" and "ignoring inconvenient data" pretty much the same thing?
I am grateful for this research.
There is a thorough discussion on this paper at Real Climate, moderated by Rasmus Benestad. It includes contributes to the discussion from Ross McKittick and Richard Tol, who both had opportunities to review the paper in an earlier form and who both advised against publication at that time.
Let's Learn from Mistakes
The fact that the 3% "skeptics" don't agree with each other is a very strong point. I would like to see some statistics on that : what % of skeptic papers claim "there is no warming", "climate sensitivity is low", "everything is uncertain" etc ? And how many climate skeptics contradict themselves ?
bvangerven - As I see it, the 97% consensus is pretty clear on anthropogenic GHgs, increased radiative effect, attributions, etc. While the remaining 3% couldn't agree that the sky is blue....
Dana,
I love this piece. I love this web site. You and your comrades do excellent and necessary work but please be careful with history.
"...Galileo, whose conclusions were based on empirical scientific evidence, supported by many scientific contemporaries..."
I’m sorry Dana but that is just not true but it is a widely held belief of those who are not aware of the actual history. Please don’t be one of those.
Sci Am had a great article last year, in the January issue I think, called “The Case Against Copernicus” that discusses the lack of empirical evidence during the time of Copernicus and Galileo. The many scientific contemporaries you mention did not exist. You could also visit a wonderful site called Renaissance Mathematicus. Thony Christie is a superlative historian of science of the Early Modern Period and he has a special section called The Transition To Heliocentricity: The Rough Guides because so many folks keep getting the actual history wrong.
If you get the history wrong when trying to correct the mistakes others are making with the history your argument loses its justified force. This is a friendly suggestion to check your history before publishing.
https://en.wikipedia.org/wiki/Galileo_Galilei; he stept on some very powerfull toes (and his theory was flawed)
M Tucker
I think Dana's statement that Galileo's conclusions were based on empirical evidence is defensible. After all, he was actually observing the Moon and planets through a telescope.
However, I too would cast doubt on the statement that Galileo was supported by "many scientific contemporaries". They were just not many scientists around! As far as I know, Jesuit astonomers like Father Clavius, who was respected by Galileo, were willing to compromise on Galileo's theory by treating it as a method of calculation of orbits rather than physically true. However, Galileo was just not the compromising type.
I've read the account by Rasmus Bernestad in RealClimate and am intrigued by the journal shopping that took place in efforts to find a journal that would accept this paper. Perhaps unsurprisingly. the journal that finally accepted the paper had the lowest Impact Facor (1.759) of the five journals approached. The title of the paper "Learnng from mistakes in Climate Science" is, again perhaps, as applicable to the efforts of the authors in getting the paper accepted as it is to the papers they criticise in their paper. Quite honestly the efforts seem typical of academics operating under the threat of "Publish or Perish". That said congratulations to Dr Bernestad for presenting the saga so frankly. I would however ask Dr Bernestad not to use "fairly unique". Something is unique or it is not. There are no qualifiers
It is interesting that the factor making the paper difficult to get published was that it looked at multiple previous studies. Papers showing an inability to replicate the results of just one study get published all the time... almost routinely. Had it been split into 38 separate papers, each showing the inability to replicate a single previous study, it seems unlikely there would have been any controversy.
Thus, rejecting the 'composite' study because it didn't seek to perform a statistical 'meta analysis' of 'skeptic' vs mainstream replication rates or some other 'collective result' seems to be missing the point. They rejected it for failure to be what it wasn't... while overlooking that it was perfectly sound and replicable science for the issue it was actually looking at.
The other objection, that the paper clearly had a 'political' point in looking solely at 'skeptic' papers is true... but should have been irrelevant. Rejecting sound science because it has a political intent is, itself, a political decision.
When you look closely at the surveys of the most qualified climate scientists — those actively doing research and publishing their results in peer reviewed journals — you find 97 to 97.5% who believe that the climate is warming and that man is a major cause and only 1% who reject this finding. The other 1.5 to 2% are the fence sitters — the undecided.
Looking at the "many scientific contemporaries" this conjectures that it would be less than a dozen. However, one of them was Kepler. I dont think any contemporary would-be-Galileo has someone in the same order supporting them.
M Tucker at @5 and Ger@6
I think you will find that at the turn of 16th and 17th century when Galileo lived, that there was little scientific evidence for any theory. Science at that time was more philosophical, based on the thinking of Aristotle. This way of thinking came from the Greeks who preferred to philosophise. They used to just sit around and think about things rather than make actual observations and do experiments to collect data that supported their theories. The prevailing scientific method at the cusp of the 16th and 17th century was to only propose theories that reinforced the Bible. The vast majority of the "scientists" at the time studied in monasteries or were alchemists trying to convert lead into gold. Copernicus was a monk and proposed his heliocentric theory as an alternative to the Ptolemic biblical geocentric theory only as an interesting thought exercise which he saw as a much simpler explanation. He supported his heliocentric ideas with geometry but was too afraid to publish until he was on his death bed. It is no coincident that the spread of the Copernican theory to scientists like Galileo was only possible due to the recent invention of printing. This allowed ideas and findings to be spread more widely. Galileo at least made observations and collected evidence. His work on falling bodies was certainly unique since the prevailing Aristolitean scientific view at the time was that heavier objects fell faster that lighter objects. And we already know that through his observations of Jupiter and the Moon through his recently developed telescope, he gathered evidence of the correctness of the Copernican theory. This correctness was further refined by Kepler, also a deeply religious man, through proposing that the planets moved in elliptical orbits, a position that took him 20 years to come to using the highly accurate data collected by Tycho Brahe, who incidently still believed in a biblical geocentric theory. The whole issue, heliocentric versus geocentric, was finally put beyond all doubt later in the 17th century by Newton, incidentally another deeply religious man, with his Theory of Gravity and the invention of Calculus using the co-ordinate geometry that had recently been proposed by Descarte. Galileo was one of the first scientists who used a modern scientific approach (hypothesise, propose a theory, create experiments, observe, collect data, and test) to justify his ideas. This is quite unlike the many so called "scientists" at the time who were more philosophers who conducted few expirements, collected little supporting data and preferred to use the bible as evidence to justify for their ideas. Galileo was a modern scientific thinker unlike most of his contemporaries. The Galilean story just shows that all scientific ideas are the meticulous work of many scientists pursuing a common truth.
Theories, where the observations and evidence are contry, are thrown out. Theories, where observations and evidence are supporting, are further refined and become increasingly accepted as scientific truth until there is only one scientifically indisputable piece of evidence that can falsify the theory. In climate science, the contrarians have not found a single piece of evidence that disproves the fundamental scientific idea that rising greenhouse gases will warm the planet. Also, they have not proposed a single coherent alternative scientific idea, nor provided a single piece of indisputable evidence that explains why the current warming is happening. All they have provided are some interesting distracting talking points, which, so far, have only served to further reinforce the idea that AGW and CC is actually happening.
Just asking. What are the views of the primary scientists who actually collect the data from the primary sources (i.e those on the ice flows gathering the ice core data, those collecting the glacial retreat data, those gathering the sediment data, those collecting the carbon dioxide data, those actually creating and verifying the climate models etc.)? Are there any contrarians amongst the primary scientists who actually collect the data, or are the contrarians only found amongst the secondary scientists who use the data collected by others, in an effort to debunk the basic AGW and CC proposition?
mancan18 @12, your account of the advance of science is too simplistic. It neglects the strong emperical vein in Aristotle's philospophy, which he put into practise especially as regards biology. It also neglects the development of that empericism by Roger Bacon (considered by some the inventor of modern science). It also neglects that the early scientific revolution developed out of medieval scientific enquiries, best illustrated by the fact that Newton's research in fact mirrored that of Roger Bacon.
Worse, you demonstrate an oversimplistic idea of empericism in science. It was the geocentrist Tycho Brahe that was the great observational astronomer, not Copernicus. And his 'refutation' of Copernicism stood at least in part on emperical grounds, notably the absence of stellar paralax (predicted by Copernicus), and initially by a wrong relation with regard to the parallax of Mars. While Copernicus's theory was in some ways emperically superior, was "intrinsically no more accurate than Ptolomy's", and commonly predicted "errors of a day" in the timing of lunar eclipses, not to mention the wrong length of year. (Kuhn, The Copernican Revolution p 288)
Likewise Galileo's theories faced emperical falsification from the start, predicting only one tide per day (for instance).
The fact is the transition from scholasticism to science was a messy affair. We cannot deprive Galileo's contemporarys of the mantle of scientist just because they were geocentrists, or employed some scholastic arguments unless we wish also to exclude Copernicus (who argued for his theory based on the "perfection" of the circle") and Newton (who was also an alchemist).
Tom Curtis @13
I am well aware of the works and the scientific approach of Roger Bacon and also that Aristotle was not just some simple philosopher who didn't study any science using empirical evidence. I am also aware that Greek philosophers like Erastothenes did make observations and used them in empirical ways, like for instance, estimating the circumference of the Earth. Yes, my statements were simplistic, but that was because I did not want give a full account of the development of modern empirical science. As for Newton, I am also aware that he spent more time on religious works and alchemy than he did on the scientific and mathematical studies that caused the revolution in the science that now underpins the modern world. The development of modern science is not a black and white affair where suddenly alchemists become chemists, or astrologers become astronomers. My main point is that at the cusp of the 17th century the study of science was done mostly by deeply religious men and was more used to verifying what was in the Bible, than about what the real world was actually like. It is true that if it weren't for the observations of Brahe, Kepler would not have been able to refine the Copernican theory. In fact, Brahe was so much a believer in the Ptolemic view of the motion of the planets because of his religious beliefs and his faith in the teachings of Aristotle, that he developed his own geocentric theory to account for the anomalies of planetary motion rather than simply accepting the Copernican view. As for Kepler, he spent most of his life trying to fit the planets into a geocentric model where the orbits fitted the platonic solids. He abandon his ideas because the actual observations, Tycho Brahe's observations, of the real world did not fit his theory. It was only then that he introduced ellitical orbits into the Copernican model. This was truly a new way of thinking for the time, as he had to abandon deeply held beliefs that he had gained through his religious faith. The vast majority of scientists of this time were highly religious and their scientific efforts were to enhance religious belief and glorify the church. This period is truly unique in that at the beginning of the 17th century, science was done mostly by religious men and used more to verify the prevailing religious orthodoxy. By the end of the 17th century, science was more focussed on the real world and what was actually being observed, i.e. done for its own sake. The Galileo story is just a part of that narrative, and yes he did have contray views to what was generally accepted at the time. However, most of what was accepted was because it verified the religious teachings of the church. Galileo's assertions were not based on religious orthodoxy, although it was obvious from his treatment that there were some religious figures who were sympathetic to his views.
@12,
mancan: well written except you forgot one thing in the scientific method.
Step 7- write the dang report!
~X^o'///,<
mancan18,
My observations of recent history, following the attempts of many people to develop improved understanding and raise awareness about the unacceptability of developed profitable and popular economic activity, indicate that a major barrier to 'the development of legitimate scientific development of the best understanding of what is going on, the general acceptance of that increased understanding, and the application of the understanding to develop lasting improvements of living conditions for all of humanity on this amazing planet' is a focus on "Scientific development that will lead to profitable and popular pursuits, including the science of marketing to promote those pursuits".
The pursuers of profit and popularity clearly have little reason to investigate and educate the population about the potential unacceptability of their pursuits. They may investigate them, but they choose what to promote about their pursuits. And since it is always cheaper if you can get away with a less acceptable action the chase after popularity and profitability naturally leads to a deliberate lack of awareness regarding the unacceptability of things, either because of a lack of investigation (the ones making the money are not interested in understanding the unacceptability of the ir pursuits), or a lack of sharing what has been learned (If the ones making the money become aware of an unacceptablilty related to their pursuits they will not want others to know about that unacceptability).
And people who pursue personal perceptions of prosperity can have little interest in hearing about the unacceptability of the cheaper ways they have been able to get away with believing they are prosperous. Cheaper ways almost always are more damaging and are ultimately unsustainable limited opportunities that people have to fight to be the biggest beneficiaries of (to the detriment of others).
The pursuit of profit, pleasure and perceptions of prosperity today can be seen to be the equivalent of religion in the 17th Century. That focus is a barrier to the development and acceptance of increased understanding of what is going on. Popularity and profitability today have clearly been a barrier to the development of human activity toward a lasting better future for all of humanity, just as the unjustified religious beliefs were in the past (and still are in many regions, including in developed nations, on many issues today).