Arguments
Skeptical Science Study Finds 97% Consensus on Human-Caused Global Warming in the Peer-Reviewed Literature
Posted on 16 May 2013 by dana1981, John Cook
A new survey of over 12,000 peer-reviewed climate science papers by our citizen science team at Skeptical Science has found a 97% consensus among papers taking a position on the cause of global warming in the peer-reviewed literature that humans are responsible.
Lead author John Cook created a short video abstract summarizing the study:
The Abstracts Survey
The first step of our approach involved expanding the original survey of the peer-reviewed scientific literature in Oreskes (2004). We performed a keyword search of peer-reviewed scientific journal publications (in the ISI Web of Science) for the terms 'global warming' and 'global climate change' between the years 1991 and 2011, which returned over 12,000 papers. John Cook created a web-based system that would randomly display a paper's abstract (summary). We agreed upon definitions of possible categories: explicit or implicit endorsement of human-caused global warming, no position, and implicit or explicit rejection (or minimization of the human influence).
Our approach was also similar to that taken by James Powell, as illustrated in the popular graphic below. Powell examined nearly 14,000 abstracts, searching for explicit rejections of human-caused global warming, finding only 24. We took this approach further, also looking at implicit rejections, no opinions, and implicit/explicit endorsements.
We took a conservative approach in our ratings. For example, a study which takes it for granted that global warming will continue for the foreseeable future could easily be put into the implicit endorsement category; there is no reason to expect global warming to continue indefinitely unless humans are causing it. However, unless an abstract included (either implicit or explicit) language about the cause of the warming, we categorized it as 'no position'.
Note that John Cook also initiated a spinoff from the project with a survey of climate blog participants re-rating a subset of these same abstracts. However, this spinoff is not a part of our research or conclusions.
The Team
A team of Skeptical Science volunteers proceeded to categorize the 12,000 abstracts – the most comprehensive survey of its kind to date. Each paper was rated independently at least twice, with the identity of the other co-rater not known. A dozen team members completed most of the 24,000+ ratings. There was no funding provided for this project; all the work was performed on a purely voluntary basis.
Once we finished the 24,000+ ratings, we went back and checked the abstracts where there were disagreements. If the disagreement about a given paper couldn't be settled by the two initial raters, a third person acted as the tie-breaker.
The volunteers were an internationally diverse group. Team members' home countries included Australia, USA, Canada, UK, New Zealand, Germany, Finland, and Italy.
The Self-Ratings
As an independent test of the measured consensus, we also emailed over 8,500 authors and asked them to rate their own papers using our same categories. The most appropriate expert to rate the level of endorsement of a published paper is the author of the paper, after all. We received responses from 1,200 scientists who rated a total of over 2,100 papers. Unlike our team's ratings that only considered the summary of each paper presented in the abstract, the scientists considered the entire paper in the self-ratings.
The 97% Consensus Results
Based on our abstract ratings, we found that just over 4,000 papers expressed a position on the cause of global warming, 97.1% of which endorsed human-caused global warming. In the self-ratings, nearly 1,400 papers were rated as taking a position, 97.2% of which endorsed human-caused global warming.
We found that about two-thirds of papers didn't express a position on the subject in the abstract, which confirms that we were conservative in our initial abstract ratings. This result isn't surprising for two reasons: 1) most journals have strict word limits for their abstracts, and 2) frankly, every scientist doing climate research knows humans are causing global warming. There's no longer a need to state something so obvious. For example, would you expect every geological paper to note in its abstract that the Earth is a spherical body that orbits the sun?
This result was also predicted by Oreskes (2007), which noted that scientists
"...generally focus their discussions on questions that are still disputed or unanswered rather than on matters about which everyone agrees"
However, according to the author self-ratings, nearly two-thirds of the papers in our survey do express a position on the subject somewhere in the paper.
We also found that the consensus has strengthened gradually over time. The slow rate reflects that there has been little room to grow, because the consensus on human-caused global warming has generally always been over 90% since 1991. Nevertheless, in both the abstract ratings and self-ratings, we found that the consensus has grown to about 98% as of 2011.
Percentage of papers endorsing the consensus among only papers that express a position endorsing or rejecting the consensus. From Cook et al. (2013).
Our results are also consistent with previous research finding a 97% consensus amongst climate experts on the human cause of global warming. Doran and Zimmerman (2009) surveyed Earth scientists, and found that of the 77 scientists responding to their survey who are actively publishing climate science research, 75 (97.4%) agreed that "human activity is a significant contributing factor in changing mean global temperatures." Anderegg et al. (2010) compiled a list of 908 researchers with at least 20 peer-reviewed climate publications. They found that:
"≈97% of self-identified actively publishing climate scientists agree with the tenets of ACC [anthropogenic climate change]"
In our survey, among scientists who expressed a position on AGW in their abstract, 98.4% endorsed the consensus. This is greater than 97% consensus of peer-reviewed papers because endorsement papers had more authors than rejection papers, on average. Thus there is a 97.1% consensus in the peer-reviewed literature, and a 98.4% consensus amongst scientists researching climate change.
Why is this Important?
Several studies have shown that people who correctly perceive the scientific consensus on human-caused global warming are more likely to support government action to curb greenhouse gas emissions. This was most recently shown in McCright et al. (2013), recently published in the journal Climatic Change. People will defer to the judgment of experts, and they trust climate scientists on the subject of global warming.
However, research has also shown that the public is misinformed on the climate consensus. For example, a 2012 poll from US Pew Research Center found less than half of Americans thought that scientists agreed that humans were causing global warming. One contributor to this misperception is false balance in the media, particularly in the US, where most climate stories are "balanced" with a "skeptic" perspective. However, this results in making the 3% seem much larger, like 50%. In trying to achieve "balance", the media has actually created a very unbalanced perception of reality. As a result, people believe scientists are still split about what's causing global warming, and therefore there is not nearly enough public support or motivation to solve the problem.
Such false balance has long been the goal of a dedicated misinformation campaign waged by the fossil fuel industry. Just as one example, in 1991 Western Fuels Association conducted a $510,000 campaign whose primary goal was to "reposition global warming as theory (not fact)." These vested interests have exploited the media desire to appear "balanced."
Open Access for Maximum Transparency
We chose to submit our paper to Environmental Research Letters because it is a well-respected, high-impact journal, but also because it offers the option of making a paper available by open access, meaning that for an up-front fee, the paper can be made free for anybody to download. This was important to us, because we want our results to be as accessible and transparent as possible.
To pay the open access fee, in keeping with the citizen science approach, we asked for donations from Skeptical Science readers. We received over 50 donations in less than 10 hours to fully crowd-fund the $1,600 open access cost.
Human-Caused Global Warming
We fully anticipate that some climate contrarians will respond by saying "we don't dispute that humans cause some global warming." First of all, there are a lot of people who do dispute that there is a consensus that humans cause any global warming. Our paper shows that their position is not supported in the scientific literature.
Second, we did look for papers that quantify the human contribution to global warming, and most are not that specific. However, as noted above, if a paper minimized the human contribution, we classified that as a rejection. For example, if a paper were to say "the sun caused most of the global warming over the past century," that would be included in the less than 3% of papers in the rejection categories.
Many studies simply defer to the expert summary of climate science research put together by the Intergovernmental Panel on Climate Change (IPCC), which states that most of the global warming since the mid-20th century has been caused by humans. According to recent research, that statement is actually too conservative.
Of the papers that specifically examine the human and natural causes of global warming, virtually all conclude that humans are the dominant cause over the past 50 to 100 years.
Net human and natural percent contributions to the observed global surface warming over the past 50-65 years according to Tett et al. 2000 (T00, dark blue), Meehl et al. 2004 (M04, red), Stone et al. 2007 (S07, light green), Lean and Rind 2008 (LR08, purple), Huber and Knutti 2011 (HK11, light blue), Gillett et al. 2012 (G12, orange), Wigley and Santer 2012 (WS12, dark green), and Jones et al. 2013 (J13, pink).
Most studies simply accept this fact and go on to examine the consequences of this human-caused global warming and associated climate change.
Another important point is that once you accept that humans are causing global warming, you must also accept that global warming is still happening; humans cause global warming by increasing the greenhouse effect, and our greenhouse gas emissions just keep accelerating. This ties in to our previous posts noting that global warming is accelerating; but that over the past decade, most of that warming has gone into the oceans (including the oft-neglected deep oceans). If you accept that humans are causing global warming, as over 97% of peer-reviewed scientific papers do, then this conclusion should not be at all controversial. With all this evidence for human-caused global warming, it couldn't simply have just stopped, so the heat must be going somewhere. Scientists have found it in the oceans.
Spread the Word
Awareness of the scientific consensus on human-caused global warming is a key factor in peoples' decisions whether or not to support action to reduce greenhouse gas emissions. However, there is a gap here due to the public's lack of awareness of the consensus. Thus it's critical that we make people aware of these results. To that end, design and advertising firm SJI Associates generously created a website pro-bono, centered around the results of our survey. The website can be viewed at TheConsensusProject.com, and it includes a page where relevant and useful graphics like the one at the top of this post can be shared. You can also follow The Consensus Project on Twitter @ConsensusProj, and on Facebook.
Quite possibly the most important thing to communicate about climate change is that there is a 97% consensus amongst the scientific experts and scientific research that humans are causing global warming. Let's spread the word and close the consensus gap.
Coming tomorrow, details about a feature that will let you test our results by rating the papers directly yourself. The Consensus Project results have also been incorporated into the rebuttals to the myths There is no consensus and IPCC is alarmist.
Congrats to everyone involved! Undertaking this research clearly required an immense amount of work and dedication, not to mention enduring an illegal hack on the SkepticalScience website (the stolen material which fake skeptics were [and are] only too happy to disseminate, is evidence that ethics and morality are extremely low on the list of priorities of fake skeptics).
Regardless, this independent study by Cook et al. (2013) corroborates previous research, and once again underscores the fact that anthropogenic warming is indeed a theory, with multiple independent lines of evidence have lead to this consensus (consilience in fact).
In stark contrast, the radical 3% cannot even seem to agree on what to disagree about, they are in a state of chaos, have an alarming propensity to engage in conspiracy ideation and routinely contradict each other and even themselves. The spectrum of positions held by this fringe element range from those who deny the existence of the so-called greenhouse effect, to self proclaimed "lukewarmers" and fake skeptics. Indeed, this fringe element seem even more disorganized than those who deny the theory of evolution.
Climate "skepticism" is in disarray, hardly surprising given that their position/beliefs are based on ideology and politics and not on sound physics.
Man, this was fast. Congrats to all co-authors and volunteers.
About the skeptics, I think a more accurate sentence would be
We fully anticipate that some climate contrarians will move the goalposts by saying "we don't dispute that humans cause some global warming."
I'll add this paper to the Wikipedia article on Global Warming (Portuguese version).
PS: SJI link is incomplete and not working.
Albatross at 09:26 AM on 16 May, 2013
I agree with you that this disagreement among 'skeptics' should be more stressed. They are:
1) a very heterogeneous group, with conflicting theories among them
2) a very tiny minority, as shown here (again)
3) not backed up by evidence
But they are very very noisy!
Congratulations to John Cook and the SkS team for this important paper. I know how much work was involved and the team that carried it out have done a marvelous job.
[JH] You were an integral part of the SkS team and made major contributions to the TCP effort. Thank you for that as well.
A huge amount of work; well done!
It's remarkable to see in black and white numbers how isolated contrarians have made themselves.
Yes, congratulations are in order. A team of volunteers, contributing significant amounts of their personal time - an accomplishment to be proud of. And to get such a large number of scientists to participate by rating their own papers - that in itself is an indication of the respect that active climate scientists give to the team and the leadership of John Cook.
And kudos to the rush of SkS readers that provided the funds (10 hours!) to make the paper Open Access. Readers that were obviously willing to put their money where there eyes are...
You can't argue with the evidence. Looks to be an excellent effort all around. Congratulations to John, Dana and all the authors and everyone who worked so hard to bring this to fruition. Also to the donors.
I see it's popping up all over the mainstream press. It's having a solid impact.
Let the whining begin. Congratulations, people! The self-ratings results are a slap upside the head of the Watts-bots.
Yeah, I'm really glad we did the self-rating thing. It removes any notion that the results are just the SkS rating being biased. In fact, it shows that the SkS ratings were very conservative in their judgement.
skywatcher@8
I've sent an email to Peter Hannam today at 14:00AET (Sydney) requesting the correction. Will see how long this simple fix takes him.
Everyone in climate blog circles (I guess also most climate scientists) know Dana. But pupolar press editors still don't know him and make big gaffes about him. It's like AGW scientific concensus vs. lack of public awareness about it.
Yes, the paper is all over the media. More coverage than we even expected, which is awesome. We can't even keep up with all the articles!
The he/she mistake actually originated in the Reuters article. Innocent enough – you only talk with people over the internet, they never actually see you, and Dana could be either gender. I've emailed Doyle at Reuters about the mistake as well.
I'll buy that T-shirt, by the way. (With maybe a bit less text)
That's a strong graphic.
Zapped onto CNN this morning (local Dutch time) to find John Cook talking to me about this survey, great :)
I think it's important to remember the specific question we asked: are we causing most recent global warming?
Logically, this means there is also a strong consensus that the rise in CO2 is man-made, the greenhouse effect is real etc. The 'skeptics' that say otherwise are backed up by basically no research that was good enough to pass peer review.
The evidence for man-made global warming is far too strong to throw out, but on some of the other details we might find more interesting answers if we get the chance to expand this sort of analysis.
I am trying to get this straight. The study examined the abstracts of 12,464 papers and about a third of them endorsed the fact that humans are causing global warming. Yet the papers conclusing is that there is a 97% consensous. I am sorry, but that just doesn't make sense. It sounds to me like most papers are scientifically proper and did not take a postition on political issues.
IAC, the issues in the anthropogenic global warming hypothesis are considerably more complex than just yes or no. This has become even more the case in the last few years as the earth's average surface temperature doesn't seem to increase -- at least for now. This will clearly result in more papers trying to explain this fact on subjects such as climate sensitivity, as well as whether or not CFCs caused more warming than originally thought, radiation of heat into space as earth's effective temperature increases, and whether the saturation of absorption of EMR by CO2 in the atmosphere actually fits the logrithmic curve. Papers on these subjects as well as recent ones already publised on water vapor feedback will be the interesting ones and they may not be found by a survey such as this.
I also note that it is clear that some global warming has clearly been caused by human activity. So, I was curious as to how the study would classify a paper which stated either that we don't know how much of the warming was caused by human activity or that stated a percentage which was less than 50% of it as being caused by human activity.
Excellent work. One question, how would a paper which accepted the basic chain of human emissions -> more CO₂ in the atmosphere -> warming -> positive feedbacks (water vapour, etc) but then proposed that there were large negative feedbacks which cancel out most of the effect be counted? I'm thinking of the UAH guys and Richard Lindzen who, as I understand things, have views like this.
(Read this article and your Guardian post but not the main paper - sorry if this is covered in the paper.)
JRT256: Your post raises several distinct questions, which need separate answers.
Firstly, only a third of the abstracts stated a position on whether mankind was the principle cause of recent warming. However the author self-ratings, based on the whole paper, increased this proportion to two thirds.
Secondly, the paper is very clear that the 97% consensus is among papers which stated a position.
Finally, your confusion is based on a false assumption that every paper which mentions global warming is trying to test whether global warming is occuring and is man made. But many papers with the appropriate keywords in the title only deal with parts of the question. For example a paper on measuring the global warming signal in the instrumental temperature record may say nothing about the cause. Thus it is expected that a significant proportion of the papers will have no position on the question. Including such papers is as meaningful as including all the papers on the colours of butterfly wings.
Whether human activity is causing the majority of recent warming is not a political question, it is a scientific question. The question 'what should we do about it' is a political question.
JRT256:
A 'we don't know' paper would be counted as taking a position but uncertain (category 4b). A 'less than 50%' paper would be counted as an explicit rejection of the consensus (category 7).
Ed Davies:
You would have to look in the database for the answer on individual papers - the study criteria were based on past warming. Low sensitivity has implications for past warming. Is depends how this is handled in the abstract (or the paper for self ratings).
18. Ed Davies, it would depend on the statement of the result in the abstract. I don't remember encountering one that did not discuss the impacts in a way that was translatable to the question are we causnig most of the recent global warming?
e.g. if someone found a very large negative feedback then the abstract would put it in context in terms of climate sensitivity or impact on warming estimates, as the abstract is supposed to provide background. A large negative feedback would likely lead to a small climate sensitivity and would therefore be a rejection by the definitions in Table 2 of the paper.
But if they found a small negative or positive feedback without specifying the overall effect on warming, then it would be put as no position.
We were cautious here, tending towards 'no position' in our ratings. This is confirmed by the scientists' responses compared with ours: more than half of our 'no position' abstracts were rated as 'endorsement' by the scientists who actually wrote the papers! A few 'no positions' were rated as 'rejection' by the authors too, but on average the scientists rated 0.6 classifications closer to endorsement than we did, on the 7-point classification scale.
Congratulations to the Skeptical Science team!
The paper has also made an impact around some news sites, I hope that grows.
www.ft.com/intl/cms/s/0/28d54536-bd7b-11e2-a735-00144feab7de.html
Thank you and congratulations to John and the team for this effort and the resulting paper and excellent post here.
While there is -- so to say -- nothing new here, i.e. the consensus was present 20 years ago and so was the (successful) denial ot it, there appears to be a difference when looking into the mainstream press: the spread of this "news" is somewhat astonishing.
I think the denier reaction may therefore provide a good study case of the pschycological effect and response to being told that you are in denial. It has already started ...
Majority is given when, out of a group of consulted people, more votes are given to one view than to other ones.
Unanimity happens when all voting people positively chose one view, without any exception.
Consensus will be reached when nobody expresses his or her opposition to one view.
All of this happens in opinion polls or formal votes, but not in science.
So, what is the fuss about grading scientific papers in the way they convey [or not] opinions?
John, Dana,
In to your paper, ‘endorsement’ comes in three different ways:
A. Explicit, with quantification;
B. Explicit, without quantification;
C. Implicit.
However, your paper (nor the supplementary info) doesn’t provide a proper breakdown of the various groups. Can you enlighten on this? In particular the results for group A respectively B.
Thanks in advance,
Bert Amesz
Michel:
First, on what basis should we accept your definition of consensus over the implicit/explicit definition/denotation used in the paper?
Second, minuscule proportions of contrarians notwithstanding, consensus absolutely does exist in other areas of science: special/general relativity, quantum mechanics, plate tectonics, evolution, germ theory of disease, to name a few.
Third, that humans are causing global warming is not an opinion. Based on the available evidence it is a settled fact.
Michel, look at the top left of this page below 'Most used climate myths', number 4 (guess that one will be updated :) ).
Michel, Wikipedia describes scientific concensus as "Scientific consensus is the collective judgment, position, and opinion of the community of scientists in a particular field of study. Consensus implies general agreement, though not necessarily unanimity. "
This seems pretty reasonable to me. If you describe "consensus" as being "reached when nobody expresses his or her opposition to one view", then there is no concensus on any scientific topic as there is no field in which there are no dissenters from the mainstream view. This means that either the word "concensus" is meaningless in a scientific context, or your definition of "concensus" is not appropriate.
Can you name a scientific topic on which there is concensus according to your definition?
Kevin C @19,
I believe that the openning statement to this post is in disagreement with your response to JRT256. You state that the article clearly states that the 97% is only for papers that took a position
The openning state is at-best misleading, and needs to be corrected.
HJones: Your paraphrase of my statement misrepresents my statement which you quote directly below it. Further, you disregard my subsequent paragraph which clarifies the issue.
It's really very simple: The study distinguished between papers which do not address the question of the human contribution to global warming, and papers which addressed the question and were undecided on the answer. To argue that papers which do not address the question should be counted as 'undecided' in quantifying the level of consensus is preposterous.
Dikran Marsupial @28:
1) How couild Michel possibly know that there was a scientific concensus on any particular theory by his definition? The only way to determine it would be by an exhaustive survey of all relevent experts (and no such exhaustive surveys exist in any field) or an exhaustive survey of all relevant literature including non-peer reviewed literature. His definition is clearly intended only to turn "There is no scientific consensus on climate change" into a trivial truth with no more information content than informing people that all bachelors are unmarried. Presumably the next step will be an equivocation so as to confuse people into thinking there is no scientific consensus as defined in the paper (and wikipedia).
2) It is informative, however, to look at some of the fields in which it is known that their is no consensus (by Michel's definition). These include not only the fields heavilly disputed on ideological grounds such as climate science and biological science, but also such theories as relativity, geocentrism, and even the theory that the Earth is not flat. The failure of even heliocentrism and the theory that the Earth is round to command a concensus by Michel's definition shows just how pointless it is.
H Jones @29, the sentence "A new survey of over 12,000 peer-reviewed climate science papers by our citizen science team at Skeptical Science has found a 97% consensus in the peer-reviewed literature stating a position on the issue that humans are causing global warming" is not a contradiction. As it is formed from the lead sentence of the article simply by adding the bolded words, and that addition amounts to a conjunction it follows that the lead sentence is true if the ammended sentence is true.
This may not be apparent because in normal communication we expect relevent facts and qualifications to be stated, which the lead sentence did not to. It could be argued, therefore, that while formally true, the lead sentence had a misleading (conversational) implication. That seems, however, a ridiculous claim given that the lead sentence is immediately followed by a massive figure clearly stating the qualification you suppose to be misleadingly left out. The best, therefore, that can be said of your comment is that it is quibbling. The lead sentence is neither false, nor misleading in context, a fact that is readilly apparent.
@Tom, well quite! ;o) I like to ask questions that help people to make their point clearly. In this case the ability to specify a scientific topic where Michel could at least argue a concensus exists using his own definition would at least show that he himself thought that the word had a meaningful use in a scientific context and that this wasn't merely an excercise in rhetoric. The ball is in Michel's court.
Tom Curtis,
Your bold highlights is very clear. However, that is not what is the lead in statement to this article. That was the purpose of my comment, and the only purpose btw. The openning statement is misleading, if your bold highlight is added, I would have no problem with the statement, or converseladd the reference to the number of papers that take a position. The way the statement now reads, 97% of 12000 papers support the position, which is not accurate!
HJones @34, that is incorrect. The way the opening sentence now reads, it states that:
1) Over 12,000 papers were surveyed; and
2) A 97% concensus was found in the survey.
As it happens, a 97% concensus of papers stating a position on AGW was found. As a matter of logic, if a 97% concensus of papers stating a position on AGW is found, then a 97% concensus is found; and hence (2) is true.
You resist this conclusion because it is a rule of conversational implicature that relevant information will be provided. Given that rule and only the lead sentence, it follows by conversational implicature (but not by logical implicature) that the concensus is of the 12,000 papers. However, the lead sentence did not appear alone. It appeared as part of an article which made it clear that the concensus was restricted to those papers actually stating a position. As it is a cardinal rule of interpretation that sentences be interpreted in context, it follows that the sentence is not misleading.
The worst that can be said of it is that poor phrasing creates an unnecessary distraction. Regardless of whether it is a reasonable distraction, I am sure no harm would be done in ammending the sentence by adding the bolded words from my 32. However, that is not your criticism and your criticism is wrong. Correctly stated your criticism is that the lead sentence creates a false impression by ignoring the rules of conversational implicature; whereas what has actually happened is that you have gained a false impression by ignoring the rule that all sentences must be interpreted in context.
I would like to know if there was a marked difference between the ratings for abstracts/papers under the two different search terms.
It is possible (likely?) that searching under 'global warming' might yield more positive results re the consensus than 'global climate change.' Would it be worthwhile comparing to see if the results have been begged by the search terms?
At Lucia's they quote one of the authors as saying that the rating was not strictly independent.
http://rankexploits.com/musings/2013/i-do-not-think-it-means-what-you-think-it-means/
And provide graphics backing up that point. How independent was the rating?
Congrats on this work. It sounds like quite an effort, and getting 1200 scientists to participate in categorizing their own work (important considering how studies are routinely spun by a certain crowd) is impressive.
barry, it seems to me that they are clutching at straws. Independent means merely that the two raters did not discuss the particular paper that they were reviewing in arriving at their judgements on that paper. That doesn't mean that uncertainty in how the criteria should be applied cannot be discussed in a more general context.
The authors of the papers themselves generally rated their work as more strongly supporting AGW than did the SKS reviewers, which is the opposite of what you would expect if "independence" os the SKS reviewers was an issue. I would say that this indicates that the SKS reviewers were actually rather conservative and self-skeptical.
That sounds rather like us lot, Dikran!
37. barry, I can specifically comment on the latter parts regarding disagreement.
We were aware of the user handles of who had done some ratings, but we didn't know who had done which ratings. In the paper we described this as 'two independent, anonymized raters'. So on Lucia's last point about hashing out disagreements, we knew we had disagreed with one of the other 23 raters, but we did not know with whom.
At the very beginning of the process there were a number of questions about difficult cases or missing aspects of the system we used. For example, a number of papers had no abstract (or a truncated abstract), and the best way to highlight these for removal from the analysis was discussed.
There were very few such special cases and we can be confident are results are solid because we found we were more conservative than the authors!
The system described in MarkR's comment is rather like the peer review process used in some of the top conferences in my field (machine learning). First the reviewers are selected by the programme committee from a pool of volunteer revewers. Like the pool of reviewers used by this project, the pool of reviewers will not be independent in the sense of not working with eachother, or not being friends, or having common interests. Each paper is assigned three or more reviewers, neither of which know the identity of their fellow reviewers. Once the reviews are completed in isolation the reviewers get to see the comments made by the other reviewers and have an opportunity to revise their review, but this is again done anonymously. The review process for such conferences provides "independent" reviews in the same sense as used in this survey, and provides a reasonable prescedent for the approach that was taken.
As I remember it, it would have been utterly impossible to discuss any specific paper between two raters. You're going through so many of them at a time and they're coming at you in a random manner. There were some discussions involving the definitions of the categories, but I think that's about it. Such as, one issue that came up for me was, mitigation. If a paper is a migitation paper, is it not, by default, then implicitly accepting AGW?
I think everyone had to interpret the categories in their own way. And the self-raters had to do the same. That's the whole point to having lots of different people doing the work and using a very large sample size.
And besides, what I think keeps getting ignored over at Lucia's place is that, heck, the SkS raters were far more conservative in their ratings than the scientists who actually wrote the papers. The whole point of getting the scientists to rate the papers was to build in a check on potential bias from the SkS ratings.
Another thing that I thought was interesting about the results was that, given the challenges of varying interpretation, how consistent the results actually were. I think that speaks volumes about the robustness of the process that John set up.
Wow. I saw a short article on this study earlier today, but somehow didn't make the connection to the SkS project. Now that I search Google News it seems like there is a lot of great press around this. Congrats all!
Amazing that so many people in this thread apparently would, "...expect every geological paper to note in its abstract that the Earth is a spherical body that orbits the sun?" Just when you think the 'skeptics' cannot get any more ridiculous...
This is a great study.
One thing it shows (and that was really driven home to me when I participated in John's follow-up survey) is how conservative most scientists are with their projections and statements. Where most of us in other fields would be screaming "is" and "will" climatalogists whisper "might be" and "could."
I contrast that with the tone of the so-called skeptics who usually speak in sweeping absolutes.
Anyway, just a few thoughts from a member of the generally unwashed public who has found this site invaluable in getting off the fence when it comes to climate change.
barry @37 - define what you (or Lucia) mean by "independent".
bert @25 - our full ratings database should be available in the supplementary material (though I don't have time to look at the moment).
When did the scientific consensus on climate change decisively emerge? Early '90s, '80s, earlier? Is this too subjective a question? I suppose another way of asking this is when was the last time anthropogentic climate change was seriously argued about in a scientific journal?
yphilj: That's a very good question. The current paper shows a robust consensus back to 1991. The only similar earlier work I'm aware of is this paper covering the period 1965-1979 (including the 70's of the mythical 'global cooling' predictions). They found 7 papers predicting cooling, 19 neutral and 42 warming.
Of the cooling papers, one was Rasool and Schneider 1991. By the end of the 70's Schneider had changed his mind. Three more of the cooling papers were equivocal on the question - see Ari's page here.
On this basis you could probably argue that the consensus was emerging during the 70's.