Arguments
A database of peer-reviewed papers on climate change
Posted on 31 March 2010 by John Cook
I'm already finding the Global Warming Links page a useful resource with many links to webpages on specific skeptic arguments. Thanks to all those who've been submitting links. However, I received an email from Mark this morning making the worthy point that links should be about quality, not quantity. And the best indication of quality is the amount of peer-reviewed literature. As it so happens, the directory is set up so when you submit a new link, you can flag it as peer-reviewed. So it was quite straightforward to set up a page that displays all peer-reviewed papers for each skeptic argument.
Of course, laying out the usual disclaimers, I haven't spent that much time populating this list. Over the last few years, I've only been adding skeptic peer-reviewed papers to the database. Since I created the directory on March 2, I've begun adding peer-reviewed papers that rebut skeptic arguments but only sporadically. Most of the peer-reviewed papers I cite throughout this website aren't even listed yet (although I did just go through yesterday's post on Greenland ice mass loss and add Khan 2010, Velicogna 2009, Vermeer 2009, Pfeffer 2008 and Kopp 2009).
So if you're a climate wonk with too much time on your hands, please feel free to submit any peer-reviewed papers to our links database (yes, I've added this as another way you can support Skeptical Science). There are plenty of peer-reviewed links scattered throughout this website. Particularly rich lodes can be found in the positives and negatives of global warming and the evidence for global warming. However, if you really want to hit the mother-lode, strap yourself in then head over to AGW Observer. The creator of this site, Ari Jokimäki, has dedicated himself to collecting links and abstracts of every peer-reviewed papers on climate change that he can get his hands on (and as if that's not enough, he's also organised translating Skeptical Science into Finnish).
Thank you for all the work you do on this site.
Hi experts. Pleas comment this link. I have feeling that it is another hoax of the denialits.
http://agwobserver.wordpress.com/index/
Many of the "sceptical papers" have been published in the Energy & Environment. Is it Peer-Reviewed, Academic Journal as EBSCO lists it? Or is it a trade journal as Scopus lists it? Which is more reliable EBSCO or Scopus list?
Then, shouldn't they be removed from the peer reviewed papers list?
I also agree that E&E shouldn't be considered peer-reviewed, because it is not a scientific journal, but a social science journal. It is not carried in the ISI listing of peer-reviewed journals. Its peer reviewed process has been widely critizised for allowing the publication of substandard papers. The editor has said “I’m following my political agenda — a bit, anyway. But isn’t that the right of the editor?”. See also this article published in Environmental Science and Tecnology (a journal of the American Chemical Society), and has recently made it clear that she is interested in the conclusions of the papers rahter than the methodology.
'It hasn't warmed since 1998'
Limits on CO2 Climate Forcing from Recent Temperature Data of Earth
'Oceans are cooling'
Cooling of the global ocean since 2003
'Hockey stick is broken'
A 2000-year global temperature reconstruction based on non-treering proxies
'Climate's changed before'
A 2000-year global temperature reconstruction based on non-treering proxies
Greenfyre’s blog has looked into some exaggerated claims associated with E&E. http://greenfyre.wordpress.com/2009/11/18/poptarts-450-climate-change-denier-lies/
Back on topic ... John, thanks for setting up this database. Is there any particular way you'd like to have the titles handled? I noticed several people had been using the article title, followed by author's name and year in parentheses (e.g., Interactive investigation of climate science topics via a collaborative web portal (Cook et al. 2010)). That seemed like a good idea to me, so I adopted it....
That web portal paper sounds like a cracker :-)
McKitrick describes Dr. Rasmus Benestad as "a blogger at Real Climate", technically true, then McKitrick goes on to dismiss Benestad's failure to follow up on an invitation to submit a comment to JGR with the rather snide remark, "Yes, yes, time pressures: I understand. I guess blogging takes up a lot of time."
Here's Dr. Benestad's bio as provided to Real Climate:
I am a physicist by training and have affiliations with the Norwegian Meteorological Institute (met.no) and the Oslo Climate Group (OCG) [My views here are personal and may not necessarily represent those of RegClim, OCG, met.no, or the mentioned societies]. I have a D.Phil in physics from Atmospheric, Oceanic & Planetary Physics at Oxford University in the United Kingdom. Recent work involve a good deal of statistics (empirical-statistical downscaling, trend analysis, model validation, extremes and record values), but I have also had some experience with electronics, cloud micro-physics, ocean dynamics/air-sea processes and seasonal forecasting. In addition, I wrote the book ‘Solar Activity and Earth’s Climate’ (2002), published by Praxis-Springer, and I was a member of the council of the European Meteorological Society for the period (2004-2006), representing the Nordic countries and the Norwegian Meteorology Society.
Here is Dr. Benestad's record of publications:
Rasmus Benestad's Publication list
Ross McKitrick jumps directly into characterizations of others' work as "fabrications" and swiftly moves to arguable distortions such as Dr. Rasmus Benestad being simply "a blogger," of course leaving a credulous reader with the impression that Benestad is insufficiently credentialed.
I suppose McKitrick is trying to harm the credibility of his opponents, but instead such tactics leave me wondering how much fact-checking I'm required to do in order to attach any credibility to his claims. Life is too short, I won't bother and I must leave the question of whether he was fairly treated in the hands of other, more easily believed folks, such as editors and reviewers with sufficient proven reliability to be found working for reputable journals. If most of those persons chose not to publish McKitrick's work, my vote of confidence must go to them because for me McKitrick has ruined his own credibility by swerving into rank hyperbole.
You further note that McKitrick 'jumps directly into characterizations of others' work as "fabrications"'. Actually, the only time he uses the word 'fabrication' is in reference to a paragraph in an IPCC draft that dismisses his work with misleading citations and an unsubstantiated assertion (which he explores and writes a paper to show is incorrect).
I believe that the reason these items in McKitrick's article feel like 'rank hyperbole' is because he is casting a shadow on people you respect, and I probably would feel the same. However, this thread is about peer reviewed papers, and regardless of who is right in the AGW debate, McKitrick's article is first person evidence that the implications of papers on AGW affects peer review. Phil Jones' email about 'redefining peer review' is additional evidence.
You mention that that you were going to 'leave the question of whether he was fairly treated in the hands of other, more easily believed folks, such as editors and reviewers with sufficient proven reliability to be found working for reputable journals.' Their reliability and absence of bias is precisely the question at hand.
McKitrick gives personal experience as evidence
As to the merit of his charges, I read nothing in the work that could not describe the experience of a myriad of other researchers working on less controversial topics. Perhaps McKitrick has been treated somewhat more critically than other workers, but then his own posture and history of remarks about other researchers almost invites that.
"Circling the Bandwagons" may be an effective title choice when talking to a private audience, but as an invitation to serious consideration by others it fails. Meanwhile, McKitrick's conclusion that "the IPCC used false evidence to conceal an important problem with the surface temperature data on which most of their conclusions rest" is hyperbole in the extreme and of course relies on the reader failing to remember that McKitrick's analysis (which he himself admits was flawed) dealt with but a single type of data from the plethora of research threads incorporated into IPCC's work.
I would like to ask generally of those who have published in peer reviewed journals if the experience of Ross McKitrick described in this essay<\a> is commonplace.
I appreciate the stance of this blog and this thread in that science expressed in peer reviewed papers should be allowed to speak impartially. However, this article throws another light on peer review. If McKitrick's experience is rare, then science is not allowed to speak impartially, and given the significance of AGWs implications, this is a critically harmful situation. If McKitrick's experience is commonplace, then peer review itself is broken, and science needs to find another voice.
All that has not been in climatology, but in the multiple fields experimental cognitive psychology, research methodology, decision theory, and human-computer interaction.
From what I can tell by looking completely from the outside at climatology's peer review process, it seems to be working just fine, and quite similarly to peer review in the my fields that I listed above. Are there problems? Sure. Is there room for improvement? Sure.
I've never been happy about being rejected. Sometimes I've been furious. Sometimes I've known that my submission was rejected despite worse work being accepted. I've reworked submissions multiple times, despite disagreeing that rework was needed, to suit the demands of reviewers and editors. I've got some rejected submissions sitting around that I've effectively given up on, though I probably could get them published if I added some experiments to them (but now I've got more interesting things to work on). But I've never publicly posted a screed about a rejection. I'm flabbergasted that McKitrick has done so.
McKitrick's experience was typical for anyone whose submitted work is severely and fundamentally flawed. Reviewers are not necessarily going to spend the time to list, nor even look closely enough to notice, all the problems with a submission if right off the bat they discover a fatal flaw. Reviewers and editors will advise the author sufficiently so the author can rewrite the paper, if the reviewers and editor think the paper is fundamentally sound enough to be a useful addition to the literature. But otherwise they will not spend the time to help. They will name just the first fundamentally fatal flaw and reject the paper without inviting resubmission. If the author fixes that flaw and resubmits, then the reviewers will read on to discover more flaws.
It's the author's responsibility to ensure their submission is high enough quality to be accepted. McKitrick's submission was not. All the things he complains about in his public screed have been publicly addressed, and his paper is just... well, wrong.
But if anyone wants to discuss the climatological substance of McKitrick's claims in that paper, I think maybe you could do it over at Deep Climate. The most relevant parts are the comments starting with the comment by Paul Middents on April 2.
Tom Dayton writes "All the things he complains about in his public screed have been publicly addressed, and his paper is just... well, wrong" and then directs me to this paper published in Jan 2009 which calls into question McKitrick's 2007 published paper. But the paper that McKitrick has been submitting since early 2008 addresses the objections that have been raised to the 2007 paper, including (according to McKitrick) those articulated in the Schmidt's 2009 paper. I am not sure how his position can be considered to be publicly addressed when it has not even been seen.
Quoting Tom again, "McKitrick's experience was typical for anyone whose submitted work is severely and fundamentally flawed." This is fair, if the editors/reviewers consistently says so. However, McKitrick's summary of the responses he has seen is the following:
"Altogether I sent the paper to seven journals before it went to SP&P. From those seven journals I received seven reviews, of which six accepted the findings and supported publication. The one that rejected my findings contained some basic technical errors, but the journal editor would not respond to my letter pointing them out. Nature, Science and Geophysical Research Letters would not even review the paper, while the Bulletin of the American Meteorological Society never acknowledged the pre- submission inquiry. Global and Planetary Change received one review recommending publication, blocked another reviewer before he could submit a report and then turned the paper down."
This is not how I would expect a working peer review system to respond to a fundamentally flawed paper, even if that system is imperfect.
Comment 27 points to an article that tries to show how peer review is effective at accomplishing quality control while avoiding censorship. I would be interested to see what it takes to qualify as censorship in the climatology field.
This issue regarding McKitrick's paper has been pushed as far as is decent, and I will leave it alone. Thanks again.
For example, the journal Science accepts less than 8% of submissions.
Consequently, editors don't even send all submissions out for review. Many submissions get rejected based on the editor reading the title and abstract, because the topic might not be suitable for that journal regardless of the submission's quality, or because the journal has been publishing many articles on that topic recently and wants to give space to other topics, or because the findings of the submission are not novel enough or do not contribute enough to the literature to justify taking up space. Submissions never are judged solely on their own merits, but also on whether the literature is better served by publishing them or their competition of other submissions. The editor's job is to make those decisions if possible without using the scarce time of the (volunteer) referees. The referees' opinions are merely advisory to the editor, though different journals have different rules about how much weight the editor should give the reviews.
Editors are busy people, so they rarely respond to rejected authors' subsequent inquiries.
It's nice if they at least acknowledge receipt of submissions, but that, too, takes time, so they don't always. It's not that different from any non-scientific periodical; they do not always (and some never) acknowledge unsolicited submissions.
Harold, this really is how the scientific peer review system works, and has for a very long time, in all fields. It handles submissions as best it can, but the reviewers all are unpaid volunteers who already are desperately busy, the editors get paid little or nothing and also usually have full-time other jobs, and there is barely enough funding to publish the small percentage of articles that do make it all the way to print.
Publishing in a peer-reviewed journal (or a conference, for that matter) is not a right, it is a privilege.
I intended to write that pre-submission inquiries such as the one that McKitrick said he made to BAMS are not always responded to. In fact, I suspect that few such inquiries are responded to by any journal in any field, because it takes time. If you want to know whether there is any chance of the journal accepting your paper, just send them your freakin' paper. The editor is disinclined to spend the time to respond to you "Well, maybe, but that depends on exactly what's in your paper." The editor can just as rapidly tell you "no" upon seeing the title and abstract of your paper, as by reading your pre-submission inquiry's description of your paper.
Also there is an inline response by one of the authors of that journal article, Stefan: "I've certainly had more papers rejected than accepted by Nature and Science." (The inline response is to comment #8 by Andy S on 6 April). Stefan's professional and grateful attitude differs starkly from McKitrick's attitude.
In either case the result is the same. Most scientists will (rightly, in my opinion) feel that it's not worth their time to read papers from E&E. Given the huge expansion in the number of journals and the number of papers being published, it's difficult enough to keep up with the literature as it is!
[E&E is] Found at 44 libraries worldwide, at universities and the library of congress. Including an additional 78 in electronic form.
That seems to be about the right order of magnitude. WorldCat says E&E is currently in 48 libraries worldwide.
Now, that's a really, really small number of libraries.
My own field is pretty narrow, and the journals are going to be inherently of less wide interest than a journal that publishes papers on "environment" and "energy" topics. I just looked up all of the main journals in my field, and they're all in at least 400-600 libraries according to WorldCat.
As far as I can tell, E&E is part of a parallel infrastructure that's grown up to provide a safe and un-skeptical venue for promoting work that (a) questions the mainstream scientific consensus on climate change, and (b) isn't of high enough caliber to be considered in a "real" scientific venue.
So, for example, there's the "NIPCC" report created by Fred Singer as a kind of shadow version of the actual IPCC reports. There's the Heartland climate "conference" which is likewise a kind of weak and nonserious response to actual scientific conferences (AGU, etc.). And then there's E&E, kind of a toy journal for outside-the-mainstream papers only.
The advantage of publishing in E&E rather than a real journal, or presenting at a Heartland event rather than a real conference, is that your work won't actually be subjected to any kind of skeptical review.
A successful journal is one that publishes papers that are cited frequently, open avenues for new productive research, and generally raise the reputation of the journal among scientists working in its field. When that happens, people will submit more papers to the journal because publication in its pages is now considered a mark of distinction. The editors can then be more choosy and pick only the best papers, which further raises the journal's reputation ... that's what every journal editor wants to happen.
This spiral can also go in the opposite direction. When a journal publishes a paper that can't be replicated, or that has clearly demonstrable analytical flaws, that's a bad sign. Likewise, if its papers don't seem to be stimulating new ideas or opening up new areas of research, it won't be cited as often and people will choose to submit their good papers elsewhere.
As I said with E&E, there have been a series of very poor papers that seem to indicate a problem with the peer review process there -- either it doesn't exist at all, or it's not very effective, or somehow papers are getting routed around the review process, or something. So people tend not to take E&E very seriously.
Yes, this is all subjective, but the end result is real. Active scientists don't have much time in their lives, especially those working in academia (as my friend says, "It's a great job, having all this flexibility ... I can work any 80 hours a week that I choose!") So they make judgments about what to spend their time on and reading E&E tends not to make that cut.
Anyway, I'd agree that what I'm talking about is very subjective. If you have suggestions for a more objective way to assess the quality of a journal, one that doesn't involve the opinions of the scientists working in its field, I'd be interested in discussing that.