Climate Science Glossary

Term Lookup

Enter a term in the search box to find its definition.

Settings

Use the controls in the far right panel to increase or decrease the number of terms automatically displayed (or to completely turn that feature off).

Term Lookup

Settings


All IPCC definitions taken from Climate Change 2007: The Physical Science Basis. Working Group I Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Annex I, Glossary, pp. 941-954. Cambridge University Press.

Home Arguments Software Resources Comments The Consensus Project Translations About Donate

Twitter Facebook YouTube Pinterest

RSS Posts RSS Comments Email Subscribe


Climate's changed before
It's the sun
It's not bad
There is no consensus
It's cooling
Models are unreliable
Temp record is unreliable
Animals and plants can adapt
It hasn't warmed since 1998
Antarctica is gaining ice
View All Arguments...



Username
Password
Keep me logged in
New? Register here
Forgot your password?

Latest Posts

Archives

Pal Review - the True Story and the Fairy Tale

Posted on 6 June 2012 by dana1981, JohnMashey

We often hear claims from climate contrarians that climate scientists are guilty of what they describe as "pal review."  The conspiracy theory goes something like this - climate scientists conduct biased research with the goal of confirming the human-caused global warming theory.  They then submit their biased results to a peer-reviewed journal with friendly editors ("pals") who pass their paper along to friendly reviewers (other "pals") who give their fraudulent work the green light for publication.  Thus, the contrarians argue, the preponderance of peer-reviewed literature supporting human-caused global warming is really just a sign of corruption amongst climate scientists.

However, while climate contrarians are never able to produce any evidence to support their conspiracy theory, John Mashey has thoroughly documented a real world example of true pal review.  Contrary to the standard conspiracy theory, the pal review did not involve mainstream climate scientists, but instead the climate contrarians themselves.

The True Story of Climate Research Pal Review

Mashey has done an excellent job documenting a real life case of pal review, which happened at the journal Climate Research between 1997 and 2003.  That particular journal was once again brought to the forefront in the recent second Climategate stolen email release.

In those emails, various climate scientists had expressed concern that Climate Research was publishing shoddy papers by a small group of climate contrarians, and discussed what they could do about it.  The most infamous of these papers was one by Soon and Baliunas (2003) which concluded that current global temperatures are not anomalous compared the past 1,000 years.  After publishing this paper, Soon was invited by Senator James Inhofe to testify before US Congress, and the Soon and Baliunas paper was used by Congressional Republicans to justify opposition to climate legislation.

However, the paper contained numerous major fundamental flaws, such as equating dryness with hotness, and was subsequently roundly refuted by an article in the American Geophysical Union journal Eos written by a number of prominent climate scientists.  This paper, and Climate Research's refusal to revise or retract it, led to the resignation of five of the journal's editors, including recently-appointed editor-in-chief Hans von Storch, who explained the reason for his resignation:

"..the reason was that I as newly appointed Editor-in-Chief wanted to make public that the publication of the Soon & Baliunas article was an error, and that the review process at Climate Research would be changed in order to avoid similar failures. The review process had utterly failed; important questions have not been asked....It was not the first time that the process had failed, but it was the most severe case....I withdrew also als editor because I learned during the conflict that [Climate Research] editors used different scales for judging the validity of an article. Some editors considered the problem of the Soon & Baliunas paper as merely a problem of "opinion", while it was really a problem of severe methodological flaws. Thus, I decided that I had to disconnect from that journal, which I had served proudly for about 10 years."

In short, the journal's chief editor voiced the exact same concerns as the climate scientists in the Climategate 2 emails - that certain Climate Research editors were systematically publishing methodologically flawed papers in their journal.  Soon and Baliunas were far from the only climate contrarians to benefit from the journal's friendly editorial policy.  In fact, the biggest pal review beneficiary bears a very familiar name.

Patrick Michaels and Pals

Mashey has examined the publications in Climate Research in great detail, and has produced a spreadsheet of its publications and a report summarizing his findings.

Prior to Hans von Storch's promotion to Climate Research editor-in-chief in 2003, the journal did not have a chief editor, and so authors sent their manuscripts to an Associate Editor of their choice.  One particular Associate Editor, Chris de Freitas, published 14 separate papers from a select group of 14 climate contrarians during the 6 year period of 1997 to 2003:

Sallie Baliunas, Robert Balling, John Christy, Robert Davis (both Climate Research author and editor), David Douglass, Vincent Gray, Sherwood Idso, PJ "Chip" Knappenberger, Ross McKitrick, Pat Michaels, Eric Posmentier, Arthur Robinson, Willie Soon, and Gerd-Rainer Weber.

As Mashey shows, from 1990 to 1996, Climate Research published zero papers from this group.  From 1997 to 2003, the journal published 17 papers from this group, 14 with de Freitas as the Associate Editor.  Serial data deleter Patrick Michaels was an author on 7 of the 14 pal reviewed papers, which also accounted for half of his total peer-reviewed publications during this timeframe.  During this period, 14 of the 24 (58%) papers accepted by de Freitas came from this group of contrarians.  After von Storch's resignation in 2003, de Freitas published 3 more papers from authors outside this group before leaving the journal in 2006.

Another on the list of 'pals', Robert Davis, was another Associate Editor at Climate Research who accepted 36 papers during his tenure, two of which were co-authored by another pal, Robert Balling.  The journal also published 5 other papers from this group by non-pal editors.  However, in total, at least 16 of the 21 (76%) of the papers published by Climate Research which were authored by this group of climate contrarians had pal review editors, mostly de Freitas (67%) during this six year window.

After von Storch's resignation, Mashey documents that the pals' Climate Research publications dried up.  Davis accepted one of Balling's papers submitted in 2004, and papers co-authored by Balling and by de Freitas were published by the journal in 2008 (Table 1).  18 of the 21 (86%) of the 15 pals' Climate Research publications were submitted in the 1997 to 2003 timeframe.

Table 1: Climate Research publications grouped by Associate Editor.  Grey bars show approximate editor tenure as derived from received dates of papers.  The "pals" papers are shown in red capitals, 14 accepted by de Freitas (bold), and 7 handled by others (red, underlined italics). De Freitas also accepted 13 seemingly normal papers from other authors (lowercase black).

pal review

Mashey also finds that the 15 'pals' were closely connected in climate contrarian activities outside of Climate Research as well, for example working for various anti-climate think tanks, most being connected with either Fred Singer or Patrick Michaels.

"all have shown persistent involvement with organizations that do climate anti-science, most of which also have tobacco connections."

There is also substantial overlap with the pals joining together to author these papers (Figure 1).

pal overlaps

Figure 1: Overlap between pal authors of the 14 de Freitas Climate Research pal review publications between 1997 and 2003.  The node numbering represents the Climate Research volume and page number of the pal publications, while the node connections represent papers written by the same pal authors (i.e. 9.3p14 and 23.1p15 were both authored by Michaels and Knappenberger).  Image by jg and Kevin C.

The Purpose of the Mainstream Pal Review Myth

For those who oppose the prudent path forward with regards to climate change, which involves major global greenhouse gas emissions reductions, the scientific consensus on human-caused global warming is a very inconvenient thing.  Despite the public relations damage resulting from Climategate, people still trust climate scientists' opinions about climate science (although political conservatives' trust in scientists in general has declined).  However, much of the public (at least the American public) doesn't realize that there is a scientific consensus on human-caused climate change.  Polls in October 2010 and September 2011 found that 44% and 37% of the American public believes that scientists are divided regarding the cause of global warming, respectively.

According to the March 2012 George Mason Center for Climate Change Communication (CCCC) national poll, climate scientists are the most trusted source for climate science information, with 74% of public trust (Figure 2).  However, a large segment of the population believes there is a major scientific debate on the subject, no doubt thanks to the false media balance which gives the ~3% minority of experts who think humans aren't the dominant cause of the current climate change (and their non-expert surrogates) ~50% of the media attention.  Therefore, many people  don't believe that humans are the primary cause of global warming (approximately 41% of Americans).

george mason poll

Figure 2: Responses to the George Mason CCCC poll question "How much do you trust or distrust the following as a source of information about global warming?"

The numbers reveal a stark picture: 76% of Americans trust climate scientists, but 41% think scientists are divided on the causes of the warming, and 41% think the observed warming is mostly natural.

Thus as Ding et al. (2011) concluded, if a larger percentage of people realized that there is a scientific consensus on the issue amongst the group they trust most on the subject (and rightly so), more people would believe that humans are causing global warming, and more people would demand that we do something about it. The lack of public awareness of the scientific consensus on human-caused climate change is one of the biggest obstacle to taking climate mitigation action.

For this reason, climate contrarians have attacked the scientific consensus from many different angles.  Some have tried to attack the credibility of the many different surveys and studies documenting the consensus.  Others simply ignore this documentation and deny the consensus exists at all. 

The third group, discussed in this post, attacks the credibility of the consensus itself, claiming it's all part of a massive fraudulent conspiracy of thousands of corrupt climate scientists (note that conspiracy theories are one of the five characteristics of scientific denialism).  Ironically, this conspiracy theory has been most recently voiced by pal review beneficiary Patrick Michaels.

"Peer review has become ”pal review.”  Send a paper to one of the very many journals published by the American Geophysical Union–the world’s largest publisher of academic climate science–and you can suggest five reviewers.  The editor doesn’t have to take your advice, but he’s more likely to if you bought him dinner at the last AGU meeting, isn’t he? That is, of course, unless journal editors are somehow different than government officials, congressmen, or you."

Michaels of course provides no evidence whatsoever to support this conspiracy theory of peer-review corruption.  He expects us to swallow his tale of "pal review" - the conspiracy theory that thousands of climate scientists are publishing thousands of biased papers every year in order to keep the human-caused global warming theory propped up - based on nothing more than his say-so.

While Michaels is indeed something of an expert on the subject, his expertise comes from himself being one of the individuals most guilty of engaging in climate research pal review.

Pal Review Summary

While Patrick Michaels has accused mainstream climate scientists of a vast conspiracy involving pal review (and exposed his own characteristic of scientific denialism in the process) without any substantiation or supporting evidence, in reality Patrick Michaels himself was the biggest beneficiary in the one actual demonstrated case of climate science pal review, as documented by Mashey.

A group of 14 climate contrarians found a sympathetic journal editor who proceeded to publish a large number of papers from this group over a very short timeframe, many of which were scientifically flawed, some of which were subsequently used by politicians to oppose climate legislation.

Ironically, the climate scientists who tried to do something about this problem have themselves been accused of trying to "hijack" or "subvert" the peer-review process.  And of course the guiltiest party of all, Patrick Michaels has accused thousands of climate scientists of the sort of pal review he himself engaged in.

Our tale is one of irony, hypocrisy, and projection.  The next time you see a complaint about the fairy tale of rampant climate science "pal review", direct the accuser to John Mashey's documentation of a pal review true story.

Note: this post has been used as the Intermediate rebuttal to the myth Climate science peer review is pal review.

0 0

Bookmark and Share Printable Version  |  Link to this page

Comments

1  2  Next

Comments 1 to 50 out of 66:

  1. I'm sure that those accusing mainstream climate scientists of undertaking pal-review could surely come up with a similar network of connections to substantiate their accusations - if the accusations had merit, that is.
    0 0
  2. Allow me to make one objection, composer99: most journals do not make it clear who the Editor is that handled the paper. Climate Research did do this until about 2006 or so.

    That means that it will be difficult to do a similar analysis for "mainstream" climate scientists.
    0 0
  3. What about the openly biased Energy&Environment, the ultimate refuge of climate inactivists? I wonder if that is good enough to qualify as pal review... Maybe editor-simpathy-review?
    0 0
  4. Alexandre @3 - I think a key difference is that Climate Research was considered a serious peer-reviewed climate science journal before the pal review incident. Energy&Environment has never been considered a legitimate peer-reviewed climate science journal. But I agree it could qualify as 'pal review'. Ineffective pal review though, since everybody knows the journal isn't serious.
    0 0
  5. Well-known PR techniques in climate anti-science include, in decreasing order of desirability:

    • Publish a paper refuting the existence of AGW in a good journal, such as Science or Nature. This basically does not happen, so next best is to publish one arguing for much higher uncertainty than the mainstream.

    • Find a otherwise-credible journal with process weaknesses. a) Either find a sympathetic editor (CR, deFreitas) or b) exploit the weaknesses, especially if the journal is unfamiliar with the context issues (Remote Sensing)

    • Publish a paper with reasonable, if unremarkable research that only experts follow, but with unrelated comments casting doubt on AGW, that can be quoted as appearing in a peer-reviewed journal. (many cases at CR)

    • Publish in a journal that will publish almost anything.
    (E&E, Journal of Scientific Exploration).
    0 0
  6. While Michaels is indeed something of an expert on the subject, his expertise comes from himself being one of the individuals most guilty of engaging in climate research pal review.

    Projection should be another category, making it 6 characteristics of denialism. From the constant references to blind belief unsupported by evidence (religion), to the conspiracy theories of 'pal' review, through the references to 'alarmist' science (what's more alarmist than a world conspiracy of scientists, or a global plot to control and tax us?), it is so often the case that those making the accusations are most guilty of that which they claim to abhor, as we can clearly see in Michaels' case. One thing is clear: the contrarians are entirely unconcerned with their own egregious hypocrisy.

    Dana1981: much though I admire your contributions, I think you misunderstand the nature of the debate in the same way as do so many with a science background. When you say 'Ineffective pal review though, since everybody knows the journal (E&E) isn't serious', you are really referring to those who are either scientists or informed lay people. The public can make no such distinction, which is why E&E papers are so frequently cited in Guardian debates, for example. They may be duff papers, but they are far from being ineffective. Underestimating the enemy is never a wise strategy, and we are engaged - like it or not - in a propaganda war, which is largely the sub-text of this article.
    0 0
  7. A few comments.

    1) You write "the ~3% minority of experts who think humans aren't the dominant cause of the current climate change". Is this correct? As I remember it the figure is 1%, with 2% being in the 'don't know' category.

    2) GPWayne is correct to call this 'projection'. Because the contrarians operate as a pals network themselves, experience tells them that this must be the way everyone works the system -- I mean, stands to reason doesn't it?

    3) I was confused by the large 'Dana1981' at the start of the second paragraph of GPWayne's comment. I thought for a moment that that was Dana's response and had to read it twice before I realised it was a continuation of Graham's comment.

    Dana: I am astonished by the quality and quantity of your output. More power to your elbow.
    0 0
  8. I'd be rather suprised if Energy & Environment didn't have a significant pal review issue. There simply are not very many skeptic climatologists competent to review the science based papers published there, so unless they regularly send papers out to mainstream climatologists to review (the E&E papers I have read suggest this is unlikely) they would have to settle for incompetent reviewers (i.e. people competent in their own field, but unqualified or inexperienced in climatology).

    Many journals publish the names of the reviewiers that they have used each year (obviously not mentioning which papers they reviewed). It would be interesting to see such a list for E&E. It would go a long way to addressing the pal review issue if more journals did this.
    0 0
  9. gpwayne @6 - maybe I'm in a quasi-ivory tower here. I almost never see E&E papers cited, though I don't read the comments on The Guardian very often.

    John Russell @7 - thanks very much. The 3% figure is a ballpark, depending on what study is being referenced. I agree that technically the figure is probably lower, with 'undecideds' making up the difference.

    It would indeed be interesting to see the reviewers on E&E papers.
    0 0
  10. I put this article as: Imitation is the ultimate flattery. Essentially, this influence network has been described on both sides of this debate (see The Hockey Stick Illusion, of which you're all familiar). (off topic snipped)
    0 0
    Moderator Response: TC: Given the nature of the OP, a well supported "tu quoque" must be considered on topic, and given some lee way to establish their case. Simple assertions, however, violate the comments policy restrictions against ad hominens and "sloganeering". If Rufus9 wants to substantiate his case as John Mashey did, he is welcome to do so. If he continues to simply assert his claim without evidence, however, his comments will not be given leeway with regard to the "no ad hominen" rule.
  11. John Russell - Hi John - funny business about Dana's name appearing like that. It's the 'strong' tag, which I've now changed to 'b' (bold) instead.

    Dana Not surprised you don't spend much time in the Guardian threads, considering what a bloodbath it usually is. However, I was using them as a proxy for a wider target i.e. lay people. I think E&E get traction by 'looking like' sites that have more credibility within the scientific community, but generally I don't think the public can tell the difference, and those seeking to reinforce their own views through confirmation bias probably don't care. But it's a small point, and one I don't want to labour any further. Your work is far too good to harp on about this small point...
    0 0
  12. Well, since it has been raised by Rufus9:
    1) Of course, well-published climate scientists know each other. Well-published scientists in any field know each other.

    2) But for an equivalent situation, one needs to find a substantial set of cases where a) demonstrably-bad papers were shepherded through peer review, over years, by members of the group, b) causing other editors to resign in protest. Even competent researchers make mistakes, but in the CR case, not-so-competent researchers were well taken-care-of. Let's see, Vincent Gray, retired coal researcher gets a *review paper* slot. Well, that makes sense.
    Competent researchers are each others' toughest critics and they especially don't give free passes if friends send around something not up to par.

    3) HSI: Montford not only relied on a non-credible article in a "dog astrology journal" for a key theme in his book, but then he falsified an already-wrong statement about it. In academe, false citation like this is one of the deadly sins, and if Montford were an academic, formal misconduct complaints would have been filed long ago. The only defense would be a plea of incompetence in not reading a key reference.

    See archived HSI dog astrology discussion on Wikipedia, which no one would even address, much less refute. They repeatedly tried to removed it (a no-no), but The Stoat kept reverting it back.

    See He Who Quotes from Dog Astrology Journal, @ Rabett Run or this comment @ Andrew Gelman's.
    0 0
  13. My main point which was addressed above was that drawing a connectivity network between parties with similar views may be interesting, but is not necessarily a sign of mass conspiracy. However, they can be a sign of group think, which is relatively commonplace in science. The peer review process itself is often flawed. One site to keep an eye on is:

    http://retractionwatch.wordpress.com/

    Papers are pulled or corrected all the time, but likely not at a sufficient frequency to correct all errors. No doubt there are flaws in HSI, but at least in my reading there were mathematical or methodological critiques as well, including a potentially biased view of the peer review and editorial review process, but still, even if half true, it raised my awareness of problems in this emotionally charged area of science.

    For a very interesting discussion on scientific research, repeatability, and issues with the peer review process, I recommend listening to this episode of Econtalk with Ed Yong.

    http://www.econtalk.org/archives/2012/06/yong_on_science.html
    0 0
  14. Rufus9 wrote "Papers are pulled or corrected all the time, but likely not at a sufficient frequency to correct all errors"

    This is a very unrealistic expectation! In my experience the majority of papers either have significant flaws in the methodology, experimental sections that do not adequately support the conclusions, or flawed intepretation of the results. Peer review is only ever a sanity check, nothing more, and it is unwise to assume any paper is correct simply because it appears in a journal.

    "Group think" is not really a problem in peer review; the scientific mindset has no problem finding faults in anything, whether it agrees with their preconceptions or not. Personally I view the "group think" as mere rhetoric designed to discredit a particular branch of science that another branch doesn't like. It has nothing to do with science IMHO.

    The major problems with peer review is the "publish or perish" nature of academia, which means there is great pressure on all of us to publish in quantity often at the expense of quality. This is becuase it is hard to make an objective metric that measures quality (at least hard to make one that operates without a delay of 5+ years). This means that (i) there are many more papers published than there used to be (ii) academics have less time to review them than they used to do (iii) comments papers have little or no value, so academics have pretty much stopped writing them. The result is an explosion of the number of journal and a reduction in quality control. Nothing to do with "group think", everything to do with the economics.

    As I said, peer review has always only been regarded as a sanity check, nothing more (unless you are lucky enough to have really good reviewrs). In the past, poor papers get published, but they get ignored by the research community, so nothing needs to be done. It used to be that particularly bad papers recieved comments refuting them to make sure the are ignored, but that additional quality control no longer really exists. The problem these days is that the papers are discussed outside the research community, e.g. on climate blogs, that don't generally have the scientific background to properly understand them, and bad papers are no longer ignore as there will always be some that take them seriously, no matter how bad they are (e.g. any paper that argues that the rise in CO2 is a natural phenomenon).

    Peer review isn't emotionally charged, at least not in the sciences, a mountain is being made from a molehill IMHO.
    0 0
  15. dana1981 added the nice graph to visualize the connections, as opposed to proving anything. I would be happy to hear critiques of my original paper's research, by informed readers who:
    a) Read my entire paper.
    b) Read all the red-labeled papers (ones labeled A-U)
    c) Read the rest of the de Freitas-reviewed papers, a-m
    d) At least read the abstracts of the rest.
    The spreadsheet was published, listing ~700 papers, all with URLs.
    e) Having done that, feel free to argue with my categorizations,i.e., columns D and E, especially of those marked in red. I do not claim these are perfect, and of course I'd be especially interested if anyone finds appears that deserve to be red, or argue informedly that a red one deserve not to be.
    f) Then, pick a random dozen or two of the non-categorized articles edited by people other than Davis or Khandekar, evaluate them in the same way as I did, and categorize them in the same way as I did, with notes like mine on pp.9-14.
    0 0
  16. Without going into the detailed research that John Mashey did, what is most damning to me is the fact that the pals began frequent submissions to Climate Research only on the appointment of Chris De Freitas as Associate Editor, directed their submissions almost exclusively to him, and ceased submissions after their "gig" was discovered. That pattern seems to indicate clearly that they did not think their articles could were worthy of publication, and would need De Freitas to shepherd their articles through peer review. It also strongly suggest that they new before submission that De Freitas would do exactly that.

    Tellingly, fake "skeptic" accusations of "pal review" are never accompanied by evidence of such selective submission patterns.
    0 0
  17. Tom Curtis @16 - I agree, the fact that the pal's CR submissions began when de Freitas started at the journal, dried up after he left, and that he handled the majority of their CR submissions, is pretty damning evidence.
    0 0
  18. Although I also agree with John, Tom, and Dana, I can also imagine that the De Freitas et al group think that their collusion just represents an example of how thorough the "establishment" conspiracy against them was - only by setting up their own "non-biased" editor at a journal outside the clutches of the mainstream scientists could they succeed at getting their "brilliance" published. When you view things as a conspiracy, it's easy to reject any "evidence" that disagrees with your world view.

    To paraphrase from Monty Python's The Oscar Wilde Sketch, De Freitas et al probably think that their papers "shine out like a shaft of gold when all around is dark", when most everyone else in the climate scientist community recognizes that they are "like a stream of bat's piss".
    0 0
  19. Ah but Bob, while the pals got a great many papers published in CR during this timeframe, they were also able to get papers published elsewhere. Michaels for example got 7 published in CR and 7 published in other journals at the same time.
    0 0
  20. Dana: think of how many papers Michaels could have gotten published elsewhere if there wasn't a conspiracy against him! Seven per journal, dozens and dozens of journals. ;-)
    0 0
  21. re: 16 Tom
    "directed their submissions almost exclusively to him, and ceased submissions after their "gig" was discovered. "

    Note that we don't actually *know* that, although it seems likely. We don't know anything about papers submitted, but rejected, only the ones actually published.

    I'd guess your conjeecture is true, but we really don't know.

    "published almost exclusively through him, and essentially ceased publication after their "gig" was discovered. "
    0 0
  22. John - nice admission that "... we really don't know." What is true of your study as well as many others is that it has established correlation but not causation. As such, the results are sufficient for generating hypotheses, but not "proving" anything (I'm not saying that was your objective). Similarly Dana notes that " Michaels for example got 7 published in CR and 7 published in other journals at the same time."

    Without some additional information, it is hard to know how to interpret that statement. Does it mean that there were 7 other journals that were also susceptible to "pal reviews", or does it lend credibility to Michaels because he was also published in those other journals. Once again, we can generate a hypothesis, but cannot prove it one way or another based on your work.

    If you listen to the Econtalk podcast I linked previously, there is a very good discussion of the pitfalls of scientific studies, and we should not be so naive to think that bad research is limited to once side of the climate debate or the other. As Dikran (14) noted,

    "The major problems with peer review is the "publish or perish" nature of academia, which means there is great pressure on all of us to publish in quantity often at the expense of quality. This is becuase it is hard to make an objective metric that measures quality (at least hard to make one that operates without a delay of 5+ years). This means that (i) there are many more papers published than there used to be (ii) academics have less time to review them than they used to do (iii) comments papers have little or no value, so academics have pretty much stopped writing them. The result is an explosion of the number of journal and a reduction in quality control."

    In other areas of science, for example cancer research, it was recently discovered that 47 of 53 "landmark" studies could not be replicated, and yet they were cited over 200 times by other researchers.

    http://blog.sethroberts.net/2012/04/03/lack-of-repeatability-of-cancer-research-the-mystery/

    Quote from a Reuters article:
    "During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 could not be replicated."

    So, while we might expect there to be some honest error in any field of research due to the limits of knowledge, an 88% non-repeatability rate is certainly shocking. Does this extend into other fields of research? It most certainly does. I'm a big fan of Gary Taubes, whose work often highlights flawed research in the area of nutrition and health. (he's written in other areas as well). For an enlightening view into this world, I suggest reading "Do We Really Know What Makes Us Healthy" or, more recently, his commentary on Harvard's latest "correlation study" showing the supposed dangers of eating red meat (which has been severely critiqued upon review of the actual data). See garytaubes.com

    So from my perspective, when I see Dikran's comment that seems to question "any paper that argues that the rise in CO2 is a natural phenomenon", it triggers my skeptical self-defense mechanism. It does not matter to me if Harvard says red meat causes increased mortality or if a stack of 500 climate research papers supports the AGW theory because it is simply naive to believe that all of those papers are correct. And, if 5% or 10% or 88% of them are flawed or not repeatable, then the skepticism is warranted.

    I think you've actually done some interesting work, but at the end of the day, as you said, we don't know.
    0 0
  23. Bad science gets published all the time, no doubt about it. The question is though whether there is any good science published at all that seriously challenges modern climate theory? The "skeptic" papers are mostly very bad science that can only get published in journals with low standards or via pal review. The perception is not helped by dubious paper sneaking through (McLean et al, Soon & Baliunas) which are then trumpeted in press statements as saying (incorrectly) more than in fact is possible.
    0 0
  24. Rufus9:

    Let's look at it another way: Michaels et al have accused the mainstream scientists of "pal review", arguing that friendly reviewers and editors make it easy for the "pals" to get papers published with the "accepted" viewpoint.

    If we accept that this is true, then we can ask the question, "What would we expect to see as a pattern in the publishing of papers by this group?" I think we'd expect to see:

    - more papers by the "pals" appearing in "friendly" journals than in "unfriendly" journals

    - a higher ratio of papers from "the pals" appearing in these journals than papers by "outsiders" (in the same journals)

    Do we see evidence of this? Yes we do, but it is in the papers from the skeptics that we see the evidence, not in the rest of the discipline. To say that this pattern exists in the authors outside of the "skeptics" group, you'd have to argue that the "pals" control virtually the entire publishing industry, and that gets you into fairy-tale conspiracy theories, not objective evidence. As is pointed out by John Mashey, feel free to search for such evidence, but it will have to be as strong as John's evidence is against the "skeptics" to have any hope of turning the pointer away from the "skeptics" and towards the mainstream consensus position.

    Are we certain? No, but there is more evidence to support Michaels et al of this behaviour than there is to support saying this about the mainstream scientists. Neither is certain, but we're a lot less uncertain about the "skeptics".

    As for the statement of Dikran's about "any paper that argues that the rise in CO2 is a natural phenomenon" - I agree with him completely. At this point in the science, to think the rise is not due to humans burning fossil fuels is pretty close to claiming that falling objects are not under the influence of gravity.

    Most of the rest of your discussion appears to be of the form "we don't know everything, and aren't always right, so I get to act as if you know nothing and you're always wrong".
    0 0
  25. Rufus9

    It's OK to be skeptical, but you shouldn't be reflexively cynical of all of science based on a few select audits. That is equally naive, in my opinion, and it produces a false equivalency. As your Begley article shows, it is possible, with due deliberation and experimentation to separate the wheat from the chaff in published science (although he oddly dissociates himself from the whole scientific process by tying himself down to non-disclosure agreements.)

    Thus, it is not because of one flawed paper that that humans are believed to be the cause of the rising CO2. There are in fact many papers using different lines of evidence that come to the same consistent conclusion. The analyses rest on basic principles that have served us well for centuries, like conservation of mass, radioactive decay, dissolution chemistry, etc. To question the role of humans in CO2 would be akin to challenging these principles. Similar statements could be made for the evidence for increasing surface temperatures, the role of CO2 as a greenhouse gas, and the role of greenhouse gasses in current warming. I think Begley would applaud this kind of reproducibility.

    The same cannot be said of skeptic papers. They typically do not address the larger body of data and the implications of their findings, choosing instead to focus almost exclusively on details that appear at first blush to cast doubt on one aspect or another of AGW. When analyzed properly or placed in context we see over and over that they do no such thing. The mistakes and the bias are often so obvious that any reasonable review process should have uncovered them. That is dana and John's point...they were forced to engage in pal review to get through peer-review. It is exactly this kind of thing that Begley is decrying in your example.
    0 0
  26. Rufus9 wrote: "So from my perspective, when I see Dikran's comment that seems to question "any paper that argues that the rise in CO2 is a natural phenomenon", it triggers my skeptical self-defense mechanism."

    As long as the same skeptical self-defense mechansim swings into action if someone claims that the rise is not anthropogenic ;o)

    Seriously, there are very good reasons why the possibility of a natural cause can be effectively ruled out completely. It is a subject I have looked into in some depth and even ended up writing a peer-reviewed comment for a journal on the subject. I'd be very happy to go over all this with you in detail on a more appropriate thread, e.g. this one.
    0 0
  27. CBD @27 - at one point a couple years ago Roy Spencer did question whether the CO2 increase was anthropogenic, on his blog. I don't know whether his position on the subject has changed since then though - I would certainly hope so. Spencer does acknowledge that the CO2 increase is causing warming though.
    0 0
  28. Whoa, I must have just engaged in time travel, responding to CBD's comment a half hour before he made it!
    0 0
  29. Dikran wrote: "Seriously, there are very good reasons why the possibility of a natural cause can be effectively ruled out completely."

    Maybe we should establish a standard of requiring that people provide a citation of at least one climate scientist making a claim before we'll entertain it. Or get a list of the published 'skeptic' scientists and check if any of them will put their name behind it.

    So far as I know there are not any who dispute that humans are responsible for the increase in atmospheric CO2 levels. Or that this increase is causing planetary warming. Even with the 'pal review' system documented in this article, there are some claims that are just too ridiculous. Lindzen, Michaels, Spencer, Christy, Pielke... they all acknowledge that human CO2 emissions are increasing global temperatures. Yet here we have Rufus professing 'skepticism'.

    Indeed, one of the few good things with the progress of the global warming 'debate' is the ever shrinking pool of contrary claims which the 'skeptic' scientists are willing to put their names behind. If we could get the online 'skeptics' to observe the complete absence of scientific support for positions like this it could wipe out about 95% of the crazy things they believe (e.g. 'undersea volcanoes are melting the Arctic ocean ice') and just leave, 'ok you are right about the physical process but we are going to assume that a powerful negative feedback will come along any day now and cancel out most of the warming'. That's still crazy, but at least it would match the actual scientific position of the remaining 'skeptics'.
    0 0
  30. Let me simplify my position.

    1. Pal review is one more factor in a long list of known flaws / issues with the peer review system.
    2. This analysis is the equivalent of observational studies in medicine - sufficient to identify a hypothesis for further research, but not proof of causation.

    I obviously picked the wrong quote to illustrate my skepticism of AGW. It is at a much more general level after reading books and blogs from both sides. However, I spend a lot more of my time reading and researching health and nutrition. I can comfortably say that "accepted science" - like saturated fat causes heart disease or salt is bad for you or red meat causes cancer or higher mortality are documented in hundreds of studies. Yet, through the wonder of the Internet, we have access to people like Gary Taubes who research these areas "settled science" and we find the evidence is not convincing. Politics, ego, placebos that beat drugs, drugs that kill people, omitted data, not publishing a report for 16 years because "... we didn't like the result", and many other issues are revealed. Something similar is going to happen in this debate if we give it time and have an open mind to accept that there are limits to our knowledge. And, there's nothing wrong with that.
    0 0
  31. I'd say blogs and books "from both sides" of any debate arent much use. You are skeptical of peer-review, but I would say that peer-review is a necessary gate-keeper. If you are not an expert in a field, then faux experts can easily pull the wool and that goes for health care in spades. Just because its peer-reviewed doesnt make it right - but if it cant get through peer-review, then its almost certainly wrong.

    Sure there are plenty of latter-day Galileo's claiming persecution but for one's I've have read in fields I know, you can see why they cant get it published. Especially if publishing outside their area of expertise. Not a few are screaming because Nature or Science wont publish them - well duh - everyone suffers that. If you think someone is hard done by with a paper, then ask to see their reviewers comments.


    That said, I would agree with 1 and 2.
    0 0
  32. Given human variation, health care is a poor analogy to climate science, and it is a red herring in this discussion.
    Statistics isn't physics.
    Some people will die from a lifetime of smoking, some won't.
    One of drugs A, B, and C will work for somebody, but doctors will have to try them one at a time to see which works.

    This is somewhat like the disciplinary-error problem I discussed at RC. Analogies can be wrong.
    0 0
  33. Rufus9 There is nothing wrong with being skeptical about mainstream scientific views. The question is "what are you going to do about it?". True skepticism involves investigating the issue to find out whether those views are reasonable, just refusing to accept them without taking the effort to look into it is merely bias. I have pointed out to you that there are good reasons supporting my view that papers purporting to show that the rise in CO2 is natural being likely to be a particularly bad, and given a link where we can discuss it. However, you have not responded to that point, which gives the impression that your apparent skepticism is actually (possibly unconscious) bias, as you do not appear to be interested in investigating the truth, merely restating your original point.

    As to your points (i) it has not been established that pal review is a substantive issue in peer review, other than in isolated incidents. (ii) Of course peer review has its flaws, but like the Churchill quote "It has been said that democracy is the worst form of government except all the others that have been tried", what would you replace it with that hasn't already been shown to be even more flawed (e.g. live public debate). (iii) observational studies can never prove causation (arguably it is fundamentally impossible to prove causal relationships without making assumptions) - so does that me we should ignore all observational studies? Of course not!

    Sure we should keep an open mind and accept there are things we don't know or are uncertain. However, it also needs to be accepted that there are things we do know with high certainty, of which I have already given an example.
    0 0
  34. CBDunkerson There have been some climatol0ogists making the claim that the rise in CO2 is natural, including Roy Spencer (as dana pointed out), Tom Segalstad, Richard Courtney, and now Murry Salby. The flaws in their arguments are pretty obvious, which just goes to show that scientists can have blind spots, just like the rest of us.

    As I said, the difference between true skepticism and stubborn bias is the willingness to investigate, rather than to simply question (questioning is only of value if you are interested in the answer).
    0 0
  35. 35, Dikran,

    I object to your label of Richard Courtney as a "climatologist". The man has no credentials or accomplishments in the field whatsoever.
    0 0
  36. 30, Rufus9,
    "...if we give it time and have an open mind to accept that there are limits to our knowledge. And, there's nothing wrong with that."
    No, there is something seriously wrong with that if:
    1. 99% of the debate and debating points are grossly inaccurate and manufactured.
    2. Most people are not educated well enough to see the difference.
    3. Those manufacturing the artificial debate are laughably unqualified to do so, and yet they are taken seriously by those who are taken in (like yourself).
    4. You really get your information from blogs and mainstream books by unqualified authors? That's where you put the weight of your faith, or from where you expect to gain a reliable understanding?
    5. Failure to act, or even delaying action, comes with serious, irreversible consequences in economic cost and human suffering.
    As to the "limits to our knowledge" nonsense... I'm afraid you fall in with the whole "man was not meant to know" or "if man had been meant to fly, God would have given him wings" crowd. You overstate your case -- a usual denial tactic, exaggerating a nonsensical position to give it apparent merit.

    But in the end your stance is vacuous. You argue "trust me, we just don't know" and "other scientists have cheated, so of course these have" and "peer review is broken, because I say it is." You speak from a position of total ignorance, raising no factual points, and conclude with the advice that "maybe it would be best to just wait and see."

    Your position is empty.
    0 0
  37. Sphaerica - you have a valid point there, he was included as he has published an article on the subject. However; having thought about it I am not sure I agree with CBDunkerson anyway - all artciles should be judged on their scientific merits (or lack of) rather than the source, so requiring a climatologist to have supported an argument is not a good criterion anyway.
    0 0
  38. John, I can, however, bring the two temporarily together in a single number. About 10 years ago, the US CDC found that of those people having small/oat cell lung cancer, 97% were or had been smokers. 97% - a familiar number for some reason. Correlation is not causation. My dad did die of lung cancer, still smoking, still claiming that the smoking killed the cancer.
    0 0
  39. re: 38 DSL
    Yes (if you've read some of my recent pieces, such as Fake science, I've studied this quite a bit.

    I'll say it again, stronger:
    1) The evidence that smoking causes {cancer, heart disease, etc} is overpowering, even if one cannot be absolutely sure any given case case was caused by smoking, that other things can cause similar effects, and that one cannot predict which smokers at age 18 will die of it.
    The statistical evidence is very powerful.

    2) The evidence for AGW is at least as strong and in some ways stronger.
    0 0
  40. [needlessly long blockquote snipped]
    *****

    [0ff-topic snipped]

    Hayek - from the Nobel speech:

    [off-topic snipped]


    I did not take up the debate on Dikram's CO2 post, but 1) I did read it 2) I did say I chose the wrong point to challenge. However, the context of the limits of knowledge, your formula looks simple enough:

    En - Un = C' - Ea

    Just 4 terms - almost anyone could just plug in the numbers and get the right answer. However, how many assumptions are buried in each of those terms? Aren't they actually a set of summations of multiple factors all with differing measurement characteristics and error rates?

    "annual emissions from all natural sources and annual natural uptake by all natural sinks."

    [argumentative snipped]

    [off-topic snipped]


    As to whether I made an appropriate case regarding flaws of peer review, I made reference to Retraction Watch and the rise in retractions. I think that case was made sufficiently previously, but here are two articles that describe the issues in more detail.

    http://io9.com/retraction-watch/

    http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?_r=1


    Finally, the complexity of the global climate and the complexity of the human body seem quite comparable to me. Certainly, this "pal study" is not complex, but even so, it is the equivalent of an observational study (as I said before). Sometimes, observational studies are useful. [off-topic snipped]

    So, while some may see AGW as the 20-fold risk of smoking, there are credible sources who do not see it that way. For example:

    [off-topic snipped]

    I hope the critique of this is neither empty or vacuous (which are synonyms).
    0 0
    Moderator Response: [DB] Please do a better job of staying on-topic to the OP of the thread and adhering to the Comments Policy (link kindly provided next to every comment input box).
  41. And you allowed Mashey's 32 and 39 and not my response.
    0 0
    Moderator Response: TC: Rufus9, your discussion was rambling, incoherent and repeatedly off topic. I completely agree with DB's assessment, and quite frankly, am surprised he did not simply delete your post because of its multiple comments policy violations.
  42. Rufus9:

    Having read through your comments on this thread, I fail to see how what you are arguing pertains to the OP.

    It appears to me that you are attempting to argue two things. If these are not reasonable summaries of your claims please clarify.
    (1) On account of documented issues in peer-review and retraction of papers, the conclusions of mainstream climatology must be considered far more provisional than they generally are.
    (2) In addition, the Earth climate is just so, well, complicated and doesn't that make for (more) uncertainty about the conclusions of mainstream climatology?

    Frankly, as far as I have seen you have not managed to substantiate either argument with direct evidence. In addition, argument (2) is off-topic for this thread. Meandering forays quoting Hayek or discussing medical science are certainly not going to do the job.

    Bringing up the latter almost certainly qualifies as a category error since the mainstream conclusions of climatology depend on our knowledge of the radiative properties of a handful of atmospheric gases, which as individual molecules or as aggregates are vastly simpler than any living organism.
    0 0
  43. Deniers keep complaining that peer-review is broken without (a) providing evidence that it is or (b) explaining why it was never a problem until climate science warned them that fossil fuels create a serious danger to economies, civilization and our current lifestyle.

    To this point the only evidence that peer-review is broken comes from the way deniers (denial scientists and denial editors alike) have been able to manipulate the system.

    Deniers began to point to issues in other aspects of science, such as drug studies, only when it became convenient as a tactic for further assaults on climate science.

    And yet I don't really see them taking other branches of science to task for it. It's as if abuse of the system in drug studies and medicine is only relevant in that it demonstrates who corrupt and evil those conniving climate scientists can be.

    Why is that?
    0 0
  44. My question is whether there is a next step after this study. It has been demonstrated that their is a potential influence or conflict, but it has not been shown what impact may have occurred.

    My suggestion is along the lines of the following:

    Document the hypothesis and develop criteria to test whether there has been an impact.
    Conduct a review of the "pal" papers
    Also randomly select and review a reasonable sample of non-pal papers
    Present the results
    0 0
  45. Rufus9 wrote "However, how many assumptions are buried in each of those terms?", this comment pretty much makes my point. If you had taken the time to understand the argument presented in my article you would know the answer to this question already (two of the terms we can measure directly, and nowhere in the argument do we assume anything about the other two - not even their numeric values). If you really want to demonstrate true skepticism, then ask questions on the appropriate thread.

    As to your post in 44, what makes you think a next step is necessary after this study? The whole point is that "skeptics" complain of pal review, yet offer no real evidence to substantiate this other than the rejection of their papers (which could potentially be explained by the possibility that their papers may be flawed and hence correctly rejected - if you are attacking mainstream science that is very likely). However it is very easy to find a clear example of pal review in the skeptic community, yet they never seem to mention that, which shows a lack of balance.

    There is no real evidence that pal review is a significant problem in peer review, there will be the occasional incident (such as this) just as there are occasional incidents of plagiarism or scientific fraud. This is because scientists are human beings with human frailties, just like in any other sphere of human activity.

    A better criticism of peer review is that many bad papers have made it through peer review, even though competent reviewers ought to have spotted them (e.g. Douglass et al, Essenhigh, ...). However those who understand the purpose of peer review know it cannot be relied upon to detect every error in every paper, so even that is not a substantive issue.

    So I challenge you again to suggest a system better than peer review, that has not already been shown to be even more flawed.
    0 0
  46. Given the context of my statements has been lost in Rufus 40, I request it be further snipped to only the few sentences and 2 links about Retraction Watch.
    0 0
  47. Has anyone tried the higher power review? Yesterday afternoon I tried putting Shakun et al. (2012) on the altar of my local church, and a beam of light shown down upon it. I tried the same thing last night with Lacis et al. (2010) and there was darkness. I am now composing emails to Lacis that will accuse him and his fellow fraudsters of fraud.

    Seriously, Rufus, what's the real problem here? Peer review is peer review. The method is not inherently corrupt, except where it requires people. People are the problem. They have all sorts of issues. Some are willing to treat their integrity as a commodity. Some just can't accept being wrong. Some want fame and glory, at least from their local acolytes. The sciences that make up climate science are not the grand repository for these folk. They're everywhere. They're in politics, organized religion, business, education, etc. Perhaps the fact that science overtly requires an individual to cast aside those sources of irrationality allows everyone to be more critical of failures to do so.

    It's hard to hide deliberate BS in science. It's possible, but very difficult. If it doesn't describe observed reality, someone will eventually start asking questions. If it's a highly scrutinized area, "eventually" could be on the order of a few days.

    Rufus, if you have trust issues, but you accept the general epistemology of science, then spend the time to do the math yourself. If it takes a few years, so what? The value of knowing the levels of honesty of others is more than enough compensation for the effort.
    0 0
  48. Rufus9 I would be more keen to continue the discussion if you were to show that you actually are interested in the answers to the points you raise. For example by either accepting that man is reposnible for the rise in CO2 or by defending your objection on the appropriate thread. Raising points and walking away from them is not acceptable behaviour in a scientific discussion.
    0 0
  49. Trust is an important issue. It is made up of several factors that can be positive or negative:

    +/- expertise of the individuals involved
    +/- methodology / measurement error
    +/- reputation of the journal
    +/- the editors influence
    +/- the peer review process
    +/- other factors

    This paper raises the issue of "pal review", which is a combination of the editor and the peer reviewers. Certainly, if we see a set of "connections" between the author, editor, and peer reviewers, we might have cause for lower trust. We could also have cause for higher trust, depending on the individuals involved. What we need when we see an "event" like this is a set of criteria that move the situation to an "incident" (Dikram called it an incident in XX). So, for example, we might list several criteria that we consistently use in making this evaluation. I'm used to calling these criteria "triggers"

    Event

    Trigger 1?
    Trigger 2?
    Trigger 3?

    If one or all of the triggers fire, then we have an Incident. From this position, we again need criteria, especially if we are going to investigate the situation and then try to determine the Impact.

    Event -> Triggers -> Incident -> Triggers -> Investigate -> Triggers -> Impact


    What I see in this story is that we have moved from Event (correlation) to Impact ("... no doubt thanks to the false media balance which gives the ~3% minority of experts who think humans aren't the dominant cause of the current climate change (and their non-expert surrogates").

    In my view, that is a leap beyond where the data supports the assertion. This seems to suffer from the ice cubes to puddle of water logic. It is easy to see ice cubes will lead to a puddle of water. However, if you have a puddle of water (e.g., the public survey results), it is much harder to work backwards to the ice cubes. It would be relatively easy to list 100 factors that could influence the survey results other than this set of authors and their papers.

    That is why I asked / suggested that additional steps could be taken to outline the approach we should take when we have suspicions or trust issues. Having a pre-set framework with tested criteria would presumably be helpful in sorting out whether Pal Review or other events are or are not a trust issue.

    As to whether there are other models, there is the Cochrane Collaboration, which is not performing the same function, but seems as though a system like this could put additional rigor in the system. Perhaps something like this exists in climate change. I have not come across it.

    http://www.cochrane.org/cochrane-reviews
    0 0
  50. Not only did the bible pass the higher power review, it was also reviewed by 12 peers ;)
    0 0

1  2  Next

You need to be logged in to post a comment. Login via the left margin or if you're new, register here.



The Consensus Project Website

TEXTBOOK

THE ESCALATOR

(free to republish)

THE DEBUNKING HANDBOOK

BOOK NOW AVAILABLE

The Scientific Guide to
Global Warming Skepticism

Smartphone Apps

iPhone
Android
Nokia

© Copyright 2014 John Cook
Home | Links | Translations | About Us | Contact Us