Climate Science Glossary

Term Lookup

Enter a term in the search box to find its definition.


Use the controls in the far right panel to increase or decrease the number of terms automatically displayed (or to completely turn that feature off).

Term Lookup


All IPCC definitions taken from Climate Change 2007: The Physical Science Basis. Working Group I Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Annex I, Glossary, pp. 941-954. Cambridge University Press.

Home Arguments Software Resources Comments The Consensus Project Translations About Donate

Twitter Facebook YouTube Pinterest

RSS Posts RSS Comments Email Subscribe

Climate's changed before
It's the sun
It's not bad
There is no consensus
It's cooling
Models are unreliable
Temp record is unreliable
Animals and plants can adapt
It hasn't warmed since 1998
Antarctica is gaining ice
View All Arguments...

Keep me logged in
New? Register here
Forgot your password?

Latest Posts


Climate Hustle

The F13 files, part 3 - the reference list analysis and other problems

Posted on 24 October 2017 by Ari Jokimäki

Elsevier's journal Renewable and Sustainable Energy Reviews published a paper in 2013 (Florides et al. 2013, "F13" from now on) which we found problematic. We analysed the paper and communicated our findings to Elsevier. Our main findings were that much of F13 text was copy/pasted from other sources without proper attribution and that F13 contained many false claims. In this series of posts I'll go through the problems in F13 and in Elsevier actions. There are four posts:

Part 3 - the reference list analysis and other problems

In addition to content analysis, we analysed F13’s reference list, as that can offer strong quality indicators. In this second part of our analysis, we checked each reference and recorded its properties:

  • Type of article (journal article / report / website / etc.).
  • Status of peer-review. In addition to journal articles, we included IPCC reports as peer-reviewed. In a few cases, F13 cited a website while the document in question had also been published in a scientific journal. These we also included as peer-reviewed.
  • Position on anthropogenic global warming or related issues. References were categorized as supporting (a) the position of mainstream climate science, (b) alternative theories, or (c) contrarian position.

Finally, we commented on general problems and some specific problems elsewhere in F13. These we also show in this post.

Results of the reference list analysis

The references can be categorized in three different ways.

First, by source (shown as a pie chart below), about 40% of the F13 references are scientific journal articles, 19% are scientific reports, 25% are websites, 12% are books or book chapters, and 4% are other articles.

Second, percentage of non-peer-reviewed material in the F13 reference list is 40%.

Third (shown as a pie chart in the beginning of this post), 62% of the references follow mainstream climate science, aligned with IPCC, but 38% support alternative or contrarian views. Cook et al (2013) [2] found that only 0.7% of their sample rejected AGW, so F13 references them 50 times more than one would expect of an unbiased sample. This is problematic in a review article, which generally aims to acquaint non-experts with the field's state of the art, but this seriously misrepresents it.

Analysis of the reference list is presented at Appendix B below.

Some other issues with F13

Although a somewhat minor point, F13 is a review on the climatic effects of CO2 and the Sun, and yet F13 has a chapter on “CO2 concentration and life” (F13 Chapter 2.5), which seems to be out of place.

F13 discussion on the climatic effects of the Sun is not covered by our detailed paragraph-by-paragraph analysis, but seems to follow similar paths as the CO2 related discussion. Alternative and contrarian viewpoints are emphasized and handled uncritically in F13 while ignoring all research showing the weaknesses of those viewpoints. F13 also ignores the evidence that points to weak solar influence to climate change of recent decades.

The hypothesized cosmic ray - climate connection is worth closer inspection. F13 discuss that cosmic rays might have a strong effect on Earth's climate. Dozens of papers have been published on the issue and most find that the effect on Earth's climate is negligible [e.g. 4, 5, 6, and 7]. However, F13 discuss this issue as if strong cosmic ray - climate connection might be on the verge of wide acceptance. The F13 discussion is an example of cherry-picking; they only cite sources in favor of the hypothesis, with no mention of papers reporting contrary results.

When F13 discuss cosmic ray effects on climate, F13 cite references using the name of the institution rather than author names, which seem like independent confirmations, but are really just different texts by the same authors.

F13 also mention CERN Cloud project that studies the cosmic ray issue, but F13 fail to mention that the first results had already been published in 2010 [8] and they were not favorable to the cosmic ray hypothesis. In support of CERN results and against the cosmic ray hypothesis, Kulmala et al. [9] have reported results from atmospheric observations of nucleation events.


About 40% the references are non-peer-reviewed, 38% of them take alternative or contrarian positions on AGW, clearly higher than would be expected in an unbiased review.

The sections of F13 we didn't analyze thoroughly seem to contain similar problems to those sections we analyzed.

Below are shown the full reference list analysis, acknowledgements, and references in our (my and John Mashey's) comment paper as sent to Elsevier. Almost everything in this post is also directly from our comment paper.

Appendix B. The reference list analysis

Table 1. The reference list analysis. Column 1 is the reference number as given by F13. Column 2 is the type of reference. Column 3 is the status of peer-review. Column 4 is the position the reference takes to anthropogenic global warming.

F13 ref.   Type Peer-review  Position 
 1  Review article  Yes  Contrarian
 2  Scientific website    Mainstream
 3  Journal article  Yes (F13 ref to website)  Alternative
 4  Report  Yes  Mainstream
 5  Book chapter    Contrarian
 6  Website    Mainstream
 7  Report  Yes  Mainstream
 8  Report  Yes  Mainstream
 9  Report  Yes  Mainstream
10  Report  Yes  Mainstream
11  Report  Yes  Mainstream
12  Report  Yes  Mainstream
13  Report  Yes  Mainstream
14  Book chapter    Contrarian
15  Website    Contrarian
16  Journal article  Yes  Alternative
17  Report  Yes  Mainstream
18  Book chapter    Contrarian
19  Journal article  Yes  Contrarian
20  Journal article  Yes  Mainstream
21  Journal article  Yes  Mainstream
22  Journal article  Yes  Mainstream
23  Journal article  Yes  Mainstream
24  Journal article  Yes  Mainstream
25  Journal article  Yes  Mainstream
26  Journal article  Yes  Alternative
27  Book chapter    Mainstream
28  Book chapter    Contrarian
29  Journal article  Yes  Contrarian
30  Website    Mainstream
31  Report  Yes  Mainstream
32  Website    Contrarian
33  Book chapter    Contrarian
34  Website    Mainstream
35  Website    Mainstream
36  Journal article  Yes  Mainstream
37  Website    Mainstream
38  Journal article  Yes  Mainstream
39  Journal article  Yes  Alternative
40  Website    Mainstream
41  Website    Mainstream
42  Website    Mainstream
43  Journal article  Yes  Alternative
44  Journal article  Yes  Mainstream
45  Journal article  Yes (F13 ref to arXiv)  Alternative
46  Journal article  Yes  Contrarian
47  Website    Mainstream
48  Journal article  Yes  Mainstream
49  Article    Contrarian
50  Conference article    Alternative
51  Journal article  Yes  Alternative
52  Website    Mainstream


Authors wish to thank Peter Jacobs, Kevin Cowtan, Daniel Bailey, Gavin Cawley, Dana Nuccitelli, Bärbel Winkler, Paul W., and Deep Climate for the discussions and help relating to this work.


[1] Florides GA, Christodoulides P, Messaritis V, Reviewing the effect of CO 2 and the sun on global climate, Renewable and Sustainable Energy Reviews 2013;26:63951. doi:10.1016/j.rser.2013.05.062.

[2] Cook J, Nuccitelli D, Green SA, Richardson M, Winkler B, Painting R, et al ., Quantifying the consensus on anthropogenic global warming in the scientific literature, Environ. Res. Lett. 2013;8(2). doi:10.1088/17489326/8/2/024024.

[3] IPCC AR4, Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt KB, et al ., editors. Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA; 2007.

[4] Laken BA, Pallé E, Calogovic J, Dunne EM, A cosmic ray-climate link and cloud observations, Journal of Space Weather and Space Climate 2012;2. doi: 10.1051/swsc/2012018.

[5] Sloan T, Wolfendale AW, The contribution of cosmic rays to global warming, Journal of Atmospheric and Solar-Terrestrial Physics 2011;73(16): 23525. doi:10.1016/j.jastp.2011.07.013.

[6] Kristjánsson JE, Stjern CW, Stordal1 F, Fjæraa AM, Myhre G, Jónasson K, Cosmic rays, cloud condensation nuclei and clouds – a reassessment using MODIS data, Atmos. Chem. Phys. 2008;8:737387. doi:10.5194/acp873732008.

[7] Udelhofen PM, Cess RD, Cloud cover variations over the United States: An influence of cosmic rays or solar variability? Geophysical Research Letters 2001;28(13):261720. doi:10.1029/2000GL012659.

[8] Duplissy J, Enghoff MB, Aplin KL, Arnold F, Aufmhoff H, Avngaard M, et al. , Results from the CERN pilot CLOUD experiment, Atmos. Chem. Phys. 2010;10:163547. doi:10.5194/acp1016352010.

[9] Kulmala M, Riipinen I, Nieminen T, Hulkkonen M, Sogacheva L, Manninen HE, et al., Atmospheric data over a solar cycle: no connection between galactic cosmic rays and new particle formation, Atmos. Chem. Phys. 2010;10:188598. doi:10.5194/acp1018852010.
[10] Visser ME, Keeping up with a warming world; assessing the rate of adaptation to climate change, Proceedings B of the Royal Society 2008;275(1635):64959. doi:10.1098/rspb.2007.0997.

[11] Diffenbaugh NS, Field CB, Changes in Ecologically Critical Terrestrial Climate Conditions, Science 2013;341(6145):486492. doi: 10.1126/science.1237123.

[12] McElwain JC, Wagner PJ, Hesselbo SP, Fossil Plant Relative Abundances Indicate Sudden Loss of Late Triassic Biodiversity in East Greenland, Science 2009;324(5934):15546. doi: 10.1126/science.1171706.
[13] Twitchett RJ, The palaeoclimatology, palaeoecology and palaeoenvironmental analysis of mass extinction events, Palaeogeography, Palaeoclimatology, Palaeoecology 2006;232(24):190213. doi:10.1016/j.palaeo.2005.05.019.

[14] Haug GH, Günther D, Peterson LC, Sigman DM, Hughen KA, Aeschlimann B, Climate and the Collapse of Maya Civilization, Science 2003;299(5613):17315. doi: 10.1126/science.1080444.

[15] Yu S, Zhu C, Song J, Qu W, Role of climate in the rise and fall of Neolithic cultures on the Yangtze Delta, Boreas 2000;29(2):15765. doi:10.1111/j.15023885.2000.tb01208.x.

[16] Plass GN, The Carbon Dioxide Theory of Climatic Change. Tellus 1956;8:140–54. doi: 10.1111/j.21533490.1956.tb01206.x.

[17] IPCC (Intergovernmental Panel on Climate Change), Principles Governing IPCC Work 1998 [cited 2015 Oct 22]. Available from .

[18] Svensmark H, Evidence of nearby supernovae affecting life on Earth, MNRAS 2012;423(2):123453. doi: 10.1111/j.13652966.2012.20953.x.

[19] Overholt AC, Melott AL, Pohl M, The Astrophysical Journal Letters 2009;705. doi:10.1088/0004637X/705/2/L101.

[20] Ruddiman WF (2006), Orbital changes and climate, Quaternary Science Reviews 2006;25(23–24):3092–112. doi:10.1016/j.quascirev.2006.09.001.

[21] Hartmann DL, Klein Tank AMG, Rusticucci M, Alexander LV, Brönnimann Charabi SY, et al., Observations: Atmosphere and Surface. In: Stocker TF, Qin D., Plattner GK, Tignor M, Allen SK, Boschung J, et al . editors. Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA 2013.

[22] Hegerl GC, Zwiers FW, Braconnot P, Gillett NP, Luo Y, Marengo Orsini JA, et al., Understanding and Attributing Climate Change. In: Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt KB, et al ., editors. Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA; 2007.

[23] Caillon N, Severinghaus JP, Jouzel1 J, Barnola JM, Kang J, Lipenkov VY, Timing of Atmospheric CO2 and Antarctic Temperature Changes Across Termination III, Science 2003;299(5613):172831. doi:10.1126/science.1078758.

[24] Shakun JD, Clark PU, He F, Marcott SA, Mix AC, Liu Z, et al., Global warming preceded by increasing carbon dioxide concentrations during the last deglaciation, Nature 2012;484:4954. doi:10.1038/nature10915.

[25] Kürschner WM, Leaf sensor for CO 2 in deep time, Nature 2001;411:247248. doi:10.1038/35077181.

[26] Retallack GJ, A 300-million-year record of atmospheric carbon dioxide from fossil plant cuticles, Nature 2001;411:287290. doi:10.1038/35077041.

[27] Crowley TJ, Berner RA, CO 2 and Climate Change, Science 2001;292(5518):8702. doi: 10.1126/science.1061664.

[28] Royer DL, Berner RA, Montañez IP, Tabor NJ, Beerling DJ, CO2 as a primary driver of Phanerozoic climate, GSA Today 2004;14(3):410.

[29] Royer DL, CO2-forced climate thresholds during the Phanerozoic, Geochimica et Cosmochimica Acta 2006;70(23):5665–75. doi:10.1016/j.gca.2005.11.031.

[30] Came RE, Eiler JM, Veizer J, Azmy K, Brand U, Weidman CR, Coupling of surface temperatures and atmospheric CO2 concentrations during the Palaeozoic era, Nature 2007;449:198201. doi:10.1038/nature06085.

[31] Trenberth KE, Fasullo JT, O'Dell C, Wong T, Relationships between tropical sea surface temperature and top-of-atmosphere radiation, Geophys. Res. Lett. 2010:37(3). doi:10.1029/2009GL042314.

[32] Idso SB, The CO2 greenhouse effect on Mars, Earth, and Venus, Science of The Total Environment 1988;77(23):291294. doi:10.1016/00489697(88)900654.



0 0

Bookmark and Share Printable Version  |  Link to this page


Comments 1 to 1:

  1. In the comments for part 1 of this series it was suggested to just have one comment thread as there is bound to be quite some overlap. In the spirit of this suggestion, here is the link to that thread:

    0 0

You need to be logged in to post a comment. Login via the left margin or if you're new, register here.

The Consensus Project Website


(free to republish)

Smartphone Apps


© Copyright 2017 John Cook
Home | Links | Translations | About Us | Contact Us