## On Statistical Significance and Confidence

#### Posted on 11 August 2010 by Alden Griffith

**Guest post by Alden Griffith from Fool Me Once**

My previous post, “Has Global Warming Stopped?”, was followed by several (well-meaning) comments on the meaning of statistical significance and confidence. Specifically, there was concern about the way that I stated that we have 92% confidence that the HadCRU temperature trend from 1995 to 2009 is positive. The technical statistical interpretation of the 92% confidence interval is this: "if we could resample temperatures independently over and over, we would expect the confidence intervals to contain the true slope 92% of the time." Obviously, this is awkward to understand without a background in statistics, so I used a simpler phrasing. Please note that this does not change the conclusions of my previous post at all. However, in hindsight I see that this attempt at simplification led to some confusion about statistical significance, which I will try to clear up now.

So let’s think about the temperature data from 1995 to 2009 and what the statistical test associated with the linear regression really does (it's best to have already read my previous post). The procedure first fits a line through the data (the “linear model”) such that the deviations of the points from this line are minimized, i.e. the good old line of best fit. This line has two parameters that can be estimated, an intercept and a slope. The slope of the line is really what matters for our purposes here: does temperature vary with time in some manner (in this case the best fit is positive), or is there actually no relationship (i.e. the slope is zero)?

*Figure 1:* Example of the null hypothesis (blue) and the alternative hypothesis (red) for the 1995-2009 temperature trend.

Looking at Figure 1, we have two hypotheses regarding the relationship between temperature and time: 1) there is no relationship and the slope is zero (blue line), or 2) there is a relationship and the slope is not zero (red line). The first is known as the “null hypothesis” and the second is known as the “alternative hypothesis”. Classical statistics starts with the null hypothesis as being true and works from there. Based on the data, should we accept that the null hypothesis is indeed true or should we reject it in favor of the alternative hypothesis?

Thus the statistical test asks: *what is the probability of observing the temperature data that we did, given that the null hypothesis is true*?

In the case of the HadCRU temperatures from 1995 to 2009, the statistical test reveals a probability of 7.6%. Thus there’s a 7.6% probability that we should have observed the temperatures that we did if temperatures are not actually rising. Confusing, I know… This is why I had inverted 7.6% to 92.4% to make it fit more in line with Phil Jones’ use of “95% significance level”.

Essentially, the lower the probability, the more we are compelled to reject the null hypothesis (no temperature trend) in favor of the alternative hypothesis (yes temperature trend). By convention, “statistical significance” is usually set at 5% (I had inverted this to 95% in my post). Anything below is considered significant while anything above is considered nonsignificant. The problem that I was trying to point out is that this is not a magic number, and that it would be foolish to strongly conclude anything when the test yields a relatively low, but “nonsignificant” probability of 7.6%. And more importantly, that looking at the statistical significance of 15 years of temperature data is not the appropriate way to examine whether global warming has stopped (cyclical factors like El Niño are likely to dominate over this short time period).

Ok, so where do we go from here, and how do we take the “7.6% probability of observing the temperatures that we did if temperatures are not actually rising” and convert it into something that can be more readily understood? You might first think that perhaps we have the whole thing backwards and that really we should be asking: “what is the probability that the *hypothesis is true* given the data that we observed?” and not the other way around. Enter the Bayesians!

Bayesian statistics is a fundamentally different approach that certainly has one thing going for it: it’s not completely backwards from the way most people think! (There are many other touted benefits that Bayesians will gladly put forth as well.) When using Bayesian statistics to examine the slope of the 1995-2009 temperature trend line, we can actually get a more-or-less straightforward probability that the slope is positive. That probability? 92%^{1}. So after all this, I believe that one can conclude (based on this analysis) that there is a 92% probability that the temperature trend for the last 15 years is positive.

While this whole discussion comes from one specific issue involving one specific dataset, I believe that it really stems from the larger issue of how to effectively communicate science to the public. Can we get around our jargon? Should we embrace it? Should we avoid it when it doesn’t matter? All thoughts are welcome…

^{1}To be specific, 92% is the largest credible interval that does not contain zero. For those of you with a statistical background, we’re conservatively assuming a non-informative prior.

tobyjoyceat 00:47 AM on 14 August, 2010Berényi Péterat 01:01 AM on 14 August, 2010tobyjoyce at 00:47 AM on 14 August, 2010The odd shape of the distribution could probably be approximated by a mixture of GaussiansYou would need a whole lot of them. The tail seems to decrease slower than exponentially.tobyjoyceat 03:12 AM on 14 August, 2010Berényi Péterat 03:51 AM on 14 August, 2010tobyjoyce at 03:12 AM on 14 August, 2010there is software that will fit as many as you likeI happen to know the algorithm itself, which is pretty straightforward. But what's the point of this exercise? There is no unique solution to this problem anyway. And tailsdomatter. Those are the parts of weather that can getcostly(both in terms of money and human lives).batsvenssonat 04:03 AM on 14 August, 2010barryat 13:56 PM on 13 August, 2010 Why are all the lines crossing at the same(?) point at about 1935?tobyjoyceat 05:23 AM on 14 August, 2010Dikran Marsupialat 09:03 AM on 14 August, 2010kdkdat 09:05 AM on 14 August, 2010"Yes, but you have to get rid of the assumption of normality. Temperature anomaly distribution does get more regular with increasing sample size, but it never converges to a Gaussian."From a theoretical perspective this is an important consideration, but from apracticalperspective it often makes little difference. It is of course quite reasonable to use non-parametric methods whenever you think it's sensible, but NP methods will always have less power for a normal-enough dataset. So it's important to consider whether there's any practical benefit from eschewing the normal distribution. In the case of the two graphs you posted in this discussion, they're what we would consider close enough to normal as makes no odds. (I've spent some time working on this as part of my day job, to satisfy myself empirically of when NP and P approaches are best)kdkdat 11:03 AM on 14 August, 2010The Skeptical Chymistat 11:28 AM on 14 August, 2010Maarten Ambaumat 20:39 PM on 3 November, 2010Journal of Climatewhich analyses the precise problem, using Bayesian statistics. In a nutshell: significance tests are generally used to quantify the validity of some hypothesis while it is is nothing like it. In fact, the significance statistic is largely irrelevant. Unfortunately, misuse of significance tests is widespread. Not only climate science suffers, also economics, medical science, social science, psychology, biology. I am afraid that significance tests have muddied the waters of several climate papers and there is a real communication problem here. We need to accept that statistics alone cannot usefully quantify the truth of some hypothesis. And significance tests are possibly the worst in this respect. For more details read Significance Tests in Climate Science.