You cannot smooth your way to significance

Imagine you have three 73-year long instrumental climate records that you want to correlate with solar activity in the last century. The instrumental records are noisy so you smooth them with a five-year moving average, and then you note that the correlation can be improved by lagging one of the records by two years behind the solar signal. Should you be excited by a correlation of -0.47, the strongest of the three correlations?

The authors, Czymzik et al, of a new paper in Climate of the Past Discussions were. They claim that the correlation of -0.47 has a probability of less than 0.0001 of occurring by chance if there is no relationship between the instrumental record and solar activity. A correlation this strong and a p-value so low would be impressive evidence of the importance of solar forcing in the last century, except that they are probably data processing artefacts.

Lets start by looking at the effect of smoothing the data with the moving average. I’m going to take two time-series of 73 random numbers, smooth them with a five-point moving average, and then find the correlation between the two time-series. I’m going to repeat this many times to find the distribution of correlations expected by chance with this data processing.

library(gtools)

x<-rnorm(73)
x5<-running(x, width=5, fun=mean)

res<<-replicate(10000,{
  y<-rnorm(73)
  y5<-running(y, width=5, fun=mean)
 
  cor(x5,y5)

}) 

quantile(res, prob=c(0.025, 0.975)) 

mean(abs(res)>0.47)

The 95% significance interval of the correlation of the time series processed with a moving average is ±0.375 (by comparison, the 95% significance interval on the original time series would be ±0.230). The observed correlation of -0.47 has a p-value of about 0.01, two orders of magnitude less extreme than reported by the paper, but still significant at p = 0.05. The smoothing has induced strong autocorrelation in the data, so the test that the authors use, which assumes that the observations are independent, is extremely liberal. By ignoring the induced autocorrelation, assuming they still have 73-2 degrees of freedom, the authors have generated a seriously misleading idea of how strong their evidence is. And I’ve not finished yet.

Next I want to consider how allowing for different lags (correlating this year’s climate data with solar data from the last year (lag 1) or the year before (lag 2)) affects the p-value. Allowing for lags is not absurd – a recent paper in Environmental Research Letters found a three-year lag between solar activity and the North Atlantic Oscillation in a climate model run. Czymzik et al test for correlations at lags from -5 to +5. Making the assumption that only lags 0-3 are physically reasonable, I modified the above code to output the maximum correlation at these lags. This is to be generous, the results would be much worse if I considered all 11 lags.

res2<-replicate(10000,{
  y<-rnorm(73)
  y5<-running(y, width=5, fun=mean)
  c0<-cor(x5,y5)
  c1<-cor(x5[-1],y5[-(73-4)])
  c2<-cor(x5[-(1:2)],y5[-((72:73)-4)])
  c3<-cor(x5[-(1:3)],y5[-((71:73)-4)]) 
  c(c0, c1, c2, c3)[which.max(abs(c(c0, c1, c2,c3)))]

 })

quantile(res2, prob=c(0.025, 0.975))

mean(abs(res2)&amp;gt;0.47)

Now the 95% significance level is at ±0.447 and the p-value of the best correlation reported is 0.035. The other two correlations are not significant (p > 0.1). So rather than having three very highly significant correlations, Czymzik et al have one marginally significant correlation.

But even this is optimistic, we also ought to consider multiple testing (three instrumental records, two versions of each) and the effect of autocorrelation in the raw data. If we take these into account, it is unlikely that any of the correlations with instrumental records reported by Czymzik et al are statistically significant.

The second part of Czymzik et al reports the correlation between flood frequency reconstructed from the sediments of the lake Ammersee in southern Germany and cosmogenic isotopes to try to infer the role of solar activity. Czymzik et al finds high correlations: 0.36 with 14C production rate, and 0.45 with 10Be concentations. The significance of these correlations is almost certainly over-estimated but it is unclear how exactly they were calculated: I don’t dispute that they are significant. I do find it strange that the authors have chosen to correlate their flood frequency record with the isotope production rate/concentration rather than estimates of total solar irradiation (Steinhilber et al 2012) or solar modulation (Muscheler et al 2007). The problem with the isotope data is that concentration and production rates are affected by both the solar magnetic field (which varies with solar activity) and the geomagnetic field. The geomagnetic field mainly varies on long timescales, giving trends in the isotope data that were removed by Steinhilber et al and Muscheler et al. I cannot think of a good reason not to use these reconstructions of solar activity (but I can think of a bad reason).

Czymzik et al is in open review at Climate of the Past Discussions. I’ll keep an eye on the reviewers’ comments to check they cover the points I raise here (and some problems with the spectral analysis).

There are some good papers looking for evidence of solar activity on palaeoclimate records. This is not one of them.

Advertisements

About richard telford

Ecologist with interests in quantitative methods and palaeoenvironments
This entry was posted in Peer reviewed literature, solar variability and tagged . Bookmark the permalink.

4 Responses to You cannot smooth your way to significance

  1. Magma says:

    Apart from everything else, the three meteorological records *should* be obviously correlated (river flow/flood levels and low air pressure and heavy rainfall events within the watershed). The methodology of counting the number of events over the 90th and 95th percentile also seems designed to tease out relationships that might not be there at all, and the 5-year running average for a 73 year rainfall/river level time series seems unnecessary. The “one to three year lag” between solar activity and flood events seems like motivated hand-waving.

  2. tallbloke says:

    The first thing to do before making any stats tests is to consider how the physical mechanism may work. If the relatively high solar activity of the C20th did have an effect on temperature then it was by warming the oceans. Water has a high heat capacity, so Ocean Heat Content (and thence SST) wouldn’t be expected to react instantaneously to solar variation. So we need to low pass filter or integrate solar data to mimic the thermal inertia of the oceans.

    Sunspot number (SSN) is a good proxy for total solar irradiance (TSI). Over the period of record since 1749, the average sunspot number is 40. This is also the number found when averaging the SSN over a period where SST didn’t vary much. Integrating SSN as a cumulative total departing from this ‘equilibrium’ value reveals a curve which shows a slight fall from 1880 to 1934, and then a rise to 2003. When combined with a suitably phased ~60 year sinusoid representing the Atrlantic multidecadal oscillation (AMO), something close to the 5 year smoothed surface temperature record is obtained.

    I took this idea a little further with by combining AMO, integrated SSN, SOI and LnCO2 (at a suitable value), producing a match between model and HADSST with an R^2 of ~0.8 for MONTHLY values since 1875 and R^2=~0.9. for more accurate data since 1960

    I can make the spreadsheet available to interested parties.

    One of the many problems with discerning the solar effect on surface temperature is that the solar derived energy building up in the Pacific Warm Pool (PWP) between El Nino events is hidden from surface temperature datasets. Big El Nino’s tend to occur at solar minimum, and so are in anticorrelation to the solar signal on SST. This flattens the apparent magnitude of the solar effect.

    There are on average three el nino events per solar cycle, so by smoothing the temperature data at 37 months, and detrending the temperature data to match the unintegrated solar data, a better solar-surface temperature correlation appears, though of small magnitude and varying phase due to the ENSO effect outlined above.

    • WebHubTelescope says:

      Tallbloke, No one cares whut you and your insane climate posse are up to.
      Laughable watching you start the time-series at 1953. Why not 1950? Don’t take us for chumps, so go back to where your yes-men can agree with you.

  3. tallbloke says:

    The correlation pattern gets disturbed for a decade near the peak of the AMO, in 1880, 1940, and 1998, when ‘super-el nino’ occurs. That makes the relationship harder to spot, though it is still there. How it’ll play out as we descend further into the grand solar minimum will be interesting to observe. OHC is at a historic high, so there may be another biggish El Nino around 2020 or 2025. How much that will affect global T given the increase in OLR is an open question. The current El Nino has an impressive high SST tongue jutting away from the Peruvian coast, but global temps haven’t shot up yet, though they may over the next couple of months. Time will tell.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s