Imagine you have three 73-year long instrumental climate records that you want to correlate with solar activity in the last century. The instrumental records are noisy so you smooth them with a five-year moving average, and then you note that the correlation can be improved by lagging one of the records by two years behind the solar signal. Should you be excited by a correlation of -0.47, the strongest of the three correlations?

The authors, Czymzik et al, of a new paper in Climate of the Past Discussions were. They claim that the correlation of -0.47 has a probability of less than 0.0001 of occurring by chance if there is no relationship between the instrumental record and solar activity. A correlation this strong and a p-value so low would be impressive evidence of the importance of solar forcing in the last century, except that they are probably data processing artefacts.

Lets start by looking at the effect of smoothing the data with the moving average. I’m going to take two time-series of 73 random numbers, smooth them with a five-point moving average, and then find the correlation between the two time-series. I’m going to repeat this many times to find the distribution of correlations expected by chance with this data processing.

library(gtools)
x<-rnorm(73)
x5<-running(x, width=5, fun=mean)
res<<-replicate(10000,{
y<-rnorm(73)
y5<-running(y, width=5, fun=mean)
cor(x5,y5)
})
quantile(res, prob=c(0.025, 0.975))
mean(abs(res)>0.47)

The 95% significance interval of the correlation of the time series processed with a moving average is ±0.375 (by comparison, the 95% significance interval on the original time series would be ±0.230). The observed correlation of -0.47 has a p-value of about 0.01, two orders of magnitude less extreme than reported by the paper, but still significant at p = 0.05. The smoothing has induced strong autocorrelation in the data, so the test that the authors use, which assumes that the observations are independent, is extremely liberal. By ignoring the induced autocorrelation, assuming they still have 73-2 degrees of freedom, the authors have generated a seriously misleading idea of how strong their evidence is. And I’ve not finished yet.

Next I want to consider how allowing for different lags (correlating this year’s climate data with solar data from the last year (lag 1) or the year before (lag 2)) affects the p-value. Allowing for lags is not absurd – a recent paper in Environmental Research Letters found a three-year lag between solar activity and the North Atlantic Oscillation in a climate model run. Czymzik et al test for correlations at lags from -5 to +5. Making the assumption that only lags 0-3 are physically reasonable, I modified the above code to output the maximum correlation at these lags. This is to be generous, the results would be much worse if I considered all 11 lags.

res2<-replicate(10000,{
y<-rnorm(73)
y5<-running(y, width=5, fun=mean)
c0<-cor(x5,y5)
c1<-cor(x5[-1],y5[-(73-4)])
c2<-cor(x5[-(1:2)],y5[-((72:73)-4)])
c3<-cor(x5[-(1:3)],y5[-((71:73)-4)])
c(c0, c1, c2, c3)[which.max(abs(c(c0, c1, c2,c3)))]
})
quantile(res2, prob=c(0.025, 0.975))
mean(abs(res2)&gt;0.47)

Now the 95% significance level is at ±0.447 and the p-value of the best correlation reported is 0.035. The other two correlations are not significant (p > 0.1). So rather than having three very highly significant correlations, Czymzik et al have one marginally significant correlation.

But even this is optimistic, we also ought to consider multiple testing (three instrumental records, two versions of each) and the effect of autocorrelation in the raw data. If we take these into account, it is unlikely that any of the correlations with instrumental records reported by Czymzik et al are statistically significant.

The second part of Czymzik et al reports the correlation between flood frequency reconstructed from the sediments of the lake Ammersee in southern Germany and cosmogenic isotopes to try to infer the role of solar activity. Czymzik et al finds high correlations: 0.36 with ^{14}C production rate, and 0.45 with ^{10}Be concentations. The significance of these correlations is almost certainly over-estimated but it is unclear how exactly they were calculated: I don’t dispute that they are significant. I do find it strange that the authors have chosen to correlate their flood frequency record with the isotope production rate/concentration rather than estimates of total solar irradiation (Steinhilber et al 2012) or solar modulation (Muscheler et al 2007). The problem with the isotope data is that concentration and production rates are affected by both the solar magnetic field (which varies with solar activity) and the geomagnetic field. The geomagnetic field mainly varies on long timescales, giving trends in the isotope data that were removed by Steinhilber et al and Muscheler et al. I cannot think of a good reason not to use these reconstructions of solar activity (but I can think of a bad reason).

Czymzik et al is in open review at Climate of the Past Discussions. I’ll keep an eye on the reviewers’ comments to check they cover the points I raise here (and some problems with the spectral analysis).

There are some good papers looking for evidence of solar activity on palaeoclimate records. This is not one of them.