The dinocyst calibration set has an extremely uneven distribution of sites along the sea-ice gradient. In the 1171-observation calibration set (the latest 1492-observation calibration set is not yet online), just over half the sites have zero sea-ice, so the remainder of the gradient has a low density of sites in comparison. This distribution of sites will bias estimates of transfer function uncertainty. Telford and Birks (2011) showed that sites in the well-sampled portions of the gradient have many potential analogues and so low cross-validation errors. The converse occurs in the poorly sampled portions of the gradient. However, because there are few sites in the poorly sampled portions, the average error is biased low. This bias is worst for the Modern Analogue Technique (MAT), less severe for weighted averaging, and least severe for maximum likelihood regression and calibration.
This bias may not be easy to avoid as the distribution of sites in the calibration set may reflect the underlying distribution of the environment, but it is important to explore the magnitude of the bias. This can be done by dividing the environmental gradient into segments, and calculating the uncertainty of cross-validated estimates in each segment (akin to the calculation of maximum bias).
I’ve analysed the 1171-observation calibration from Radi and de Vernal (2008). The application of MAT follows the usual dinocyst procedure (Euclidean distance on log-transformed dinocyst assemblage data expressed in per mil; five distance-weighted analogues. I haven’t used the threshold for accepting analogues – it probably makes little difference).
The R code I used is here.
The root mean squared error of prediction (RMSEP) is 1.05 months of sea ice – pretty impressive. Dividing the gradient into ten segments and calculating the RMSEP in each segment shows that uncertainty is not constant along the gradient. It varies from about 0.6 months in the most densely sampled portion of the gradient to over 2 months in the least densely sampled portions.
The overall RMSEP of 1.05 months is of little relevance. More important for any reconstruction is the uncertainty at different points on the environmental gradient. For reconstructions from sites where there was little or no sea ice, the overall RMSEP is much too high. Conversely, for reconstructions from sites where there was several months of sea ice, the overall RMSEP is much too low. Given that most of the reconstructions in de Vernal et al (2013) show several months of sea ice, I think it reasonable to conclude that the uncertainty on the reconstructions has been underestimated by 50-100% (NB – this is without considering spatial autocorrelation). I don’t know how high the uncertainly can be before the reconstructions have no utility.
One obvious tactic to deal with this problem is to remove some of the sites at the warm end of the gradient. All those sites in the Mediterranean Sea, for example, have zero sea ice and the cross-validation estimates are perfect. So these sites improve the apparent transfer function performance, but never provide any suitable analogues for sites near the sea-ice margin. Removing the 397 sites with a winter SST greater than 7°C, the overall RMSEP increases to 1.28 months. The distribution of sites is still severely uneven, but less than it was for the full dataset, so the bias in this uncertainty estimate is reduced, but it is still biased.