Odd ordination of the day

Your challenge is to identify half a dozen unexpected features in this Detrended Constrained Correspondence Analysis from Larocque-Tobler (2010). Bonus points if you can interpret the last two sentences of the caption.


DCCA analysis of the 101-lake training set (empty circles) constrained to temperature (July Temp), depth and DOC (arrows). The Lake Silvaplana chironomid assemblages were added passively to this analysis. A 20-point moving average was made of the axis scores of the samples (thick line). Note that the two axes were shortened to represent the changes in the fossil records.

Posted in Peer reviewed literature, Uncategorized | Tagged , , | Leave a comment

Air temperature, lake temperature

I’m interested in lake temperatures because I want to go swimming. I’m also interested because some palaeolimnological proxies, for example chironomids, are probably most sensitive to water temperature but are calibrated against and used to reconstruct air temperature. The stronger the relationship between air temperature and water temperature, the better the reconstructions are going to be.

I’ve found Sharma et al (2015), a global database of lake surface temperatures collected in situ and by satellite methods from 1985–2009. The database includes 286 summer lake surface time series with more 10 years.

The relationship between mean air temperature and mean water temperature is, as one might expect, strong (r = 0.91).



That’s good, but I’m more interested in the year-by-year correlations between lake and air temperature.

The median Pearson correlation is 0.69 with a wide spread. This is lower than I had expected, but the database includes metre-deep and kilometre-deep lakes; 0.03km2 ponds and the 380 thousand km2 Caspian Sea; and tropical lakes and Arctic lakes. Perhaps surface area, depth and latitude are related to the correlation.



Latitude seems to be the most important predictor of the correlation between air and water temperature. This is probably because inter-annual air temperature variability is lower in the tropics. Lake area has a slight effect, at least at mid to high latitudes (absolute latitude > 35°), with the correlation weakening in large lakes. The relationship with lake depth is unimodal, with the maximum correlation in lakes about 10m deep. Reservoirs (not shown) have a slightly lower correlation than lakes.

Taking these factors into account, the median correlation between air and water temperature for a mid to high latitude lake, less than 10km2 in area and between 5m and 50m mean depth is 0.81. This is similar to the correlation reported by Livingstone and Lotter (1998) for Lake Zurich. Livingstone and Lotter (1998) report that the correlation weakens with depth, becoming negative at 10-20m depth. Correlations reported by Livingstone and Dokuli (2001) for Austrian lakes and Rösneret al (2012) for a lake in northern Germany are also similar.

These results suggest that ~0.8 is the maximum correlation that could be expected between air temperature and an annual resolution reconstruction using an ideal transfer functions.

Real reconstructions are likely to perform less well. For example, some chironomids will live below the thermocline where the temperature will be weakly coupled to air temperature; other factors, such as oxygen concentrations will also affect the chironomid communities; and taphonomic processes will distort assemblage composition.

I suspect that on short timescales (i.e. monthly rather than whole summer), the correlations will decrease but only slightly as the thermal response time of the epilimnion in a small lake is fairly fast. On decadal to centennial scales, typical resolution for palaeolimnology, I suspect the correlations will increase slightly.

Code, as usual, on github.

Posted in transfer function | Tagged , , | 2 Comments

Pollen Spikes

Relative pollen abundance data can be difficult to interpret. If percent Pinus increases, it could be because pine trees became more common, or because other species became rarer. Pollen concentration (or ideally influx rates) can help resolve what is happening.

I’m still exploring the neotoma database and associated R package, and decided to look at how, and how often, pollen concentrations have been calculated (code for all this is on github if you want to see how I did it, and tell me how I can do it better).

There are two basic methods for estimating pollen concentrations. The first is to count all the pollen in a carefully weighed amount of pollen (known in Neotoma as the Jørgensen method). The second is to add a spike of exotic pollen/spores/plastic microspheres and calculate the concentration from the pollen:spike ratio. This spike method is by far the most popular method of calculating concentration in the Neotoma database, with a variety of exotics used.

Table1: Frequency of different methods/spikes. Percent do not sum to 100% as a few data sets use more than one spike.
Spike Percent
Eucalyptus 10.6
Fagus 0.1
Fern-spore 0.3
Lycopodium 15.5
Kochia 0.1
Microspheres 2.6
Unknown spike 3.9
Jørgensen method 2.6
None 65.3

There is some geographic patterns in the method used. Eucalyptus is, not surprisingly, not used in Australia, and the Jørgensen method is inexplicably popular in eastern Canada.


In my previous post on Neotoma, I noted that the very high concentrations of Eucalyptus in Hockham Mere suggested that the exotic spike had been mislabelled as a tree/shrub. I want to know if this is the only site with this problem.

Several Eucalyptus morphotaxa are found in Neotoma, but all except plain Eucalyptus are only found in Australia.


Assuming that no palynologists working on Australian material would use Eucalyptus as a spike, we only need to consider the 29 non-Australian data-sets. Of these, many have just a few Eucalyptus grains, and these are mostly from recent sediments and so the pollen are perhaps from introduced Eucalyptus trees growing near to the studied site. About ten records have, judging by the abundance of Eucalyptus pollen, mislabelled spikes. In some cases, for example the Lake of the Clouds, most levels have microsphere markers, the others have many Eucalyptus grains.


Lycopodium (clubmoss) is the other taxon commonly used as a spike. Given its worldwide distribution, it is going to be more difficult to identify rogue spikes. Twenty eight Lycopodium taxa/morphotaxa are found in 1421 data sets. The vast majority (95%) of these data sets have a maximum abundance of fewer than 50 spores and so are unlikely to be spikes. Of the remainder, some are almost certainly spikes (e.g. 4296, 4355), others are difficult to tell without going back to be original literature.


Combined Lycopodium distribution

In Neotoma, as with any big data set, there are inevitably errors in a small proportion of the data. It is user’s responsibility to know the data well enough that they can recognise unlikely data and to check that their exciting results are not being caused by errors or artefacts.

Posted in Uncategorized | Leave a comment

All the pollen

“And some things that should not have been forgotten were lost. History became legend. Legend became myth. And for two and a half thousand years, the metadata passed out of all knowledge.”


Michener et al 1997 Figure 1. Example of the normal degradation in information content associated with data and metadata over time (“information entropy”). Accidents or changes in storage technology (dashed line) may eliminate access to remaining raw data and metadata at any time

A couple of months ago, Eric Grimm gave an introduction to the Neotoma database in Bergen for participants in the HOPE project (PhD and postdoc positions to be advertised soon).

I started to tinker with the neotoma R package and downloaded some fossil pollen data. Actually, all the pollen data: over 2700 sites; 110 thousand levels; and 120 million terrestrial spores and pollen grains.


Location of Neotoma fossil pollen data sets

But what to do with 170 MB of data?

The first thing I do with any data-set is to test it for oddities: improbably values or patterns that might indicate errors or misunderstandings. I developed a set of methods last year to test the ever-so curious chironomid data from Lake Żabińskie: I’m looking forward to applying them to this huge amount of pollen data.

Expections 1: counts should be integers

The vast majority of the pollen data in Neotoma is labelled as being count data – I’ve omitted a small amount labelled as percent data. Counts should be integers, so any non-integers values would be cause for concern except that pollen analysts often count half grains of Pinus and some other conifers (some Tsuga, Podocarpus, Pinaceae) with pollen which often splits into two identifiable parts (half counts are also common in diatom and chironomid counts). So I am expecting integer and half values for some conifer species but that is not what I found.

The vast majority of counts are integer (or half) values; only 7537 (0.3%) are not.

Of these, 3923 are near integer (or half) values (absolute difference less than 0.001). These are probably because some of the count data have been back-calculated from percent data (or read off pollen diagrams) and there are rounding errors. These errors are inconsequential and are trivial to fix.

The counts for Pinaceae were more variable that I had imagined. While the vast majority of counts are integers (309096) or half values (11104), 374 counts appear to be tenths, quarters, or thirds of a grain with a few odd values that might be percent – see below.

I also discovered that some analysts count Acer grains in thirds.

Excluding the conifers and Acer which have non-integer counts, there are still several thousand non-integer counts in the database. These may represent typos, which should be sporadic, or indicate that the data are not counts, but are instead percent or pollen concentration/influx data, which might have pervasive non-integer values. It is also possible that some analysts count half grains of a broader range of taxa, in which case the non-integers should be restricted to a few taxa.

Eighty four data sets have at least one unexpected non-integer value; 37 have more than five. These are the 5 data sets with more than half the counts non-integer values.

Table 1: Proportion and number of non-integer values.
.id Proportion Number
15059 0.99 1806
16209 0.91 524
16210 0.83 171
15696 0.73 517
16090 0.59 237

We can examine these data sets with the browse function.


The very high numbers in 16090 suggest that these are influx or concentration data – one would need to check the original publication. The others look more like percent data, but need to check as the values sum to more that 100 in each sample for all but 15059. I’m going to drop these data sets from the remainder of my analysis.

The other samples with non-integer values mostly have half integers. These could be from calculating percent from a count sum of 200, or even more enthusiastic counting of half pollen grains. The remaining values look like errors of one kind or another.

It should be relatively easy to flag data with unexpected non-integer, but I’m going to ignore these for now and for simplicity round all fractional values up.

Expectation 2: Count sums should be reasonable

Many palynologists count three hundred or five hundred pollen grains per sample. I don’t think anyone ever counted twenty thousand grains per sample. It would just take too long.


The white lines at ~250 and ~850 are plotting artefacts

Very high counts might indicate that influx/accumulation rates have been entered instead of counts. Alternatively, some palanyologists might be really enthusiastic, or, in low diversity samples, the abundance of the dominant taxon might be estimated which could lead to high counts without high effort.

The data reports counts as high as 25241. I’m going to arbitrarily set 5000 as my threshold for concern. This flags 0.1% of the samples. These are some of the 36 data sets with count sums over 5000.

.id Proportion Number Minimum Median Maximum
4404 0.01 2 29 528 25241
488 0.23 9 572 2500 19062
4355 1.00 23 11626 11876 12072
16091 0.03 2 418 590 10838
3568 0.41 7 490 3766 10641
294 0.13 5 1095 2687 10481

Some of these are probably easy to explain. 4355 is either in the middle of the densest stand of Lycopodium since the Carboniferous, or the Lycopodium spike has been mis-labelled. Likewise, the Eucalyptus count suggests that 4095 (Hockham Mere) is a portal to the Antipodes.

Others appear to be typos. For example data set 20643 reports 4080 Abies grains in the first sample: none of the other samples have more than 6 Abies grains. And I’m fairly sure that the two counts of 9999 for Corylus/Myrica in data set 16091 are not real. It might be possible to use taxonomic dissimilaries within the data set to identify odd samples but as data sets can span the deglaciation large taxonomic changes are expected anyway.


Expectation 3: Few samples without singletons

“rarity is the attribute of a vast number of species of all classes, in all countries.” Charles Darwin

One of the curious aspects of the chironomid counts from Lake Żabińskie is the lack of rare taxa in many of the samples. I suggested that it is likely that in any census of any species-rich assemblage, the rarest taxa will to be represented by a single individual.

How well does the pollen data conform to this expectation. At 3%, the proportion of samples lacked singletons is higher than I had expected. The samples without singletons are not evenly distributed: 73% of datasets have no samples without singletons; 2% lack singletons in more than 50% of samples.

There is a strong relationship between the proportion of samples in a data set and the number of taxa in the data set.


About a fifth of samples in datasets where the number of taxa is 25 or fewer lack singletons. Conversely, only 2.1% of samples from datasets with more taxa lack singletons, and 1.2% of those from datasets with over 40 taxa.

I don’t think this is a caused by low diversity, but is due to a large extent to the limited taxonomic resolution and scope of some of the pollen datasets. In at least the older data, it was common to focus on a limited number of common taxa and to ignore rare species. The lack of singletons is not a useful flag for such data sets.

The almost complete lack of singletons in some species rich data sets is curious and warrants a flag.

Expectation 4: Samples that lack singletons should not have lowest common denominator > 1

It was the many assemblages without singletons were the counts were integer multiples of the rarest taxon, that first alerted me to the problems with the chironomid data from Lake Żabińskie. Such counts should be very rare, but will occur if the counts have been multiplied.

I want to flag samples without singletons where all/most of the counts are integer multiples of the rarest taxon.


Of the 3340 samples without singletons (and taxonomic richness > 5), 300 have all count integer multiples of the rarest taxon. In one sample, all counts are multiples of 43.

In one data set, 99% of values are multiples of 3, the minimum count of all samples.

I have no hesitation in suggesting that in both these examples the data are not the raw counts. Possible scenarios include the data being 1) accumulation rates or concentrations, 2) per mille, 3) back-transformed from percent after rounding, 4) the result of someone pulling a fast one.

Expectation 5. Zeros. Lots of them.

Community data usually have a many zero values and few samples will contain all the taxa found in the whole data set (unless there are very few samples), especially if the richness is high.


Data set 4082 has 56 taxa and 71 samples but only 3% of the counts are zero. Flagged as curious.

Other testable expectations?

Suggestions for other tests that could reveal errors or other problems in putative count data would be very welcome. I’m hoping that, in collaboration with Simon Goring, some of these tests can be implemented in Neotoma and that the clearly erroneous data sets can be cleaned.

Posted in Data manipulation | Tagged , , | 1 Comment

Forthcoming quantitative palaeoecology PhD and Postdoc positions in Bergen

There are vacancies for a 3-year PhD position and a 3-year post-doctoral fellow position at the University of Bergen’s Department of Biology within the Ecological and Environmental Change Research Group as part of the European Research Council funded project Humans on Planet Earth – Long-term impacts on biosphere dynamics (HOPE).


These positions are now advertised: PhD; Postdoc.

About the HOPE project:

A critical question in Earth system science is what was the impact of prehistoric people on the biosphere? There is a wealth of information about human impact through clearance, agriculture, erosion, and modifying water and nutrient budgets. Humans have greatly changed the biosphere patterns on Earth in the last 8000–11,000 years, but have humans modified the major ecological processes (e.g. assembly rules, species interactions) that shape community assembly and dynamics? To answer this question, patterns in pollen-stratigraphical data for the last 11,500 years from over 2000 sites across the globe will be explored consistently using numerical techniques to detect quantitative changes in 25 ecosystem properties. Patterns in these properties will be compared statistically at sites within biomes, between biomes, within continents, and between continents to test the hypothesis that prehistoric human activities changed the basic processes of community assembly and that interrelationships between processes changed though time.

The PHD position

Qualifications and personal qualities:

  • The applicant must hold a Master’s or an equivalent degree within quantitative palaeoecology, biogeography, or ecology, or related fields relevant to the PhD project.
  • The successful candidate should be highly motivated, enjoy the challenge of working with very large data-sets, and understand the relevance of the data and the results.
  • The successful candidate can work independently and in a structured manner, and have the ability to cooperate with others within HOPE’s consortium as well as within the EECRG, and to follow through challenging ideas.
  • Proficiency in both written and oral English is essential.

Special requirements for the position:

The successful candidate should have experience in quantitative analyses of palaeoecological or ecological data using the statistical software R or related programs, as well as documented skills in one or more research fields relevant to the position (e.g. Quaternary palaeoecology, palaeoclimatology, applied statistics, numerical ecology, quantitative palaeoecology, biogeography, macroecology, community ecology, biodiversity), and some experience of using large databases.

Special responsibilities for the position:

The successful candidate will be primarily responsible for developing quantitative procedures for evaluating taxon co-occurrences and co-correlations from pollen-stratigraphical data expressed as ‘closed’ percentages, for applying these procedures to pollen data across the globe as part of the HOPE project, and for evaluating taxon co-occurrence analysis in palaeoecology.

About the PhD position:

The duration of this position is 3 years. As a PhD candidate the successful applicant must participate in an approved educational programme for a PhD degree within the three-year period.

The Postdoc Position

Qualifications and personal qualities:

  • The applicant must hold a PhD or an equivalent degree within quantitative palaeoecology, ecology, biogeography, or a related field.
  • The successful candidate should be highly motivated, enjoy the challenge of working with very large data-sets, and understand the relevance of the data and the results.
  • The successful candidate can work independently and in a structured manner, and have the ability to cooperate with others within HOPE’s consortium as well as within the EECRG, and to follow through challenging ideas.
  • Proficiency in both written and oral English is essential.

Special requirements for the position include

  • The successful candidate must have experience in quantitative analyses of palaeoecological or ecological data using the statistical software R or related programming language and in using large databases.
  • The candidate will be able to document skills in one or more research fields relevant to the position such as Quaternary palaeoecology, palaeoclimatology, applied statistics, numerical ecology, quantitative palaeoecology, biogeography, macroecology, community ecology, and biodiversity.
  • The candidate will play a major role in the publication of HOPE results.

Special responsibilities for the position:

The successful candidate will be responsible for developing the HOPE database of pollen-stratigraphical data and associated chronological palaeoenvironmental and site data within the framework of state-of-the-art palaeoinformatics, for the numerical and statistical analyses of many large pollen-stratigraphical data-sets, and for developing appropriate software for particular palaeoecological and diversity analyses.

General Information

Closing date: 15 September 2017.

Detailed information about the position and about how to apply can be obtained by contacting: Professor John Birks, Department of Biology, University of Bergen (+47 5558 3350 or +47 5593 7717 / email: john.birks@uib.no).

Posted in Uncategorized | 1 Comment

There is a memory in the dirt at the bottom of the sea

There is a memory in the dirt at the bottom of the sea. It is in the number of different sorts of small dead animals which we can use to find out how warm or cold the sea was in the past. There are different approaches for doing this. All the approaches have problems if things are more like things near to them in space than expected. If this happens, our guess of how warm the sea was can appear to be much better than it really is. Some often-used approaches are much worse than others when this happens. This problem is hard to wipe out but ignoring it is not a good idea.

Written after #TenHundredWordsOfScience with a Simple Writer.


Posted in transfer function, Uncategorized | Tagged , , | Leave a comment

Abstract abstracts

So I am searching through the Web of Science for papers reporting 11 yr (Schwabe) cycles in palaeoproxy data (especially tree-rings) when I find this title, which looks promising:


The abstract in WoS is, well, abstract

The pilgrim fathers of the scientific imagination as it exists today are the great tragedians of ancient Athens, Aeschylus, Sophocles, Euripides. Their vision of fate, remorseless and indifferent, urging a tragic incident to its inevitable issue, is the vision possessed by science. Fate in Greek Tragedy becomes the order of nature in modern thought. …the essence of dramatic tragedy is not unhappiness. It resides in the solemnity of the remorseless working of things. This inevitableness of destiny can only be illustrated in terms of human life by incidents which in fact involve unhappiness. For it is only by them that the futility of escape can be made evident in the drama. This remorseless inevitableness is what pervades scientific thought. The laws of physics are the decrees of fate.

But when I go to check the paper, I find a much more conventional abstract

Power spectrum analysis of 81 long and 202 short Chinese dryness-wetness indices yields evidence for two peaks with periods near 18.6 and 10.5 years, both of which are statistically significant at confidence levels of 99.9 per cent. They are identified as induced by the 18.6-year luni-solar, Mn, constituent tide and a 10–11-year solar cycle, Sc, variation in the Sun’s luminosity of the order of 0.1 per cent. Amplitude and phase of Mn wavetrains are highly non-stationary with respect to both time and geography; in particular, abrupt 180° phase changes in wave polarity are often observed. Amplitude and phase of Sc waves are also highly non-stationary, with those in northern China out of phase with waves in the south since 1895 (they were in phase from 1815 to 1845). For the 202 short records variance contribution of the two signals to total variance in raw data varied from 6 per cent to 53 per cent, with a mean of 22 per cent, again demonstrating their extreme non-stationarity. Construction of a dry and very dry drought index (DVDI) shows that since 1470 by far the most prolonged, continuous, and serious drought (due to constructive interference and concomitant high amplitudes of the two waves) occurred from 1633 to 1643; the Ming Dynasty collapsed in 1644 and, in agreement with Hameed and Gong (1990), it is concluded that this climatic disaster was a causal factor in the fall of the Ming Empire.

The abstract in WoS is actually a long quote from Alfred North Whitehead.

So far I have about 100 candidate papers (sometimes it is not clear from the abstract if the paper is reporting Schwabe cycles or some longer term solar variability).

Posted in Peer reviewed literature, solar variability, Uncategorized | Leave a comment