Debunked!

Serendipity (also known as procrastinating on Google) guided me today towards an abstract for an oral presentation given at the 22nd International Diatom Symposium in 2012.

The title,

A null model analysis debunks widely-cited evidence of dispersal limitation among freshwater diatoms

caught my attention for two reasons. Firstly the language is unusual – I don’t think I have ever attended a presentation that claimed to debunk widely-cited evidence. Secondly, I have a paper on dispersal limitations of freshwater diatoms, that has been reasonably well cited…

As I read the abstract, I soon realise that it is indeed my paper that has been “debunked”.

Are diatoms ubiquitously dispersed? Or, like larger organisms, do diatoms only disperse over limited geographic extents? Unequivocal answers to these questions remain elusive, primarily because direct measurements of diatom dispersal are scarce. In 2006, Telford et al. published a study in Science purporting to provide indirect evidence against the controversial “ubiquitous dispersal hypothesis”. Their article has garnered widespread attention: it has been cited almost 100 times (including articles published within Diatom Research), and citations continue to accumulate rapidly. Here, I use a traditional null model analysis – involving a randomization procedure with the original datasets – to first show that the patterns documented by Telford et al. were expected entirely by chance, and therefore do not constitute evidence of dispersal limitation. Then, drawing upon large calibration datasets from North America and Europe, in conjunction with null model analyses of pH tolerance, I provide preliminary evidence in support of the hypothesis that taxa adapted to the most common pH environments are the least dispersal‐limited.

The author is Jason Pither. It is unfortunate that Pither thought it more important to mention that the “debunked” paper had been cited in Diatom Research than to describe what his “traditional null model” did.

Predicted patterns of lake diatom species richness in relation to pH (A) for three regions with different pH availability (gray areas) if metacommunities operate at the regional (solid blue) or global (dashed red) scale. The thick black line shows a hypothetical global species richness. Observed lake pH availability and standardized rarefied species richness in individual lakes (B) for North Florida (black), South Norway (red), and Finnish Lapland (blue). Thick lines show trends in species richness estimated with generalized additive models. Species richness pH optima against modal lake pH (C) in 16 regional data sets (7) from Europe (a to i) and North America (j to p). Regions are labeled from south to north in each continent. The modal lake pH is estimated from the pH density. Species richness pH optima are estimated using LOWESS (span = 2/3). If richness reached a plateau rather than an optimum, the break of slope was used instead. The linear regression of richness optima on most common lake pH (solid line with dashed confidence interval) is highly significant (P < 0.001; excluding two outliers, regions o and k).

Predicted patterns of lake diatom species richness in relation to pH (A) for three regions with different pH availability (gray areas) if metacommunities operate at the regional (solid blue) or global (dashed red) scale. The thick black line shows a hypothetical global species richness. Observed lake pH availability and standardized rarefied species richness in individual lakes (B) for North Florida (black), South Norway (red), and Finnish Lapland (blue). Thick lines show trends in species richness estimated with generalized additive models. Species richness pH optima against modal lake pH (C) in 16 regional data sets (7) from Europe (a to i) and North America (j to p). Regions are labeled from south to north in each continent. The modal lake pH is estimated from the pH density. Species richness pH optima are estimated using LOWESS (span = 2/3). If richness reached a plateau rather than an optimum, the break of slope was used instead. The linear regression of richness optima on most common lake pH (solid line with dashed confidence interval) is highly significant (P < 0.001; excluding two outliers, regions o and k).

The paper Pither is attacking is a brevia I published in Science with Vigdis Vandvik and John Birks. Using several diatom-pH calibration sets developed for palaeoecology, we noticed that in regions where lakes were predominantly acidic, the diatom species richness optimum was more acidic than in regions where lakes were predominantly alkaline. We realised that we could use this as a test of the ubiquitous dispersal hypothesis, as such a pattern could only readily be explained by metacommunity dynamics which implies that there are dispersal limitations.

Previous (and many subsequent) tests of the ubiquitous dispersal hypothesis have been bedevilled by taxonomic issues (is a taxon endemic, or is it just a synonym), environmental gradients (is a taxon not at a site because of dispersal limitations or because it is outside its niche), and undersampling (is the taxon really here, but too rare to find). Our analysis neatly sidestepped these problems.

Our analysis lacked just one thing, a formal statistical test to demonstrate that the pattern we observed was not just a chance occurrence or an artefact of how the datasets were arranged on a pH gradient. Pither noticed this and, hours before the six month period in which Science will consider comments expired, submitted a comment on this paper. His first version introduced a null model that was so bad the reviewer gave him an example in Excel to show how it could be done better. Better, but still not good. A more appropriate null model showed that the pattern we found was statistically significant.  

It is not clear if the latest “debunking” is a new analysis or a reincarnation of Pither’s earlier comment.

This comment of our Science brevia was not the first time I had come across Pither’s work. In 2005 he had published a paper that claimed to find, using a null model analysis, that most diatoms are environmental generalists rather than specialists. This came as a great surprise to those who had been successfully using diatoms as an environmental and palaeoenvironmental indicator for decades. Intrigued, I tested Pither’s methods with some simulated data – could the method recognise a specialist when given one. Hardly. Pither’s method had a very high Type-II error: the null hypothesis that species are generalists was accepted even when it was false. With standard tools like GLM, most of the species in the dataset analysed by Pither were identified as environmental specialists (those that were not included soil-dwelling diatoms washed into the lake). This analysis was written up and became the first comment published in Ecological Letters.

Relationship between truth/falseness of the null hypothesis and outcomes of a test
Null hypothesis (H0) is true Null hypothesis (H0) is false
Reject null hypothesis Type I error
False positive
Generalist classified as specialist
Correct outcome
True positive
Specialist classified as specialist
Fail to reject null hypothesis Correct outcome
True negative
Generalist classified as generalist
Type II error
False negative
Specialist classified as generalist

Pither and his coauthor’s reply relied on a claim that whereas for their analysis a Type-I error was to misclassify a generalist as a specialist and a Type-II error was to incorrectly classify a specialist as a generalist, our analysis did the converse. This complete nonsense was apparently suggested by a reviewer!  For our GLM analysis, the null hypothesis was, of course, that the species had no relationship with the environment, so a Type-I error was to believe a species was specialised when it was not. The rest of the reply was no better, and did not get to grips with the fundamental problem that when given a simulated specialist species Pither’s methods were typically unable to identify it as a specialist.

So given Pither’s previous luck with null model analysis, I shall not fret about my “debunked” paper tonight.

Advertisements

About richard telford

Ecologist with interests in quantitative methods and palaeoenvironments
This entry was posted in Peer reviewed literature and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s