Of all the modes of post-publication peer review, comments published in the same journal as the original article are the most visible, and because they have survived editorial and reviewer scrutiny, carry at least modicum of credibility. Unfortunately, comments are much rarer than expected given the number of papers our journal club savages. For example, QSR has published just a couple of dozen in the last decade. Part of the reason for the sparsity of comments must be the hurdles placed in the path to publication.
These are the requirement from Science
Technical Comments (up to 1000 words, 2 figures or tables, 15 references, and no Supplementary Materials), are published online and critique the core conclusions and/or methodology of research published in Science within the previous 3 months. Technical Comments should not present new data or other previously unpublished work nor be based on new findings/concepts that would not have been accessible to the authors when the paper was written.
The word limit is tight and the deadline restrictive (confession: I don’t always read Science papers the day they are published), but otherwise this is reasonable.
How about the requirement from Nature Communications?
Submissions should challenge the main conclusions of the Nature Communications paper and contain new, unpublished data to support the arguments.
• They should not exceed 1200 words (main text).
• Contributions should start with a brief paragraph that summarizes the message of the article without specialized terminology, for a non-specialist readership. This paragraph should be used as the abstract for submission purposes.
• Contributions should have a simple message that requires only one or two small figures or tables. Contributions with more than two figures and/or tables will not be considered.
• As a guideline, contributions allow up to 15 references.
• Supplementary Information is permitted at the editor’s discretion.
The requirement for “new, unpublished data” is the most important difference: Science forbids it, Nature Communications demands it.
I’ve helped write a few comments showing that:
- The method used by Pither and Aarssen (2005) to show that most diatoms taxa are pH generalists had a very high type II error rate which voided their conclusions.
- That changes of the sedimentation rate in Lake Erhai caused artifacts that Wang et al (2012) misinterpreted as flickering prior to a critical transition.
- The methods used by Lyons et al (2015) caused a spurious breakpoint in the mid-Holocene.
The common feature of these comments is that none of them contain new unpublished data.
Nature Communications doesn’t require articles to contain new unpublished data. This is good, many of my papers don’t contain new data; new methods and ideas are as important as new data. However, it potentially leads to a bizarre situation that an article presenting a novel but flawed analysis of previously published data extracted from Neotoma or some other database could not be critiqued in a comment unless the authors were able to find some new unpublished data (and describe it in 1200 words).