The idea is old, of course. In scientific journals already in the 1950s, if not earlier, there were a "Comment" section after some papers. See samples here, the first one being from 1954:

http://ie.technion.ac.il/serveng2014W/Lectures/Dantzig.pdf

http://pubsonline.informs.org/doi/abs/10.1287/mnsc.33.12.1635

I am not sure how these things were arranged, but it is an appealing thing, as long as they are not meant to simply discredit or point out errors but to actually bring about fruitful discussions. These days when publications are found online on individual, dedicated web pages, this should be rather easy to arrange on the web, directly below the link to the full-text of a paper. Perhaps there already are!? It is indeed an interesting idea, and I hope it is followed up!

The idea is old, of course. In scientific journals already in the 1950s, if not earlier, there were a "Comment" section after some papers. See samples here, the first one being from 1954:

http://ie.technion.ac.il/serveng2014W/Lectures/Dantzig.pdf

http://pubsonline.informs.org/doi/abs/10.1287/mnsc.33.12.1635

I am not sure how these things were arranged, but it is an appealing thing, as long as they are not meant to simply discredit or point out errors but to actually bring about fruitful discussions. These days when publications are found online on individual, dedicated web pages, this should be rather easy to arrange on the web, directly below the link to the full-text of a paper. Perhaps there already are!? It is indeed an interesting idea, and I hope it is followed up!

Thanks very much for your comments, Michael. I agree that comments and letters to the editor are old, but I think the above-linked things have some novel aspects:

• The speed with which one can comment is much faster now (for the 1954 example you give, the paper was published in May and the comment in August; not bad for 1954 -- I'm sure some modern letters to the editor are much slower than that! Now though, the comment could appear while the paper is still in the online proof stage).
• There can be sustained debate (e.g. authors respond and the original or other peer can add further comments, etc., whereas most journals allow only one response from authors and no further comments from peers).
• The option for anonymity enables junior (but often highly expert) grad students and postdocs to comment on the work of senior investigators in ways that would be unwise in the context of letters to the editor. It of course also brings potential problems.
• Ratings/evaluations are also rather different in character to comments/letters, and it's easy to imagine them becoming quite important at PLoS One and other "mega journals", due to the huge volume of papers they are now publishing.
• As mentioned, Publons also has a novel mechanism for getting credit for your pre-publication reviews.
• Journals that publish first then review afterwards (F1000 Research, perhaps others) are also rather revolutionary in my opinion.
• Somewhat related to this is the novel (I think!) idea of transferable peer-review (i.e. get reviewed first, submit to journal afterwards!) as in Rubriq and Peerage of Science.

http://www.rubriq.com/

http://www.peerageofscience.org/

Just for information: There was also an attempt by Nature to implement an open (pre-publication) peer review; this approach is discussed and the results of this attempt are summarized here: 

http://www.nature.com/nature/peerreview/debate/nature05535.html

http://www.nature.com/nature/peerreview/debate/nature04991.html

http://www.nature.com/nature/peerreview/debate/

Thanks Neslihan, PubPeer's view on anonymity is that "Blind peer review has been employed by most major scientific publications in order to allow reviewers the ability to critically assess the work of their peers without fear of retaliations." Personally, I think there is a place for anonymous post-publication review, but it perhaps needs to be moderated to prevent abuse.

https://pubpeer.com/faq

Artur, thanks for the very helpful links; there's a lot to read there! One interesting counterpoint to Nature's apparent lack of enthusiasm for changing their reviewing system (at least in 2006) is that in 2013 they invested in the publisher Frontiers, which has a rather unusual review process: "after initial review, authors and reviewers discuss the paper together in an online forum, and reviewers’ names appear publicly on the final paper." (note that reviewers of rejected papers remain anonymous, a compromise that I find quite reasonable, though anecdotally I have heard several complaints from reviewers about how difficult it can be to get a Frontiers submission rejected).

http://blogs.nature.com/news/2013/02/nature-publishing-group-buys-into-open-access-publisher.html

An interesting related development at the BMJ -- they have just announced (2014-09-03) that they will make their (pre-publication) reviews fully open to readers.

http://dx.doi.org/10.1136/bmj.g5394

I think the idea of post-publication is very interesting, but had no experience with this yet.

Have you heard of Philica? It is an open access online journal with only post-publication review.

philica.com

It is interesting to note that at the Q & A site for theoretical physics physicsoverflow there is a special section for reviews and discussion of published papers and preprints from arXiv.org:

https://physicsoverflow.org/answers/refereeing