Generally speaking, it's the more the merrier: the more annotators, the lower are the chances that half (or more) of them is wrong. If you have the possibility of using 4 annotators instead of 3 you should do it, but it's a rule of thumb.

However, using many annotators is not always feasable and, in my experience, a small set of "good" annotators (which means even though non-experts, competent, motivated, and not biased by other annotators) can produce far better results than a large pool of  incomptent and unmotivated subjects whose main aim is to fill your form in the shortest time possible. Therefore I suggest you to include a couple of gold questions in your experiment in order to identify non-competent annotators and to exclude their further contribution from your corpus. In order to prevent malevolent/subversive annotators from shilling your experiment, gold questions should be randomly mixed with regular questions, providing no clues to identify them.

Once you have filtered out bad annotators, an average of 3 annotators should be enough.

This technique works paricularly well with crowdsourcing, which you should consider in case your corpus is bigger than a hundred sentences or so.

Dario, thank you very much for your thoughts on this.

I had the same thoughts in mind but was wondering if there is any research work which explicitly examines this so that I have evidence for argueing why I select 3 (reliable) annotators and not e.g. 4 or 5.

I couldn't agree more with Dario up there. But you should take into account the size of your text and expertise required for evaluation. For POS or Machine Translation, I do not recommend non-experts. 

We have  done many serious annotation tasks ,including manually validating 1.4 million annotations on a medical corpus, and the common wisdom above is nonsense. If you want to create a gold standard then the aim has to be make it correct by building a silver standard first. More annotators do not necessarily improve the quality just the quantity. What you need is an iterative approach to completing some annotations , building a language model, testing it against the current silver standard and then correct the standard and recompute the model. The two go together and should not be separated. We built our own tools so that we could view for a corpus all the annotations that were FPs and NPs and then go through and made corrections to the standard. Until the standard is as good as possible your fighting uphill. IN answer to your question 1 annotator is sufficient, any more is an issue of increasing volume of annotations not quality.

Jon, I might have misunderstood your answer. You suggest to "correct" the existing labels that were obtained in the first manual annotation round by using the labels of the classifier in order to tune for best accuracy? Isn't that some kind of cheating as you assume the output of the classifier to be correct (and be better than the human annotation).

Retro-feeding the system can be used to improve anotation, but not used for evaluation. That would give no real criteria for the cross-validation. And "yes" just one annotator is enough, if time is not an issue, and if he is 100% reliable, and the doesn't tire, etc.

No its not cheating. It is getting the correct gold standard that has 1. no errors of human fallibility in applying the designed classification scheme, and 2. minimal errors due to a model that is insufficiently featured to capture the complexity of the text to represent the classification/annotation.

It does not give you a measure of the consistency between two annotators which is entirely irrelevant to any practical task, but is still used by academics as it gives them another performance statistic to talk about. If you want an assessment of your classifier on unseen data then hold out a sample and test once you have finished optimising the language model and correcting the gold standard as a tandem process.

At least 3 annotators

Abdelrahim, and what do these 3 annotators do that 1 doesn't do?

@Jon: From my perspective, a label in a gold standard should represent the common understanding of it.

Having only one annotator could be biased...since he/she might tend to use a particular label or might have misunderstood the labeling task in a minor aspect. Using several annotators lowers the risk of that by comparing the opinion of multiple people which allows to recognize unclear instructions or subjective aspects of an annotation task.

Let us assume we have a collection of symptom descriptions written by patients and we want to train a automated classifier telling if the patient should consult a physician or not according to the opinions of physicians. If you ask one single physician to label a data set then he might have a "laissez faire" attitude (labeling everything as "no") or he might be a very worried person (labeling everything as "yes"). Or he even might not know about the risk of a particular combination of symptoms (and hence labeling this combination as "no" but it should be yes). Or he might label everything as "yes" because he needs money. Asking multiple physicians and using the majority label would lower this risk somehow. (OK, we still need to cope with the conflict-of-interest somehow).

@Abdelrahim: Why at least 3? Why not 2? Why not 5?

@Lionel: Thanks for the hint. I know this concept as described e.g. in [1].

This is reasonable in order to get the training instances from which the classifier can benefit most in order to fine-tune the decision-boundary (as in SVM). But I would say that this does not help in case the initial data set is biased. In this case, the classifier would not recognize these "tough" cases which can not be classified with a high confidence....

[1] Eric Ringger, Peter McClanahan, Robbie Haertel, George Busby, Marc Carmen, James Carroll, Kevin Seppi, and Deryle Lonsdale. 2007. Active learning for part-of-speech tagging: accelerating corpus annotation. In Proceedings of the Linguistic Annotation Workshop (LAW '07). Association for Computational Linguistics, Stroudsburg, PA, USA, 101-108. 

Sebastian, your a talking about a different problem. The task of defining the tag set is vitally important and usually done badly. Multiple experts in creating those definitions are valuable, but they don't all have to do the annotating. In fact in one of our studies  we showed that the experts were more constant between themselves but missed more of the peripheral examples expressed in non-standard language, higher precision, but lower recall. We use linguists for doing annotations as they are much more aware of the nuances of language and more conscientious about the accuracy of the tagging. I would always advise  to not use domain experts to do the annotating as they don't have the language analysis tools. Their role is to specify the task the tag definitions and arbitrate difficult cases, however it is the linguists who refine the tag definitions after the experts have created them.

normally, the number of annotators is not interesting, the most interesting is the correlation between the annotators. if the annotators are so different in their judgements we cannot expect that a machine learning algorithm will do so well.

For example, in SemEval competition, the task of Sentiment Analysis in Twitter or Aspect_based Sentiment Analysis where the annotators are asked to classify a text into three classes. they used Mechanical Turk annotators without a special intrest with the number of annotators.

The correlation between annotators is of little practical use. It only tells you how bad your definition of your tag set is and the inability of the annotators to apply it. Or put another way: how lousy your silver standard is. As the aim of any practical task should be building a perfect gold standard corpus and then a language model that resembles it as exactly as possible. Building a gold standard annotation corpus and a gold standard language model are two interlocking tasks. Any absorptive interest in Inter Annotator agreement  (IAA) is a waste of time and failure to grasp the essential ingredients of the real task.

some thoughts in "Anveshan: A Framework for Analysis of Multiple Annotators’ Labeling

Behavior" - e.g. "Beyond Interannotator Agreement (IA)" - might be interesting. (and "Good Kappa’s Not Enough" - "Good Kappa’s Not Necessary, Either" is more support for what has - mostly - already been mentioned.)

in "Active Learning with Amazon Mechanical Turk" they 'trade off number

of annotated examples against quality of annotations' - this, generally speaking, seems like a viable option: just in case your particular problem is this 'unusual' exception to the otherwise well-behaving text classification landscape...

http://www.aclweb.org/anthology/W10-1806

http://lingpipe-blog.com/2008/07/28/good-kappas-not-necessary-either/

http://dl.acm.org/citation.cfm?id=2145597