I have a project where I need to classify both documents that are news articles and short messages/blog comments for the articles.
I have tried the following with the same sentences and get different confidence levels. So now I am a bit worried as to which service to use best. I have tried Alchemy API, MonkeyLearn, Algorithmia and Aylien with the following sample:
"While I fully agree that the pilots, like any other worker, have the right to know the state of affairs in their company, I think it is very stupid to start thinking about industrial action at this stage. If they are not careful there may not be an airline left anyway. Please do not shoot yourselves in the foot and think carefully before you take any 'action'."
The answers where:
- Alchemy: negative 0.473512
- MonkeyLearn: negative 0.484
- Algorithmia: Score of 1 (0 is very negative - 2 is neutral and 4 is very positive)
- Aylien: negative 0.62843 (in tweet mode), positive 0.9867 (in document mode).
I tried different comments as text and have obtained mixed results, as for example the nearness of the Alchemy and MonkeyLearn is not repeated often.
My question is: which service shall I choose? :-)
It is likely that these different tools might be employing models that were built using different machine learning algorithms to estimate the confidence values (e.g.: neural netowrks, naive bayes, SVM, ...). Some tools may also be sensitive to parameter configuration.
In general, different algorithms will obtain different results.
As well, not all the APIs take into account the same features to determine the polarity of text. For example, some APIs make uses of ponctuation or emoticons, while some other no. Maybe these article could lead you some clues about which system you should take:
http://dl.acm.org/citation.cfm?id=2837216
http://link.springer.com/chapter/10.1007%2F978-3-319-41754-7_48
