Dear Thushari,

You can use Apache Tika to do this (https://tika.apache.org/1.4/detection.html#Language_Detection).

Kind regards,

Olivier PARISOT

Hi Thushari,

For intrasentential code switching, a probabilistic approach seems appropriate. Even though this is not an opensource tool, the following paper may be a good start:

http://193.6.4.39/~czap/letoltes/IS14/IS2014/PDF/AUTHOR/IS140227.PDF

Cheers,

Mohammad

Dear Olivier and Mohammad,

Thanks for your answers.

@Olivier - I have tried this. But it didn't work for my problem in detecting multiple languages in a single sentence (intrasentential). Thanks anyway

@Mohammad - Thanks for this paper. Yep, it looks like it will be a good start!

Cheers,

Thushari

you can use language identifier: Nakatani Shuyo. 2010. Language Detection Library

for Java. python version also available.

https://code.google.com/p/language-detection/ 

Thanks Santanu! Yep, I already tried that tool for detecting language in text. However, it does not work well for intrasentential language detection problem. Thanks for your suggestion!

Cheers,

Thushari

One of my colleague has used this tool, he passed unigram, bigram and trigram tokens to identify intrasentential language detection :=)   This may solve your problem.

Thanks Santanu. It's a great suggestion. I'll give it a try with n-gram model

Cheers,

Thushari

hi , this is an old thread, but Mohammad Ghoniem , do you have an updated link to your recommended paper. Preferrably one without ip in it ?

Thank you !