Hi Gaurish Thakkar,

If you want to represent morphemes or words into vectors, I can recommend you to use Word2Vec

Word2Vec

Word2vec is a set of methods/algorithms that are usually used in word embeddings.

How does this work? You can easily find out here:

- https://www.quora.com/How-does-word2vec-work , or here

- https://en.wikipedia.org/wiki/Word2vec

There is a good paper about that, which is done by Mikolov in NIPS 2013 (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)

Also, there are many really nice library for this

- Using DeepLearning4J (https://deeplearning4j.org/word2vec)

- Using TensorFlow (https://www.tensorflow.org/versions/r0.11/tutorials/word2vec/index.html)

- Original Google version (https://code.google.com/archive/p/word2vec/)

Roughly speaking, it consists of two main methods: Skip Gram and Bag of Words (CBOW)

https://www.quora.com/How-does-word2vec-work

https://en.wikipedia.org/wiki/Word2vec

https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

Hi Gaurish,

In addition to IDF and TF, sveral ways can be applied, such as word occurrence in the document, i.e., if the word appears in the documet, then its value is 1, but 0 if not. Another method relies on word counts in the document. 

HTH.

Samer

this seems a word level. Is there any morpheme specific representation ?

Just appeared on Arxiv:

https://arxiv.org/abs/1610.00479

Why don't you use the Levenshtein distance?

https://en.wikipedia.org/wiki/Levenshtein_distance

Regards,

Joachim