Dear Shyambhu Mukherjee

Have a look at the following link :

https://github.com/RandyPen/TextCluster This is a Cluster method specific to short text, which outperforms KMeans etc. No Need to set latent variable number : The basic idea is to tokenize the sentence into words. Then direct to different bucket according to text component. In each bucket, calculate similarity between the sentence and the bucket. If the similarity score is higher than specific value, append this sentence into that bucket, else search for the next bucket.

So, go out there and build something.

All the best,

Samawel JABALLI

https://github.com/RandyPen/TextCluster

This is a Cluster method specific to short text, which outperforms KMeans etc. No Need to set latent variable number.

The basic idea is to tokenize the sentence into words. Then direct to different bucket according to text component. In each bucket, calculate similarity between the sentence and the bucket. If the similarity score is higher than specific value, append this sentence into that bucket, else search for the next bucket.

Dear Shyambhu Mukherjee

Actually, researchers use BOW (bag of words) and tf-idf for text embedding in short-text clustering. Then a simple K-means algorithm will be useful.

You can see the following paper,

https://aclanthology.org/W19-4322

References to the work of Prof. E.V.Lutsenko on ASC-analysis of texts

ASC-analysis of texts allows you to:

- to form generalized linguistic images of classes (semantic cores) based on fragments or examples of the transformation of texts in the age language to them;

- a set of linguistic images of people, or a description of an object, a process with generalized linguistic images of group (classes);

- generalized linguistic images of classes with each other and the creation of their clusters and constructions;

- the study of the simulated subject area of the study of its linguistic system-cognitive model;

- carry out intellectual attribution of texts, i.e. determining the probability of authorship of anonymous and pseudonymous texts, dating, genre and semantic orientation of content texts;

- all this can be done for any natural or artificial language or coding system.

Lutsenko E.V. Synthesis of semantic nuclear scientific specialties of the Higher Attestation Commission of the Russian Federation and its scientific specialties: mechanization, agronomy and veterinary medicine / E.V. Lutsenko, N.V. Andrafanova, N.V. Potapova // Polythematic State network electronic scientific journal of Kuban Agrarian University (Scientific Journal of KubGAU) [Electronic resource]. - Krasnodar: KubGAU , 2019. – №01(145). P. 31 - 102. - IDA [ article ID ]: 1451901033– - Access mode: http://ej.kubagro.ru/2019/01/pdf/33.pdf , 4,5 cu.p.l.

Lutsenko E.V. Formation of the semantic core of veterinary medicine by Automated system-cognitive analysis of passports of scientific specialties of the Higher Attestation Commission of the Russian Federation and automatic classification of texts in the fields of science / E.V. Lutsenko // Polythematic network electronic scientific journal of Kuban State Agrarian University (Scientific Journal of KubGAU) [Electronic resource]. - Krasnodar: KubGAU , 2018. – №10(144). P. 44 - 102. - IDA [ article ID ]: 1441810033– - Access mode: http://ej.kubagro.ru/2018/10/pdf/33.pdf , 3688 u.p.l.

Lutsenko E.V. Intellectual binding to literary sources in bibliographic databases using ASK analysis and the Eidos system (on the example of the Russian Scientific Citation Index - RSCI) / E.V. Lutsenko, V.A. Glukhov // Polythematic network electronic scientific Journal of Kuban State Agrarian University (Scientific Journal of KubGAU) [Electronic resource]. - Krasnodar: KubGAU , 2017. – №01(125). P. 1 - 65. - IDA [ article ID ]: 1251701001. - Access mode: http://ej.kubagro.ru/2017/01/pdf/01.pdf , 4062 u.p.l.

Lutsenko E.V. Application of ASK-analysis and intellectual system "Eidos" for solving common problems of determining literary sources and authors by approximate, non-standard and scientific bibliographic descriptions / E.V. Lutsenko // Polythematic network electronic scientific journal of Kuban State Agrarian University (Scientific Journal of KubGAU) [Electronic resource]. - Krasnodar: KubGAU , 2014. – №09(103). P. 498 - 544. - IDA [ article ID ]: 1031409032. - Access mode: http://ej.kubagro.ru/2014/09/pdf/32.pdf , 2,938 cu.p.l.

Lutsenko E.V. ASC-analysis of the problem of the article of the KubGAU Scientific Journal in dynamics / E.V. Lutsenko, V.I. Loiko // Polythematic network electronic scientific journal of the Kuban State Agrarian University (KubGAU Scientific Journal) [Electronic resource]. - Krasnodar: KubGAU , 2014. – №06(100). P. 109 - 145. - IDA [ article ID ]: 1001406007. - Access mode: http://ej.kubagro.ru/2014/06/pdf/07.pdf , 2,312 cu.p.l.

Lutsenko E.V. Attribution of anonymous and pseudonymous texts in system-cognitive analysis / E.V. Lutsenko // Polythematic network electronic scientific journal of Kuban State Agrarian University (Scientific Journal of KubGAU) [Electronic resource]. - Krasnodar: KubGAU , 2004. – №03(005). P. 44-64. - IDA [ article ID ]: 0050403003. - Access mode: http://ej.kubagro.ru/2004/03/pdf/03.pdf , 1312 u.p.l.

Lutsenko E.V. Attribution of texts as a generalized task of identification and forecasting / E.V. Lutsenko // Polythematic network electronic scientific journal of Kuban State Agrarian University (Scientific Journal of KubGAU) [Electronic resource]. - Krasnodar: KubGAU , 2003. – №02(002). Pp. 146 - 164. - IDA [ article ID ]: 0020302013. - Access mode: http://ej.kubagro.ru/2003/02/pdf/13.pdf , 1,188 cu.p.l.

Lutsenko D.S., Lutsenko E.V. Intellectual dating of the text, definition of authorship and genre in the history of Russian literature of the XIX and XX centuries, 2020 // The article is in the operational archive. 38 p. - DOI : 10.13140/RG.2.2.28824.01281 , https://www.elibrary.ru/item.asp?id=43796415

Lutsenko D.S. , Lutsenko E.V. Intellectual attribution of literary texts ( finding the date of the text , determining authorship and genre on the example of Russian literature of the XIX and XX centuries ) , 2020 // The article is in the Russian archive . 9 p . – DOI : 10.13140/RG.2.2.15349.81122 , https://www.elibrary.ru/item.asp?id=43794562

Regardless whether its clustering or classification, you can also try to vectorize parts of words (say, 3-4 successive characters), morphemes, punctuation and add other meta-information (e.g. hashtags, 'likes', geotags, NER: person, location, etc; sentiment, swear words). You'll get more features to consider.

Characters can indicate morphological and semantic patterns. E.g. 'vegan', 'vegetarian', 'vegetables' contain the same 'veg' pattern. In 'oxidizer', 'atomizer', itemizer', 'utilizer' the pattern 'izer' hints at a formal style and science-orientedness.

I've come across papers using this method, but cannot recollect the exact titles. I remember systems using it for propaganda detection in P. Nakov et al. project and SemEval track.

Here's one of the algorithms for part-of-word tokenization: https://towardsdatascience.com/wordpiece-subword-based-tokenization-algorithm-1fbd14394ed7

Thanks a lot Elena Mikhalkova

Maybe one idea is to use character embedding instead of word embedding. Higher-level embeddings also could be useful such as morpheme level or unsupervised subword tokenization.