I want to classify a text of more than 1000 words into a categories. I have a database of 10,000 reports of which each contain more than 1000 words. I want to classify them and would like to know the what has been discussed in common (Categories) in all 10,000 data through NLP.
I am new in this field and striving to know more about this. Further guidance in this section would also be appreciated.
If you don't have access to labeled data then extract some meaningful features and use unsupervised learning (i.e., clustering).
The clustering approach is an appropriate method to do classification the unlabeled data e.g. K-Means.
I think the best way to approach your problem and extract first insights from your texts is to train a topic modelling model (it's unsupervised approach. No labelling is needed.). For this reason you can use LDA from gensim. From advanced research I would suggest BigARTM library, because it contains huge amount of regularizers and it's optimized in C++ for processing of big amount of texts.
Latent Dirichlet Allocation (LDA) is the best choice for your problem.
This example will help you to start:
https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/
LDA in gensim package:
https://radimrehurek.com/gensim/models/ldamodel.html
- Badges
- Science topic
- Similar topics
- Biological Science
- Histology
- Sectioning