As a student in Bachelor degree program in computer science field we have a project in course of "Language Theory", our project related with Natural Language Processing:
- First phase talks about giving a dictionary of "Physical Objects Name's" and give it a "Text" (all this in input) after that it gives us a list of "Physical Objects Name's" in our "Text" (this is the output as a file).
- Second phase is to use the last list to as input and implement a code that can classify words by topics and the result will be the general topic or idea of our text.
In this project I did the first phase but in the second one I don't understand how can I implement my code.
P.S: I try to add a python file but I can't, so for all those who wanna help me I can send them my work.
You can use spaCy NER (named entity recognition models) or hugging face transformers, depending on the topic each model is trained to detect.
https://spacy.io/universe/project/video-spacys-ner-model-alt
https://huggingface.co/dslim/bert-base-NER
You can also add a rule for your own entities:
https://spacy.io/api/entityruler
Hope it helps,
Az
Dear Abdelkrim Yassine Taki Eddine Dib
In the second phase of your project, you'll implement a code to classify words by topics and determine the general topic or idea of the text. You can use embedding models in Natural Language Processing (NLP) to achieve this. Embedding models represent words or phrases as dense vectors in a continuous vector space, capturing semantic relationships between words. Python offers libraries and frameworks for embedding models, such as TensorFlow, PyTorch, and Gensim. To convert words into numerical representations, you can use pre-trained word embeddings like Word2Vec, GloVe, or FastText. Then, you can employ machine learning techniques such as clustering or classification algorithms to group words by topics based on their embeddings. Alternatively, you can fine-tune pre-trained models or train your embedding models on domain-specific data for better performance. By integrating these techniques into your code, you'll be able to analyze the list of physical objects' names and determine the overall topic or idea of the text concerning the classified words.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
# Preprocessing
# Implement your preprocessing steps here (tokenization, removing stop words, etc.)
# Feature Extraction
tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(preprocessed_text_data)
# Topic Modeling
lda_model = LatentDirichletAllocation(n_components=5) # Number of topics to discover
lda_matrix = lda_model.fit_transform(tfidf_matrix)
# Training a Classifier
svm_classifier = SVC(kernel='linear')
pipeline = Pipeline([
('tfidf', TfidfVectorizer()),
('clf', svm_classifier)
])
pipeline.fit(preprocessed_text_data, topic_labels)
# Testing and Evaluation
# Evaluate your classifier using a separate test dataset
