If a dataset contains more punctuation, what will be its effects on the training phase using deep learning techniques.
Hi.
The impact of punctuation symbols on the effectiveness depends on your task. In some cases, all punctuation symbols (comma, semicolon, etc.) can be removed at a preprocessing stage (for instance, for semantic representation of words or sentences). Such a choice can be explained by the lack of semantic meaning of these symbols, therefore, they are considered as noise data. But if you are performing a syntactic analysis, correct punctuation symbols can provide your model with helpful information about connections between the different parts of a text. Also, don't forget about grammar correction tasks that require taking into account all tokens too.
To sum up, the decision to remove punctuation symbols or not is up to you. You can verify it in an experimental way. In general, punctuation symbols are usually removed but there are a variety of cases where their presence can improve the effectiveness of your model.
Artem Kramov , Thanks for the points that you have added here. To be precise, I am currently working on Chatbot implementation (Retrieval Based). In most of the implementation that has been conducted in this field so far considers the punctuation as noisy data.
If possible can you refer me some article or research paper which provides a detailed explanation for the same.
Thanks
Sumit Singh Chauhan Unfortunately, I'm not an expert in Chatbot development. However, according to this paper (Conference Paper Building a Production Model for Retrieval-Based Chatbots
) the punctuation needs to be accounted for. This paper is open-access.- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training