I am using GloVe for the first time and I've discovered that some words are present both alone and with punctuation signs. For example, all the following tokens are present in GloVe:
- you
- you,
- you.
- ,
- .
What is the best practice? Should I separate the words from the symbols use them as two tokens or use them together as one single token?
Moreover, I've found that some common expressions are present too:
- you'll
- you're
- you
- '
- 'll
- 're
Same question: should I split these expressions into different tokens or use them as a single one?
Hi Andrea,
In general, if the punctuation is not important in your task, you may need to either
- obtain two tokens (including the punctuation) using code, for instance, then delete the existing punctuation, or
- tokenize the text based on the existing punctuation, or
- such tokens can be existed in the stopwords list.
HTH.
Samer Sarsam, PhD.
Hi, Andrea,
As Samer mentioned, this depends on your end task. But in most tasks, having less same word's forms is better to lower data sparsity. Thus, it is better to prepare your data by separating these attached suffixes from the word. i.e replace (you'll )by ( you 'll) , then you can either consider ('ll )as stop word and remove it, or keep it if it is important for your end task.
Tiba Zaki, PhD.
I generally remove leading and trailing punctuation signs, such as . , ; . " ! ? ( ), since they do not contribute to the word semantics.
A problem may occur with - at the end of a token, since this could indicate, that the word was hyphenated. However, identifying what to do with the hyphenated parts (whether to join them) is difficult and language dependent.
Additionally, I recomment to expand any abbreviation, such as the ones you mentioned yourself.
- Badges
- Science topic