I found a paper entitled "Multimodal representation: Kneser-Ney Smoothing/Skip-Gram based neural language model". I am curious about how the Kneser-Ney Smoothing technique can be integrated into a feed-forward neural language model with one linear hidden layer and a softmax activation. What is the purpose of the Kneser-Ney in such a neural network, and how can it be used for learning the conditional probability for the next word?
Have a look at https://arxiv.org/pdf/1901.09296.pdf
https://arxiv.org/pdf/1901.09296.pdf
https://ieeexplore.ieee.org/document/7532765
Muhammad Ali your answer is not relevant. It would be useful to remove it.
The Kneser-Ney Smoothing. Probability smoothing (algorithms) is not used for learning the conditional probability. Probability smoothing aims to correct zero probability problem. The problem arises in the case of, the dataset includes some rare events or contain various topics.
To summarize the case zero-probability, just mean, the event has zero probability, i.e. in some cases zero probability can indeed happen (ex. poor data collection or contradictions), therefore, assigning zero weights to rare event not a reasonable thing to do so.
Zero probability hurts the accuracy and performance, as model assign zero weights for rare events, and this might cause overflow, in case of sigmoid activation. Most APIs, or even our implementation, we have to handle dividing by zero, by adding very small value epsilon, to the exponent of the exponential function (clipping) zero, this usually hurts the numerical stability as dominator becomes very small.
In case of, feed-forward neural language model with one linear hidden layer and a softmax activation, Probability smoothing resolve the problem of having sparse weights, which consumes the memory (useless space), moreover improve model accuracy by enforcing model assign small weight values, to rare events (at least).
Kneser-Ney Smoothing, gives better representation, for zero probability than Laplace smoothing, (ex. clipping, extended value function).
Mohamed Zayan Thank you for your answer. You explained the reason of why we use the Kneser-Ney Smoothing very well, but in practice, in which phase is this smoothing applied?
For example, does one train a language model and in prediction phrase the smoothing algorithm re-estimate the probabilities? Or, are the class labels re-estimated with the use of smoothing algorithm during training?
So, In practice how can a neural network and a smoothing algorithm be both in a system/software/architecture?
- Badges
- Science topic