I'm new to LeakGAN or SeqGAN or TextGAN. I know GAN is to generate text and let discriminator un-judge-able to real text and gen-text.
LM(language model) is the task of predicting the next word and can also be used to generate text.
The main idea of the use of the two methods may be one, but the working principle is different. Note that LM may follow the statistical fundamentals to reach the result. Either the GAN method depends on the networks.
The work of these two methods may also differ as data types work it.
.
the difference is the "exposure bias"
.
LM are trained on the ground truth (predict the next symbol from the the _true_ past symbols) but deployed in a different setting (when generating a string, the LM predicts the next symbol from the _generated_ past symbols)
such difference between training and deployment does no occur with GAN
.
see https://arxiv.org/pdf/1810.12686v2.pdf
.
I believe that is the correct definition of a language model, whether the predicted next token is a character or a word.There are many statistical language models that do rely on probability distributions over sequences of words, such as N-grams and Markov Chains. However, there are also black-box language models that are pretty common in the literature, and are proven to yield better accuracy in baseline data sets, such as RNN and C-RNN. This is not to say that they have completely hidden features; much work is put into this recently and the attention mechanism, for example, is a good way to visualize the learned relationships between tokens.
My point really is that GANs need not be so different from what we call a language model. The generator portion of these networks are usually a language model indeed, whether is a standard RNN or a different version of it. The only major difference is the adversarial training with the discriminator as opposed to training based on solely maximum likelihood and categorical cross entropy.
Based on my understanding, the main difference of TextGAN and conventional LM is the traing process and model architecture. GAN model must contain a generator(s) and a discriminator , the model is trained in a adversarial manner. The conventional LM is not trained in a adversarial manner and not always has a generator and discriminator. I used the word 'conventional' above means that the definition of ML is broad, the state-of-the-art ML can involve GAN components.
- Badges
- Science topic