First, I'm not sure whether the model contains the encoder during training.
EOS means end-of-sentence. Encoder and decoder are part of transformer network.
If without-encoder, training time:
target: [E, F, G, H, EOS]
decoder input: [0, E, F, G, H]
If without-encoder, testing time:
decoder input: [0]
If with encoder, training time:
encoder input: [A, B, C, B]
target: [E, F, G, H, EOS]
decoder input: [0, E, F, G, H]
If with-encoder, testing time:
encoder input: [A, B, C, D]
decoder input: [0]
Am I exact right?
Yes, at inference time, you feed the decoder with a start symbol and then it gets its own outputs as the following inputs, whereas at the training time, you use the target sequence as the input to the decoder. If this was what you were asking.
One way of think in text generation is to abstract it even a little more and apply to an application. For instance, imagine that you need a text generator that predicts every very next words. You can think of this example:
' i want a cat'
You can split this example into the following sequences: ['i want'], ['want a']. Let's ignore the last one, as we would not have that many words to predict, just to make it simpler.
Each pair would be a training sample, with length equals to 2, and its target would be the very next word it is supposed to predict. In this case: ['a'], ['cat'].
Your dataset would be:
X: ['i want'], ['want a']
Y: ['a'], ['cat']
If you think of each word as a timestep, you are basically creating a sequence of timesteps 't' and trying to predict 't + 1'. And the amazing part is that recurrent neural networks are great when dealing with sequences.
This is an interesting basic idea to try with a text generation model. On a small corpus, a vanilla RNN should learn some interesting patterns. You can even go deeper and try with newer kinds of RNNs, composed by LSTM or GRU units.
- Badges
- Science topic