The inference speed of transformer-xl is faster than transformer.
Why?
If state reuse is the reason, so it is compared by 2 32seq_len + state-reuse vs 1 64seq_len + no-state-reuse?
https://www.lyrn.ai/2019/01/16/transformer-xl-sota-language-model/
For vanilla Transformer language models (Al Rfou et al), you process [1 2 3 4], predict 5, process [2 3 4 5], predict 6, and repeat.
For a Transformer-XL language model, you process [1 2 3 4], predict [5 6 7 8], predict [9 10 11 12], and repeat. Note that we don't have to reprocess [5 6 7 8] when predicting [9 10 11 12], because the representations can be reused.
https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html
Following link can be useful:
https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html?m=1
- Badges
- Science topic