Sequence-to-Sequence (or Seq2Seq) is a neural net that transforms a given sequence of elements, such as the sequence of words in a sentence, into another sequence. (Well, this might not surprise you considering the name.)
Seq2Seq models are particularly good at translation, where the sequence of words from one language is transformed into a sequence of different words in another language. A popular choice for this type of model is Long-Short-Term-Memory (LSTM)-based models.
The paper ‘Attention Is All You Need’ introduces a novel architecture called Transformer. As the title indicates, it uses the attention mechanism we saw earlier. Like LSTM, Transformer is an architecture for transforming one sequence into another one with the help of two parts (Encoder and Decoder), but it differs from the previously described/existing sequence-to-sequence models because it does not imply any Recurrent Networks.
Is it possible to replace LSTM with transformers in all tasks or the LSTM has its particular applications?
Dear Riadh Ayachi
Have a look:
https://stats.stackexchange.com/questions/472822/is-lstm-long-short-term-memory-dead
Kind Regards
Qamar Ul Islam
Dear Riadh Ayachi
The following papers compare the LSTM and transformer, which could be useful to you:
[1] A Comparison of Transformer and LSTM Encoder Decoder Models for ASR, in
https://ieeexplore.ieee.org/abstract/document/9004025
[2] Incorporating Transformer and LSTM to Kalman Filter with EM algorithm for state estimation, inPreprint Incorporating Transformer and LSTM to Kalman Filter with EM ...
[3] Transformer with Depth-Wise LSTM, in Preprint Transformer with Depth-Wise LSTM