How do you interpret different concepts in NLP for time-series? For example “self-attention” and “positional embeddings” in transformers.
Dear Milad Eyvazi Hesar
There are numerous benefits to utilizing the Transformer architecture over LSTM RNN. The two chief differences between the Transformer Architecture and the LSTM architecture are in the elimination of recurrence, thus decreasing complexity, and the enabling of parallelization, thus improving efficiency in computation.
Kind Regards
Qamar Ul Islam
Main concept of such tasks are irrelevant features elimination. See e.g., its advantages: https://www.youtube.com/watch?v=lWkFhVq9-nc
https://www.youtube.com/watch?v=8Fwy7raYZwA
Hi.,
Kindly Check this link:
https://dataspace.princeton.edu/handle/88435/dsp01kk91fp583
Best Wishes..
RNN/GRU/LSTM/etc.
- The current token impacts the next/succeeding token(s)
- The impact is smaller, the further away the succeeding tokens (What is short-memory; GRU/LSTM (and ResNets) are an attempt to circumvent the information loss)
- A recurrent kernel can delay information flows, i.e. one input spike at t=0 might generate an output sequence pattern. However, this very theoretical and is practically not accomplished by dense recurrent kernels.
Bi-directional RNN/GRU/LSTM/etc.
- The current impacts the previous and succeeding tokens, i.e. we read from left-to-right and right-to-left
- The impact is smaller the further away the tokens are from the current token (see above)
Transformer
- Every token impacts any other token
- Tokens are directly connected (i.e. no short-memory effect due to degenerating informations)
Positional Embeddings
- The transformer connects all tokens with each other, but we must specify the position of a token explicitly (In RNNs we implicitly know that all information comes from the previous token).
- Without the position information the impacted neurons wouldn't know where the information comes from.
Masked Attention Matrix:
- Transformers are bi-directional by default (e.g. BERT). If you want to impose unidirectional information flow (like plain RNN/GRU/LSTM), you can disable connections in the attention matrix (e.g. GPT)
- Badges
- Science topic