Refer to the attached image (Source: https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714)
Why do we need two extra sigmoids (marked with red in fig), when the outcome of the first one (marked with blue in fig) can be passed to the input and output gates as well?
Doesn't it increase the computational burden by requiring dedicated weights and biases for the additional sigmoids?
.
this should clarify the role of the three "gates" :
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
.
Dear Fabrice, Christopher Olah's blog is the most famous one on this topic. Just google 'LSTM tutorial' and this would be the first one to come up. You already know i am not looking for explanation on three different- "gates" but "sigmoid activation functions". I am looking for a profound explanation, not webpage links.
.
i do not understand the difference between "gates" and "sigmoids" ; moreover, the term "gate" is used by Schmidhuber and Hochreiter in their original paper ...
.
anyway, the original paper is worth reading
https://www.bioinf.jku.at/publications/older/2604.pdf
(apologies for my linkmania)
.
now why would you want the "blue" sigmoid, which controls how the previous memory is allowed to modify the internal state, be "passed" to the last "red" sigmoid which controls how the internal state is allowed to pass through the memory channel to the next cell ?
these are two different functions and depending on the circumstances, one gate could be open and the other closed
.
i must be completely missing your point ; thanks for reading and apologies for the useless ramble
.
Thank you for the reply Fabrice! I have already read that paper (plus many other blogs and related papers, but i am still unable to get a clear idea of LSTM functioning). From that paper, I understood a few things, but the mathematics is too complicated for me to understand.
For the difference between gates and sigmoid, I think the gates comprise of one or more activation functions, but a sigmoid alone cannot be considered as a gate. Forget gate is just sigmoid, but output and input gates are a combination of sigmoid and tanh functions.
The question: Sigmoids in forget and input gates take same inputs (C_t-1, h_t-1, and x_t) and squash the outcomes between [0,1]. So, why cannot we use the output of first sigmoid and directly multiply it with the tanh outcome in input gate? Is there a specific reason for having a dedicated sigmoid in input gate? To further clarify my question, i am posting a new image. This could eliminate the requirement of an extra set of weights and bias (and thus reduce some extra computation).
.
strictly speaking, the (soft) gate function is the "multiply" block in the diagram
the sigmoid, rather the sigmoid output, is the (soft) gating signal
and yes, we're talking "soft gates" here, not stictly 0/1 gates ; this may seem a misuse but it is traditional in the neural network community since ... the early days !
.
the weights of the sigmoid inputs are learned and therefore the three gating signals can be different at every step in time, depending on the cell context
here "cell context" means signal, output of the previous block and memory output of the previous block
yes, the cell context is the same for the two gates on the left (the context of the right gate does not include the memory signal of the previous block) but as the weights are different, the gating signals will be different
.
so, when considering the two gates controlling the input of the cell state (the two "multiply" blocks, left and center of the figure), it is clear that they have two different roles : the left one controls how much the memory signal of the previous block can affect the internal state and the center one controls how much the output of the previous block and signal can affect the internal states ; there is no reason for those two gates to receive the same gating signal : at a given point in time, context may have that one gate should be opened (softly) and the other closed (softly)
.
.
cool
i would suggest you go back to Colah's blog post now ; i bet the "somewhat clear" will at least become a "clear enough" !
.
Just to share with the followers of discussion.... a very lucid introduction on LSTM can be found on:
https://www.analyticsvidhya.com/blog/2017/12/fundamentals-of-deep-learning-introduction-to-lstm/
It gives various practical examples which helps in understanding the LSTM architecture in a better way.
Recommendations for further reading related to this topic:
1) https://www.analyticsvidhya.com/blog/2017/12/introduction-to-recurrent-neural-networks/
2) https://www.analyticsvidhya.com/blog/2017/12/fundamentals-of-deep-learning-introduction-to-lstm/
3) https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21
4) https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/
5) https://stats.stackexchange.com/questions/47590/what-are-good-initial-weights-in-a-neural-network
6) https://medium.com/usf-msds/deep-learning-best-practices-1-weight-initialization-14e5c0295b94
7) https://medium.com/@aidangomez/let-s-do-this-f9b699de31d9
- Badges
- Science topic