I'm learning how to use the Recurrent Neural Network model (RNN). I'm not entirely sure about the feed-forward procedure in RNN. It includes, for example, input, hidden state, and output. As far as I know, the hidden state is a type of multi-layer perception (MLP). However, in this case, a hidden state is derived from both current input and a previously hidden state. Unfortunately, "we can note that everyone reported the total number of memory cells, but no one specified the number of neurons inside each memory cell," which confuse me.
Second, I am confused by the RNN backpropagation procedure. I searched Google, and everyone only mentioned the generic steps (calculations of gradients) but no one conducted step-by-step backpropagation on example. I'm desperate for the entire RNN training process (two iterations is sufficient), even on a single layer with three to four memory cells.
Can anyone available concrete examples of the RNN model training process?
I found a good blog it could be useful
https://towardsdatascience.com/recurrent-neural-networks-by-example-in-python-ffd204f99470
Actually, RNN, and LSTM are good, However they will be of the past
The Future is for the transformers, and the attention mechanism, which make more sense.
@Ahmad M.OS thank you for the answer. Yes, nowadays attention and transformer are highly useful. But it is comes as replacement of RNN, am I right. So to make a solid foundation, I want to take in depth concept of RNN.
Dear Nafees Ahmad ,
Backpropagation Through Time, or BPTT, is the application of the Backpropagationtraining algorithm to recurrent neural network applied to sequence data like a time series. A recurrent neural network is shown one input each timestep and predicts one output. Conceptually, BPTT works by unrolling all input timesteps.
https://machinelearningmastery.com/gentle-introduction-backpropagation-time/
Regards,
Shafagat
may be this video helps you https://www.youtube.com/watch?v=bVGdxHgxG34&ab_channel=ColinReckons
- Badges
- Science topic