Hello
I am searching for the latest trends in the long term memory neural networks for sequence learning. If you know any model (recurrent or not) good to learn from a large sequence thank you for sharing it with me.
Cordially
Salah Eddine Ghamri
Look the link, maybe helpful.
https://www.researchgate.net/post/What_is_the_best_approach_to_select_input_variables_in_Artificial_Neural_Networks_ANNs
Article Output Range Analysis for Deep Neural Networks
https://medium.com/@pushkarmandot/build-your-first-deep-learning-neural-network-model-using-keras-in-python-a90b5864116d
Hi Salah Eddine Ghamri
You may use TensorFlow or Keras packages in python for example, to access latest version of many DNN methods for your data.
see :
https://keras.io/
https://www.tensorflow.org/
for more information.
In principle there are neural networks with internal (LSTM, GRU) or external memory usage (Memory augmented NN). Alex Graves is one of the leading scientists in this field. Also you can use use non-memory models like masked or dilated CNNs or attention based NNs to process sequential data. LSTM and GRU are provided as standard blocks in DL libraries. It merely depends about what relationships you try to model, so only inbetween your time series or over long time intervals. Typically LSTM or GRU are more or less complex filters able to track and memorize changes over maximum 15 or 20 time steps. Attention mechansims can extend this to 40 or 50 time steps. Beyond that i suggestiv to incorporate external memory modules. But this is just my suggestion.
André Kummerow Thank you for your answer. It is quite pointy. I didn't get what are the external memory modules ?!
Well these are matrices where a NN can read from or write to which gives Access to past information in a direct way similar to attention mechanism. But these models are very hard to build and train.
On top of what André Kummerow said, I'd mention that Transformers seems to be the SOTA when you have tons of data and computation (https://arxiv.org/abs/1706.03762). When the former is lacking, I would definitely give a try to QRNN https://github.com/salesforce/pytorch-qrnn/
Mirko Torrisi
Thank you for this reply, I will check that.
cheers
if you can model your sequence data and represent it as a graph, then I suggest using the Graph Neural Networks, since they prove their capabilities in capturing the global temporal dependencies in the complex question answering task in NLP, especially a model called GRAPHFLOW.
Seeing time series as a Graph looks very similar to Markov chains. This would also be a nice approach. But this would be probabilistic model where inference can be hard to find compared to deterministic models like the Ones mentioned before. Correct me if im wrong
- Badges
- Science topic