Dear All,
I am interrested in how the input's temporal sturcture of a recurrent neural network (or LSTM/GRU) should be constructed? Generally we have a 3 dimensional tensor (matrix) with dimensions
(number of samples) x (number of features) x (timestep)
The (windowed) timesteps should overlap or should not? If they should, how much? Are there any general guideline?
Eg. timesteps are overlapped a possible input:
|123456|
|234567|
|345678|
or
|123456|
|456789|
...
E.g. timesteps are not overlapped:
|123456|
|789abc|
...
Thanks a lot,
Bálint
In my experience it is based on two criteria that I have followed since I read [1]. Note: this is a very old reference but has given me guidance on dealing with this issue.
1) It is domain dependent.In the reference the window overlap is based on the contribution
2) It depends on the sampling frequency of the data and its dependency with the previous sample contribution.
Hope this helps
[1] "Parallel Networks that Learn to Pronounce English Text", by Terrence J. Sejnowski and Charles R. Rosenberg
Thank you very much for your answer and for the great paper!
According to your experiences, is there any simple relation between sampling frequency and overlap? Eg. higher frequency data -> less overlap is better.
Bálint,
My conjecture would be on a negative, but it could in principle apply under constrained circumstances.
Where I think it may apply is, if you are thinking of time delay NN or even shift register typed NN. The reason being is that the output of these types of NNs does not impact the next input of the NN. If noise is added due to increased frequency resolution (as opposed to a sample shift) it has limited propagation on the training( Note: higher overlap-> higher frequency definitely does not hold)
On the other hand NNs such as Elman, Jordan, Temporal Flow, Bidirectional,... have intermediate nodes and outputs that provide feedback and may impact more than the next sequence. The increased frequency resolution may act as noise that may propagate through the unrolling.
Again, take this only as a conjecture, since this would require testing as well as theoretical analysis
Hope this helps
- Badges
- Science topic