Hi. I am working on a prediction task. I have a bunch of images with bounding boxes and I want to predict the future bounding boxes. I am using a bi-directional encoder-decoder RNN with an attention mechanism. The size of the hidden layer is 512 and the number of layers is 3.
The input to the RNN encoder is a tensor of size (seq_len, batch_size, input_size). For the moment, I am using a batch_size and seq_len of 5. On the other hand, the input_size is 74 (I extract a feature vector from each image in a sequence and concatenate with a bounding box vector).
So, my question is, is my approach reasonably correct so far? While training my neural network, the loss function starts at an incredibly high value and decreases very slowly (and sometimes does not decrease at all). Is my architecture flawed or am I making mistakes in my design?
Hi,
You need grid search method for example to tune the hyperparameters of your rnn.
Hi,
The decrease in the loss function values should depend on the type of optimizer you are choosing. Perhaps, rather than changing the number of layers, and neurons in each layer, you should first run the baseline model and find which optimizer works best with your data. And the as Dairi Abdelkader suggested, a grid search could help you find the best (of the values in grid) model architecture.
Hi Sarim,
Even though there are some certain methods mentioned in the literature regarding the hyperparameters, often empirical method is adopted.
Moreover, visualisation would also work in this case. For the number of hidden layers, for example, loss function can be plotted against the number of hidden layers. Then, sort of elbow method can be used to determine how many layers you should have in the network. But please keep in mind that you should always be aware of the overfitting problem; huge amount of hidden layers and too many training samples could be some reasons of it.
Regards,
- Badges
- Science topic