I am having an issue while training a custom deep learning model. The training accuracy increases gradually until 0.82 then it drops back to 0.62 for every epoch. The model uses residual blocks with batch normalization and dropout probability of 0.1. The optimizer of the model is Adam and the learning rate is 1e-3. all the activation functions are Relu.
This occurs often in model training. You may try the following. 1) Gradually decrease the learning rate to 0.0001. 2) Add more data. 3) Gradually increase the Dropout rates to ~0.2. Keep it consistent throughout the network. 4) Decrease your batch size. 5) Try a different optimizer (choose one that gives you the smallest loss).
Hello Fathi,
This is almost inevitable as Anas said. Note that many parameters in deep networks are adjusted by trial and error including learning rate, momentum factor, size of convolutional filters in ConvNets, weight initialization, number of hidden nodes, number of epochs, batch size, etc. In addition to Eric's suggestions, I also recommend stopping training when a high enough accuracy has been reached, preferably near 100%. Accuracy of results also depend on the application, classification or regression. It may be helpful to see our paper on deep learning localization (link below). Simulation information therein is detailed in the Results section, including number of epochs for each simulation example and the localization accuracy obtained. Confining training period to that needed to attain high accuracy enables the ConvNet to work with real data very accurately.
Article Deep learning techniques for noise-resilient localisation in...
- Badges
- Science topic
- Similar topics
- Correction