I have been doing a lot of experiments with ResNet and Inception lately... I was able to get decent accuracies. However, no signs of convergence even with over 100 epochs... Or is 100 epochs not enough?
I tried many different combinations of hyperparameters already. It seems to be not converging in anyways. I reviewed my dataset and even tried on other sets but no good.
I'm implementing through Keras btw. Hoping to get some good discussions with everyone. Thank you!
Training of Deep CNN is based on non convex optimization. There is always a chance that CNN gives you high accuracy on validation set in one epoch and then accuracy decreases in next epoch. For training CNNs, I always use callbacks (Keras) to save the parameters corresponding to best accuracy on validation set. You can also try to anneal learning rate during training.
Abhishek Kumar Sinha hello friend. Yes I tried it all... Yet, it does not seem to make too much changes. Even with the different methods of augmentation, data cleaning, tuning with random search does not make ResNet converge in Keras
Can we have more details about your dataset?
Also, you can try to overfit your model on only 1 observation. If you have a working algorithm, your CNN will of course overfit and achieve 100% accuracy.
I had a similar problem with resnet not converging. By trying to overfit I saw that even on 1 observation the model didn't performed well. It was just a mistake in the cost function.
Clément Poiret I'm using an image dataset with 4000+ training and 600+ validation. It fits well for both acc and loss using other models. However, with ResNet and Inception its really not doing well. I'm using sparse categorical cross-entropy. I tried using others and ended up with the same non-converging results.
Francis Jesmar Perez Montalbo So, if you can overfit on a small sample of your training set, if it's okay with other models, and if your resnet/inception are correctly implemented, the problem may lie in your hyperparameters.
You can try different optimizers (RMSProp, Adam, etc...), tweak momentum/learning rate, or even the architecture of the network (eg.: trying resnet v1, v2, building a deeper resnet, etc.).
If you want, you have that kind of libraries to help you explore the hyperparameter space through Bayesian Optimisation : https://autokeras.com/ (if you're on pytorch, you have similar libs too).
Otherwise, if your parameter space is non-convex, your optimizer will of course have troubles to find an appropriate local minima, leading to convergence problems.
What are your scores for this problem using resnet or inception? And with other models you tried?
- Badges
- Science topic