My colleague and I are training a custom SSD Inception v2 model and running into some research related issues.

In order to not create a duplicate post, I'm sharing the link to original posting here (from stackoverflow).

Link: https://stackoverflow.com/questions/61091655/what-is-a-good-learning-rate-graph-to-gauge-training-peformance

Brief summary of the problem:

we are training a SSD based object detection network from scratch. The dataset is comprised of around 250,000 images. It has a bias for some majority classes, but minority classes are also well reprsented (> 2000 occurences).

The current issue is that the model is not training well. Even after 150k steps, it has only reached 8% precision and 25% recall rate. The learning rate graph is also not particularly smooth.

What kind of learning rate graph is generally expected and what other approaches can be used to further improve the training? graphs and other resources are attached ...

Anyone has any experience with the above problem? and what are your 2 cents on tackling such issues.