I want to do incremental training of a deep convolutional neural network (CNN) model as new classes are added to the existing data. The CNN model is initially fully trained for classifying, say, 500 classes with 1 million images. Now, new data is available which has 50 new classes with 10000 images in addition to earlier 500 classes.
This addition of new classes in the data will be an ongoing occurrence every few days or every week.
Incremental training: I want to avoid training with full 1 million + 10000 images because of the training time. Any technique using the new 10000 images (50 classes) + tiny fraction of original 1 million (500 classes) will qualify as incremental training. I am aware of 'catastrophic forgetting' in NNs; however, I am looking for way to work around that. Does anyone have experience with this type of training?
Dear Kiran Maheriya,
Look the links, may be useful.
http://www.robots.ox.ac.uk/~vgg/practicals/cnn/
https://www.microsoft.com/en-us/research/wp-content/uploads/2014/11/Error-Driven-Incremental-Learning-in-Deep-Convolutional-Neural-Network-for-Large-Scale-Image-Classification.pdf
Regards, Shafagat
you can save the weights of your pretrained network, then modify the network: remove the top layer (classification for 500), change it to classification for 550 classes, and start training with dataset of your size (since anyway, the re-training is done using stochastic gradient descent, which looks at small chunks of the data). only the final layers parameters need to be learnt from scratch, rest of the weights can be initialized from earlier training epochs (done on 500 classes)
@Shafagat: Thanks for the reply. I had seen the Microsoft Research paper (and I like it for other purposes). However, it is still not incremental training -- it is incremental learning. The actual training still uses all the data. A good approach nonetheless.
@Vikas: Thanks. The idea you suggest is basically transfer learning, and variation of fine-tuning. In fact, that is what I am doing right now as I haven't found any other alternative (hence my question!). However, this approach requires using entire dataset. Yes, the training does finish faster that the first time around since we only do fine-tuning (or only final output layer) and there are less layers to back-propagate. Also, this is not sustainable if new classes contain more information that require retraining more layers.
Basically, I am specifically looking for techniques that are truly incremental as specified in my question using (all of the new data) + (none of old data OR only a tiny fraction of the old data).
I am also interested in same concept where new labels are added incrementally without disturbing previous training. This will reduce training time by great margin on larger dataset like imagenet
As sort of same curiosity, in recommendation system, we may have not only new items(labels) added to the top layer dimension, but also new users added to the input layer dimension. I want to come up with a solution of dealing with it other than just embedding-based solution which is not usually feasible in some areas.
- Badges
- Science topic