What parameters or factors are affected if I checkpoint and retrain the model from the same point instead of training altogether. Is there some kind of tradeoff between the two methods?
e.g. In place of running 40 epochs straight, I run it 4 times with 10 epochs each.
The parameters I was focusing on were :
1. Accuracy ( model performance)
2. Time for training
any others I should consider?
Hello,
theoretically, there will be no change (as per my knowledge) if we do 40 epochs straight or 4x10 times keeping checkpoints. Although the second option will be much handier for a huge amount of data as you can alter the configs between every checkpoint run. There is a process called model split train (or something like this where you can change configurations every time and apply it on same ckpt or pretrained weights). This process is quite handy.
Thanks.
Dear @ Ashay Singh,
I hope this tutorial help you.
Keras: Starting, stopping, and resuming training
https://www.pyimagesearch.com/2019/09/23/keras-starting-stopping-and-resuming-training/
Dear Singh
Stopping and resuming model training process doesn't effect Accuracy ( model performance) and Time for training by itself;technically speaking deep learning practitioners use model split training for two reasons:
1)your training session is eliminated and training procedure is stopped (due to a power outage, GPU session timing out, etc.).
2)modify your learning rate to improve your model accuracy.
good luck
- Badges
- Science topic
- Similar topics
- Biological Science
- Kinesiology
- Running