While training a deep learning model I generally consider the training loss, validation loss and the accuracy as a measure to check overfitting and under fitting. And different researchers have different opinion on this. Some argue that training loss > validation loss is better while some say that validation loss > training loss is better. For example in the attached screenshot how to decide if the model is overfitting or underfitting. Is there a rule of thumb or an intuition which can help while deciding early stopping. Or any research study which talks about a similar trade-off?
Dr @Yaakov J Stein's comment was excellent.
As an engineer, I suggest you to look at data sets related to your problem. Applying data from benchmarks to your system can help to identify possible sources of errors.
Furthermore, the source and amount of data can impact results. If possible, you can preprocess your data to remove outliers or noise. Most young researchers feed a deep learning structure without reflecting on the nature of the data. Believe me, deep learning is excellent but it is not a panacea.
Another point is that the term "deep learning" applies to a multitude of architectures. They all have strong points and weaknesses.
Please, refer to the following links:
https://arxiv.org/abs/1901.06566
https://machinelearningmastery.com/introduction-to-regularization-to-reduce-overfitting-and-improve-generalization-error/
Validation error is a way of estimating the generalization (i.e., the error on arbitrary unseen data) without tainting the true test set. As such it should never really be lower than the training error, unless your validation set was incorrectly chosen to be unrepresentative of the total space (e.g., restricted to easy examples).
However, in practice sometimes various artifacts of the training process can cause the validation error to appear lower. For example, during batch training you might measure training while accumulating the error and the validation error at the end of the batch or epoch. Or, if you are regularizing, then you need to be consistent with handling the additional error term.
So, train until while the training error is still decreasing the validation error is what you want the test error to be. Alternatively, in special cases you might know (based on the amount of training data compared to the number of degrees of freedom of your network) how much higher the validation error should be than the training error and stop then.
Y(J)S
Dr @Yaakov J Stein's comment was excellent.
As an engineer, I suggest you to look at data sets related to your problem. Applying data from benchmarks to your system can help to identify possible sources of errors.
Furthermore, the source and amount of data can impact results. If possible, you can preprocess your data to remove outliers or noise. Most young researchers feed a deep learning structure without reflecting on the nature of the data. Believe me, deep learning is excellent but it is not a panacea.
Another point is that the term "deep learning" applies to a multitude of architectures. They all have strong points and weaknesses.
Please, refer to the following links:
https://arxiv.org/abs/1901.06566
https://machinelearningmastery.com/introduction-to-regularization-to-reduce-overfitting-and-improve-generalization-error/
There's a bit of a conceptual issue here. When you train, you are trying to find the network parameters minimizing a loss function (F) over your training set. When you validate, you put another dataset through your network and see how well it performs by using a cost function (G) that might not coincide with F (though sometimes it will). So the relationship between the two is not what you should be looking at.
From your screenshot, it seems that your train loss might be going down too fast (as it's roughly in the min after the 1st epoch). Have you tried lowering the learning rate or changing the optimizer?
Typically the validation loss is greater than training one, but only because you minimize the loss function on training data.
I recommend to use something like the early-stopping method to prevent the overfitting.
The results of the network during training are always better than during verification. How much better? To the same extent that the training sample is representative of the test sample.
I am agree with all of the above answers, however., I would like to suggest you a further follow up : https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/
When the input sample is applied to the training data it is also applied to the validation data, which is similar to the training data (i.e. class labels are known) but on which the training algorithm is not applied. So effectively it is like an unknown data but with known class labels. Initially the errors both for the training as well as the validation data are expected to decrease as the weights are adapted for convergence. But at the point where over-fitting starts, it will produce a small error for the training data (because of over-fitting), but large error for the validation data (because it is unknown data for the system). Because the class labels are known, we can calculate that the error is increasing and therefore we should stop the training process at that point, because it will behave in a likewise manner for the test data too (i.e. large error) and that is what we do not want.
if validation loss becomes better than train loss or you can say that as validation loss < train loss it is probably your validation dataset is weak. By the word weak I meant easily reachable/detectable.
Put some hard nuts to crack to the validation set and observe how the training is working.
Harsh Panwar
Assuming your train-val split is correct (distribution). Validation loss (as mentioned in other comments means your generalized loss) should be same as compared to training loss if training is good.
If your validation loss is lower than the training loss, it means you have not split the training data correctly. Correctly here means, the distribution of training and validation set is different. Ideally, it should be the same. If you plot training loss vs validation loss, some people say there should not be a huge gap in both the learning curves. Some say, if the validation loss is decreasing you can keep training no matter how much the gap is. You can try both scenarios and see what works better for your dataset.
Basic steps to consider:
1. train-val sets should be of same distribution.
2. Augument your data to reduce the trn-val loss gap
3. Regularization
4. Early stopping
Remember that the train_loss generally is lower than the valid_loss. Because the validation dataset is used to validate de model with data that the model has never seen. For this reason, the valid_loss generally is higher compared with the train_loss. If you are a lucky guy the values could be similar. You need to check if the model is using early stopping, I recommend you an early stopping with patience equal 2 or 3 to avoid overfitting problems.
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude