If (1) the over training manifests itself in failure against instances that are new or excluded from the training set, and if (2) OOD instances are problematic because they are new and by definition excluded from the training set, then aren't instances of OOD really just evidence of over training?
The rephrasing of the question can be: How do we know if the training set + the method we apply to learn can achieve OOD generalization.
Let's hypothesize that you don't have access to OOD data over your training set and you want a system to run in a real environment where OOD data can skew the predictions with random results.
The solution can be actually driven by hedging the system to learn robust features from your training dataset, whereas over-training, is more suitable that you push your classifier to learn the noise of the observation.
Long story short; It's the combination of domain expertise and the application of Occam's razor that would drive the correct answer to this problem.
A good paper that can provide some additional intuition is the following: Preprint Adversarial Examples Are Not Bugs, They Are Features
Ronald Kessel It may seem so because you can have a case where the model fits well to train data and not new ones outside the training and testing distribution.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training