The aspect of over-fitting is typically viewed from the perspective of both- accuracy and model complexity.
To mitigate over-fitting, we usually have the practical approach of having k-fold validations, training-validation-test set.
Question: theoretically, can we leverage the Statistical Learning theory (COLT) to draw bounds on the confidence of how well learning has happened,
and how well classification can happen over unseen examples?
Example, many a time we consider the minimum number of samples (upper bound/sufficient samples) needed to learn (VC Dimension).
Agreed, it is an overestimate, plus |H| or VC Dimension may be unknown in practice.
The other perspective wherein, given the number of samples 'm', and 'delta' (probability of failure) -
we find error bound (on unseen examples)- Can we theoretically interpret this error bound to be an estimate of over-fitting?
Appreciate any pointers.
Typically this generalization bound you are talking give you a bound on the estimation of your algorithm. In other words, this is the component of the error which is due to the fact that your are learning from a finite sample and so your algorithm might fail in identifying the hypothesis that optimize its objective.
A large estimation error is correlated to a likely overfitting. As an example, the estimation error resulting from the VC analysis is something proportional to \sqrt{\frac{VC(H) + \ln(1/delta)}{m}}. If the sample size m and the confidence parameter delta are fixed, then using more "complex" hypothesis classes leads to a larger estimation error. What is happening is that enlarging the class H gives you more opportunities to fit well the training data but not hitting the optimal predictor in the class. The effect is likely to be a small training error but an true error, namely overfitting.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Learning