Deep neural networks have high representational capacity and have gained much success in recent years. However, with the ability of high representational capacity, it can also suffer from overfitting, especially when the data is not big. So, what are the strategies that could avoid overfitting deep neural networks (other than drop-out).
Some common strategies are:
- paramater norm penalty (weight decay): penalize large paramter values
- data augmentation: synthetically increase training dataset
- separation of training and validation data (but this is usually always done in deep learning)
Check output at different number of iterations for testing dataset and stop training at the point where performance degradation starts for testing dataset
Regularization (modifies the objective function that has to be minimized) and Dropout (forces the network to become dependent on certain neurons) are the two different techniques to avoid overfitting.
Thanks. There is also a need of selecting a stringent cost function. For example, in some cases, AUROC/ AUPR may provide generalized NN than simple Accuracy.
Steps for reducing overfitting:
- Badges
- Science topic