If we limit (4) to adding a penalty term to the cost function, I would add:

(5) Early stopping

(6) Ensembling, especially with different training data partitions

(7) Adding noise to the training data

Note that most methods, in one way or another, are forms of regularization. Equivalent mathematical equations can be derived theoretically (ex. Training with Noise is Equivalent to Tikhonov Regularization "https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-tikhonov-nc-95.pdf"), though it's not always practical.

Arun Anoop Mandankandy Ceh Chfi Khalid Youssef

decreasing the complexity of your network also could be helpful for overfitting , and using methods for few data if the problem is with low data:

https://www.researchgate.net/post/Deep_learning_for_small_datasets#view=60b74f0160132f6dd209ec19

Dear Arun Anoop Mandankandy Ceh Chfi ,

I would divide the problem in three steps:

- what is overfitting ? It is an excessive adaptation of the model to the training data, thus leading to a poor generalization capability.

- how to verify whether overfitting occurs ? you can train and validate your model while increasing the training set size, then plot training and validation error vs training set size. If the training set error is much lower than the validation error, you probably have overfitting.

- how to reduce overfitting ? I think your answers are pretty exhaustive.

You have overfitting when your model is much more complex than necessary, as Amin Honarmandi Shandiz said, thus you can reduce the complexity of the model. If you are dealing with deep networks, you can do so by reducing the number of hidden layers, number of neurons, and number of filters, or you can use dropout.

If you are dealing with shallow machine learning models, probably you have too many features, a small training set, or a not robust model (support vector machine and ensembling may solve the problem, as Khalid Youssef said).

Also, increasing the training set size is a good practice but it is often time-consuming, thus data augmentation is preferred initially (be careful on this, as data is synthetically generated starting from data belonging to the training set).