for example for approximation of a 2-D function (2-D Gaussian distribution) a neural net with 2 hidden layer with 3 neurons in each layer is a much better approximator than a one hidden layer neural net with 30 hidden neurons?(at least for the function I attached)
the shape of the function is attached.
Hi,
When you increase the number of neurons, you increase the complexity of the search space. The NN may then over-estimate the complexity of the function it is trying to learn, causing overfiting and consequently bad generalization. You could check that ploting training/validation error curves. If you see a through the plot that the NN fails to converge, it is due to the increased number of local minimuns, which could be tackled using reinitialization and adaptive learning rate mechanisms.
Usually simpler is better, should you try a NN with one hidden layer and few neurons and you may get faster and accurate results as well.
Regards.
Dear Ricardo
In fact in my training I used regularization (Bayesian regularization) method for both one layer and multi layer networks.regularization methods avoid overfitting. So I do not think that overfitting is the problem.
Hi Ali,
Have you tried to plot the estimated function shape generated by both NN topologies to see how they look?
I'm not sure that Bayesian Regularisation guarantee no overfitting, although it may help. The best way to check it is plotting training/validation curves, then you can easily see if it has either overfitted or not converged. Since your function is not too complex and you are using Bayesian Reg, it's likely that the NN with 30 neurons has not converged. This could be solved by adding more examples for training.
Another point is that the accuracy may not have to do with the number of layers but the number of neurons. As you add neurons, which meas more parameters, the learning process becomes more difficult. I believe you can achieve results as good as your 2 layer NN, if not better, with a single layer NN with few neurons, let's say 3~8.
If you had a more complex problem, then you probably would do better with a more complex NN.
If you check the training/validation plot, please share the results with us!
Best wishes.
Dear. Ricardo
I think Regularization will guarantee no over-fitting, by the way, I have attached the estimated function shape with two structure,
1- one hidden layer with 30 hidden neurons
2- two hidden layer with each 5 hidden neurons
as it can be seen the two hidden layer gives a far more better approximation of the function.
also I have attached the normalized training data-set.
I simply used the Matlab neural network toolbox, and train the network with trainbr (Bayesian Regularization) function.
- Badges
- Science topic