I looked up this topic on the internet and haven't found a good method or technique yet. For the time being, I just used cross validation technique where you train a certain range of hidden neurons number and select the lowest validation error among them. Yet, this also trigger another question of how big is the 'certain range' is. I hope some experienced NN user can help me with this.
The question is not very fascinating, but you will find some attempts in this direction in old literature, e.g., Barron (1991) or Vysniauskas (1993). Early stopping is probably the most widely used regularisation technique for MLPs. In some circumstances, early stopping can be shown to be equal to MAP.
Thank you both of you for replying to my question.
@Patrick I had read the Vysniauskas (1993) paper. The paper is good but it is a bit too advance for me to understand. Sorry. Is there are anymore simpler techniques for beginner user like me? FYI, I had also use the early stopping method. But the method is used to avoid the neural network become over-fitting right?
@Mahdi Well, the problem here is to choose the "proper number of hidden nodes". I mean how big it should be? What is the minimum or maximum hidden number of nodes should be etc?
Early stopping can _then_ be used to determine how your approximation generalises over data that is not learned upon, and that measure varies with the number of hidden units!
Thanks Patrick for the clarification. But, I still in searching for the appropriate method for determine the best number of hidden neuron. I found a good website:
ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu But it did not offer any solid conclusion.
Train a large network and then prune with validation [3] using orthogonal least squares [1,2]:
[1] Chen, Cowan, and Grant, "Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks", IEEE Trans. Neural Networks, March 1991.
[2] S. Chen, S.A. Billings, and W. Luo, “Orthogonal least squares methods and their application to nonlinear system identification,” International Journal of Control, vol. 50, No. 5, November 1989, pp. 1873-1896.
[3] P. L. Narasimha, W.H. Delashmit, M.T. Manry, Jiang Li, and F. Maldonado, “An Integrated Growing-Pruning Method for Feedforward Network Training,” NeuroComputing, vol. 71, Spring 2008, pp. 2831-2847.
- Badges
- Science topic