In the artificial neural network BP method, how are large training samples usually selected? 80% of the total set, 50%, or 40% and how to choose? If choosing few, the results are not ideal. How to solve this problem? Any comments will be appreciated.
Jie,
a rule of thumb in NN learning and more generally machine learning says that more learning data is better. However, this raises two issues:
First, using more data for training leaves less data for testing. Thus, you may have problems to reliably estimate the quality, say predictive accuracy, of your NN. Here, resampling schemes such as cross-validation, bootstrapping, etc. can help.
Second, you may face computational problems when trying to train a NN on a large data set. In this case, I find it useful to perform a learning curve analysis [Perlich, C., Provost, F., Simonoff, J. S., & Cohen, W. W. (2003). Tree induction vs. logistic regression: A learning-curve analysis. Journal of Machine Learning Research, 4, 211-255.]. Basically, you draw a relatively small random sample from your data to train a NN, and use it to predict a randomly sampled test set. Then, you iteratively increase the size of your training sample and keep predicting the same (!) test set. Track the degree to which predictive accuracy on the fixed test set increases with training set size. It's a heuristic approach but will often give you useful insight how much training data is really needed for your task. Depending on your task, you may want to repeat the learning curve analysis multiple times with different test set samples to further reduce variation in predictive accuracy.
I direct you to the subinterval approach introduced in my papers. This approach leads you to choose the number of the training samples based on the behavior of the neural network.
One way is to use about 80 to 90% for training and then use the unused 10 to 20 % for testing to see how good your training works. Of course, the ultimate success depends on the intrinsic separation between the two classes.
1) An effectiveness definition of the separation needs the subinterval approach. 2) The solution refers to over fitting problem, however the main question, I think refers to an effectiveness training for large samples.
Jie,
a rule of thumb in NN learning and more generally machine learning says that more learning data is better. However, this raises two issues:
First, using more data for training leaves less data for testing. Thus, you may have problems to reliably estimate the quality, say predictive accuracy, of your NN. Here, resampling schemes such as cross-validation, bootstrapping, etc. can help.
Second, you may face computational problems when trying to train a NN on a large data set. In this case, I find it useful to perform a learning curve analysis [Perlich, C., Provost, F., Simonoff, J. S., & Cohen, W. W. (2003). Tree induction vs. logistic regression: A learning-curve analysis. Journal of Machine Learning Research, 4, 211-255.]. Basically, you draw a relatively small random sample from your data to train a NN, and use it to predict a randomly sampled test set. Then, you iteratively increase the size of your training sample and keep predicting the same (!) test set. Track the degree to which predictive accuracy on the fixed test set increases with training set size. It's a heuristic approach but will often give you useful insight how much training data is really needed for your task. Depending on your task, you may want to repeat the learning curve analysis multiple times with different test set samples to further reduce variation in predictive accuracy.
Jie Dou: If you are able to get "good" results using 50% of the samples in an SVM based approach, I guess that your samples are separable. If the samples are intrinsically separable, then you should be able to verify that using a BP algorithm for ANN as well. The number of sample needed to train ANN would definitely differ from those needed for SVM since the basic ideas of these two approaches are different. Pl. let me know your thoughts on this matter.
I just get published the paper "Sample selection via angular distance in the space of the arguments of an artificial neural network." This paper describes a new method for selecting a subset of samples to be provided to an ANN. Samples are selected based in the angular distance formed by them, discarding such ones that are too close to any already selected. As it is demonstrated, with a fraction of the total available samples, an ANN can be trained much faster while obtaining similar quality in the estimations. For the application that I used the method the training time was reduced by a factor of 12. It performs better than a selection based in K-means for the same number of elements.
sub-interval introduced in my paper as a new approach to lead an ANN to an effectiveness training. The results of my experiences address that the iteration is not capable of providing this effectiveness.
- Badges
- Science topic