I am using neural network for medical image classification.(two class: patient and control ).
The problem is that train error is very low and it can classify training data with 90% accuracy but the test error is high.
The accuracy for test data is around 55%.
Hello,
It could be that you are overtraining the neural network so that generalization capacity decreases. To avoid this effect, you might use early stopping: training is stopped when the error on the validation set starts to increase. Doing this, you're keepin generalization capacity.
Hi,
Maybe the neural network overfits (there is too many neurons; complexity of neural net is greater than the complexity of the problem, see overfitting in Wikipedia). The second reason is that the training set does not represent the target function (big noise in data, too few points). Use crossvalidation.
In general, you'll never find a test error comparable to the training error. This is because you use the training set to train the classifier through a given number of learning epochs. So your NN knows very well the data it learned. However, the training error is not a measure of your NN performance since, after training, you need to assess its generalization capability through the test set which gives you the true classification performance. This latter set is composed of samples the NN has never seen before so the error will be higher w.r.t. the training error. Obviously, as Pierre Lauret and Grzegorz Dudek correctly wrote, if you used the wrong model parameters you could get two potential problems: (1) the NN model overfits the data, so you get very low training errors but the model is not able to generalize well on unknown samples thus providing high test errors (this is your case); (2) at the opposite, the model underfits the data so you get high training errors and test errors. To address these problems be sure to use a well balanced NN model (neither too complex or too simple w.r.t. your data ), then use a validation set (if you have enough data) or crossvalidation. Please refer to:
C.M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.
Hope this helps you.
Crossvalidation is the procedure for evaluating the performance of your net on new data. 10-fold crossvalidation is as follows. You divide your dataset into 10 subsets of the same size (more or less). Then you train your net on combined subsets 1,2,..,9 and after training you test the network on 10th subset. Repeat this 10 times taking different subset as test one. Then calculate mean accuracy on these 10 test subsets. This is estimation of generalization error of your net. To prevent overfitting you can treat some of the training points (9 subsets) as validation points. You do not use this validation points for training but only for controlling the error during training. When you see that the validation error does not decrease as the training error or even increase stop training.
These are the basics when using machine learning.
Good luck.
I am totally agree with previous posts by Antonino and Grzegorz. Your ANN seems memorizing the data point (overtrained). For two class case as you mentioned, MLP with one hidden layer is appropriate. Vary the number of neurons in the hidden layer and record the performances in a table. Pick the best classifier, that is the one which gives the highest correct classification rate (CCR).
This is how I would try do it: First randomize the data, second, normalize the data, third split it into 3 parts (e.g., 60% for training, 20% for crossvalidation (CV), 20% for testing) Use TANH as activation function in both the hidden and output layers. Stop the training when error is minimum on CV set (early stopping strategy). make a confusion matrix for testing set and find the CCR.
Under this conditions, maybe the patterns presents a complex scenario to divide in the space.
- Badges
- Science topic