I am coding a neural network. I am following k fold cross validation technique to avoid overfitting. So, I divided dataset into 10 parts and ran 10 fold cross validation. But, I found that cross validation error is never decreasing and is continuously in. Can any one please tell some steps to follow in this case?
My dataset consists of 570 examples. It has 65 input features, and 1 target. The network has 10 hidden units. I am also using weight elimination method ( a regularization technique ) to reduce overfitting.
Your data are maybe not independent and identically distributed. When you learn on a part of your training set that is not representative of your validation set, the error increases. Have you tried to shuffle your dataset ?
Several factors may be the reason:
1- the percentage of train, validation and test data is not set properly.
2- the model you are using is not suitable (try two layers NN and more hidden units)
3- Also you may want to use less inputs (only those which are more relevant or statistically significant). for this many inputs, 570 instances may not be enough.
@Soutrik Banerjee, Thank you very much for your reply. I am using conjugate gradient descent with line search. So, I think there is no need to again search for good learning rate and momentum. The learning algorithm itself adapts it. Are there any other reasons why this can happen?
We usually call this 'over-fitting'.
One other reason might be-- too many input features. I think you can try some feature selection before going into ANN learning. That would effectively reduce the number of input features from 65 to, say, the range of 10-20.
Rough Set (using a reduct) is a good feature selector... but PCA is perhaps simpler.
Does the training error continuously decrease ?
If it is not the case, may be the regularization parameter is tuned to high.
Your data are maybe not independent and identically distributed. When you learn on a part of your training set that is not representative of your validation set, the error increases. Have you tried to shuffle your dataset ?
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining