I am currently coding a neural network on weather data. So, I have only 570 training examples. But, the relation between output and input is highly non linear. How can I reduce overfitting on this data. I am using conjugate gradient descent with line search.
My input dimensions are 125. I have 1 target. Our project is to learn complex features and hence we are using an auto encoder. We can't shift to SVM due to this reason. I am using weight elimination penalty method for reducing overfitting. Is there any other penalty method that better fits this situation? I have lot of hyper parameters in my network. Can you please suggest any good way to reduce hyper parameters?
I think you should reduct the dimension of the data using feature selection or extraction method like principal component analysis (PCA). I do not know your data but some features can be correlated and some of them can not be correlated with output variable. So try eliminate some of them using a method of feature selection (filter or wrapper method) or try to use PCA to create new features on the basis of original ones.
Good luck.
As one of the previous comments I also would suggest to incorporate regularization techniques. Depending on the problem this might be l1-regularization for regression problems. For classification, relevance learning as introduced for learning vector quantization might be a good alternative.
the best way might be using k-fold testing approaches. when the data size is small, this is the optimum way to reduce the over fitting problem. keep me updated if you could not find any reference about this approach.
@Soutrik, Extremely thank you for your reply. But, why isn't line search optimal when there's interaction between parameter values.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining