The main results of LS-SVM model are so perfect but I'm hesitant about using LS-SVM , due to it's results that indicates over-fitting occurrence. Do you have any similar observations? Is it safe to use LS-SVM for data modeling? I haven't any similar problem with one layer Artificial Neural Networks(ANN).
Also i am not sure how important are sigma and gamma values? they are very far apart in any train of algorithm , but their primary results are very similar.
Thanks for your contribution
with regards
M.Keyhani
http://www.esat.kuleuven.be/sista/lssvmlab
Dear all,
If you select the regularization constant and kernel parameters in a good way, there will be no overfitting. The LS-SVM model will generalize well then.
In the LS-SVMlab software
http://www.esat.kuleuven.be/stadius/lssvmlab/
this is also automatically tuned for you. The use of regularization prevents overfitting.
If you would like to learn more about all of this, please consult the book
J.A.K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, J. Vandewalle, Least Squares Support Vector Machines, World Scientific, Singapore, 2002 (ISBN 981-238-151-1)
Best regards,
Johan Suykens
----------------------
Prof. Dr.ir. Johan Suykens
Katholieke Universiteit Leuven
Departement Elektrotechniek - ESAT-STADIUS
Kasteelpark Arenberg 10
B-3001 Leuven (Heverlee)
Belgium
http://www.esat.kuleuven.be/stadius/members/suykens.html
LS-SVM toolbox is very well done. It is including 10-fold CV and LOO evaluation. A smart optimization of the 2 hyper-parameters is done using for example 10-fold CV.
LS-SVM is providing a good "baseline" for
Hi Mohammed, SVM is usually regarded as the data modelling approach with the least potential for over-fitting. That's because SVM is just a linear model, or a hyperplane classifier. It is hard to imagine overfitting a dataset when all you can do is rotate a hyperplane. That said, you check for overfitting by dividing your data set into two parts training and test, usually in 75%-25% ratio, training SVM on the "training" dataset and then calculating the prediction error on the "test" dataset. If training dataset error is comparable to the test dataset error, then there is no overfitting. You have overfitting if the test error becomes much much greater than the final training error.
Drawing the training error and the test error as a function of a parameter, such as your second picture, is also helpful. If you have an overfitting problem, you will see the training and test errors first decrease together, and then diverge with training error continuing to drop and test error beginning to grow.
Dear Dr. Yuriy Mishchenko I see your short and sharp notes. They are highly useful for me. When i divided the data randomly into different parts (75% train :25%test) and run the algorithm the results improved and the over-fitting problem were solved. According to my observations mean absolute error of testing data are just 1.16 times higher than training data. I attached final results of the initial and modified LS-SVM in comparison of each other and ANN.
This discussion was very useful for me and i understand important notes : machine learning algorithms are useful but we should not trust them simplistic.
Thanks for your time
When training using any multivariate technique(LSQ, aNN etc.), the key is the continuity of the range of all parameters(inputs & outputs). Remember that you are trying to model a process you don't understand and want to apply this model to possibly infinite number of similar situations. Therefore, the training set should be as small as possible.
Dear all,
If you select the regularization constant and kernel parameters in a good way, there will be no overfitting. The LS-SVM model will generalize well then.
In the LS-SVMlab software
http://www.esat.kuleuven.be/stadius/lssvmlab/
this is also automatically tuned for you. The use of regularization prevents overfitting.
If you would like to learn more about all of this, please consult the book
J.A.K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, J. Vandewalle, Least Squares Support Vector Machines, World Scientific, Singapore, 2002 (ISBN 981-238-151-1)
Best regards,
Johan Suykens
----------------------
Prof. Dr.ir. Johan Suykens
Katholieke Universiteit Leuven
Departement Elektrotechniek - ESAT-STADIUS
Kasteelpark Arenberg 10
B-3001 Leuven (Heverlee)
Belgium
http://www.esat.kuleuven.be/stadius/members/suykens.html
- Badges
- Science topic