K =10- Fold is the best choice is almost any data set.

Dear Gopi,

Did you use SVM for regression or classification purpose?

Generally, if you want to validate your SVM model (SVR(for regression)), it is recommended to use 20% of your whole training set (leave group out) and then measure RMSE of your generated model. You need to optimize also C, sigma and epsilon values and measure RMSE values for each value of them (which is a little time consuming). You may do leave one out (high precises) so that you could understand if the effect and presence of each of your data point is  influential to the final model.

In case of using SVM for classification purpose, you need to calculate ROC plot for your classification and decide whether your model works properly or not.

Hope you could find it helpful,

Regards,

Reza

Thank you  Ashraf and  Aalizadeh. 

Aalizadeh could you elaborate more based  on below information:

       I am using SVM for classification. I have two class (cancer and Healthy). I followed below steps

1. I developed  cross vallidated SVM model  CVSVM with Kfold =5.

SVM and QDA classification has below limitations for CV model :

http://www.mathworks.com/help/stats/fitcdiscr.html

Functions

classify is usually more awkward to use.

·        classify requires you to fit the classifier every time you make a new prediction.

·        classify does not perform cross validation.

·        classify requires you to fit the classifier when changing prior probabilities.

2. Due to above limitation I could not use developed CVSVM model directly to classify/predict my testing set of data.

3. Then I developed Compacted model   Compactmodel1= CVSVM.Trained{1}  (this gives me model 1). By replacing 1 by 2-5, I will get 5 diff models.

4. Then I can predict Test data using 

[label,score,cost] = predict(Compactcvmodel1,XTest)

same repeated for 5 models

5. I can get only "bias value"  for each compact model  and I will have predicted labels (classification accuracies for each model) to differentiate between models in SVM and not even that for QDA.

6. Now I have only option I select CV SVM model based on "bias value" and final testing accuracies for classification and for CV QDA I have only option to select based on final testing data accuracies for classification.

7.Other wise I just need to select option of  Holdout (20% data) instead of K-fold =5 so that I will have only one CV model. 

8. ROC plot is used generally in Artificial neutral network. I am not sure to plot ROC curve with SVM.

Thank you.

Dear Gopi,

Thanks for above information. I know this function in stats package, however I don't like to use it more often.

As I can see, you would like to use non-liner classification tools for the mentioned problem. There are some other methods such as PLS-DA or SOM which are far better than SVM in some situations. I would recommend to try writing m codes on your own for optimizing the above three parameters automatically for better results. 

Your intention can be done so easily and it is also common task. May I ask you about number of variables and samples?

I can perform you SVM (classification) suggested by Vapnik furnished with a lot of automatic procedures if it is vital for you. Actually, ROC plot is proposed for evaluating the accuracy of your classification regardless of the methods used. You can apply it based on the actual and predicted classes so easily and also can be another alternative for  selection of models. As you can see, in the following link, ROC curve was performed for SVM as well!

http://www.mathworks.com/help/stats/perfcurve.html?searchHighlight=roc%20plot

Regards,

Reza

Thank you Reza

My sample size is 120 (healthy-60 and Cancer-60). I have total features of 9 (X1, X2...X9) with reflectance data for each.

I need to follow with SVM.

What you say about reliability of selecting model based on Bias value?

eg. bias value for modle1(M1)=0.232, M2= 0.2168, M3=0.1283, M4=0.3211, M5=0.2698.

 Low bias value=0.1283 , so best model is M3. Is it reliable or scientific  to conclude like this?

Dear Gopi,

First, you need to do correlation analysis between your variables to remove inter-correlation ones and in other words, you need to find important variables out of these ones for simplicity and also preventing redundancy.

I think that PLS-DA will work for you better than SVM since you can include optimum number of latent variables based on Cross validation analysis. Unfortunately, to my mind, the bias will not be enough to show the best model and its robustness. Generally, in SVM (classification) to show the model works, we need to calculate ROC plot for each step of parameters (C, sigma, epsilon). The boundaries and number of supports vectors can be compared with accuracy of classification for internal purposes and for external validation, you can use test sets as you already mentioned.

Regards,

Reza

Thank you Reza.

Dear Gopi,

You're welcome.

Regards,

Reza

Easiest  solution for my Problem

cverror_average=kfoldLoss(CVSVMmodel)

cverror_individual = kfoldLoss(CVSVMmodel,'Mode','individual')

2nd algorithm gives error for each k fold models (1-5). Based on error best model can be selected (Kfold=5 or 10)..

Dear Dr Gopi,

it is preferable to take the average of the 10 fold, or you can try to take the best subset via voting using weight technique.

You can try the caret R-package in CRAN. Here are some R codes implemented to find the optimal parameters (Poly kernel function):

library(caret)

indx

Thank you  Rashid, Carrington and Nguyen.

In most of the cases/ problems and data sets 10-fold scheme is suitable