Hi Krupal.

For 4 class (or mulitclass) classification problems, libsvm calculates the decision function a little bit more complicated. It basically calculates the probability for each possible class against each other class.

In your case (class 1, 2, 3, 4):

 1 vs. 2, 1 vs. 3, 1 vs 4, 2 vs. 3, 2 vs. 4, and 3 vs. 4.

Thus you get 6 decision values. For classification libSVM counts which class won the contest most often. E.g., if class 1 lost against 2, and 2 won against 3, and 2 won against 4 -> class 2 is our candidate.

The exact way how libSVM calculates it's decision function is coded in svm.cpp; function: double svm_predict_values(const svm_model *model, const svm_node *x, double* dec_values).

If you're using libSVM with matlab, I might provide you with code that calculates the decision values for single- and mulitclass.

Thanks Frerk for the answer.

I know I am getting 6 decision value due to k*(k-1)/2 classes.As, LIBSVM is using one-to-one algorithm for multiclassification.

But, I don't know how to make changes in SVM.cpp or how to write seperate code in MATLAB which gives me the decision value for that particular class. As, my main problem is to find the value of f(z)=sum(y*alpha*k(x,z).

Its greate help to me if  you send me the code which calculate  the decision function f(z) of my predicted samples.

I am using LIBSVM with matlab.

Dear Krupal.

Attached you will find my matlab function model_predict_fast. It is very similar to libsvm model_predict. The parameter x is the features and the parameter model is the trained libsvm model. Because x might become really huge (>2GB) this function splits the features in several chunks if necessary. In the foor loop I first calculate the kernel product before I evaluate the decision function.

The output paramters p gives you the decision values, e is always empty, and y is the classification output.

This method ist called model_predict_fast because it predicts the decision function way faster than the c++ version from libsvm, mainly because I exploit matrix multiplications at its core and the c++ version has a very unfortunate storage system of the support vectors.

If you have any questions, feel free to ask :)

Edit: A colleague just found a bug in my code. If you already downloaded the file, please download it again.

Thanks  a lot !.

I am not from IT field. Basically I am from Mathematics background. So, please  excuse me if I am asking very trivial questions.

May I have to replace 

[predClass, acc, decVals] = svmpredict(test_labels, KK, model);

by  your predict function.

where  KK = [ (1:numTest)' ,rbfKernel(test_features,train_features)];

If so, then what is the meaning of   ~  sign. what should I have to write in place of this sign. 

Yes, it can simply replace svmpredict.

But KK are your test features only, nothing else. The rbf kernel will be computed in  model_predict_fast.

Parameters with ~ are just placeholders. I.e. the test_labels you pass to my function will not be used by matlab at all. I basically do that to maintain the same parameter list as svmpredict. You will also see that your return parameter acc will always be empty.

May I ask you why you want to compute the decision function by yourself? In your parameter decVals you already got it. My function only speeds it up. Otherwise it provides exactly the same results just as svmpredict..

First of all thanks again for your quick reply.

Actually, I want to calculate value of decision function for ecah class. From FAQ I read that that distance of a point to the hyper plane is |decision Value|/|w|. So, I want decision values for each class. But, using one-to-one algorithm it is difficult to decide the decision values of a particular class. 

So, now I am trying to modify my code using one against rest algorithm. 

I hope it will slove my problem.I have results of four classes.

Only thing now I am trying to calculate value of f(x)=sum(alpha*y*kernel) using following code which is not using decision values and directly giving me the value of f(x).

I am using following code. 

[m,n] = size(model.SVs);

w = model.sv_coef;

b = -model.rho;

for i = 1:m

d(i) = norm(model.SVs(i,:) - v);   % v a test vector [1,n],n=no. of features

t(i) = exp(-sigma* d(i) .^2); % RBF model, t is 1*m vector

end

s=w*t+b;

Will you please tell me how I give the vector v. It seem that it is a single row vector. So, if I am giveing v(j,:), where  j run from 1:no of traing examples. Then it is dimension mismatch problem. 

I had to think about your question a bit.

I think what you want is not possible:

If you train a multi-class svm, there is not a single hyperplane. There are multiple hyperplanes. In fact, there are k*(k-1)/2 hyperplanes. You will not be able to calculate a one-vs-rest distance based on a multi-class svm. But: I'm not 100% sure if there is really no possibility. It might work, if you find a method to combine the support vectors in such a way that you're able to calculate a one-vs-rest distance. But I'm not aware of such a solution.

The usual way to calculate a one-vs-rest distance is by training k times a 2-class SVM. In each of the k training procedures, you train class k as positive class and all others as negative class. You will now get k different svm models. Each trained as one-vs-rest.

Regards,

Frerk

Frek, 

I am convience with your suggestion that in multi class we don't have a single hyperplan. Actually I am trying to learn the method which is given in a paper

In this paper they have used f(x):decision function. So, my main problem is how to find f(x) for different classes.I don't know how to write code. 

May I have your email. I will send to the paper. So, that you can understand my problem in a better way.

Dear Krupal,

I'm not giving private lessons. You have to learn how to program by yourself. There are plenty tutorials out there and especially matlab is a quite good language to start with. If you have detailed questions about a paper, please consider contacting the authors or posting the link to the paper to discuss it within the researchgate community. Especially in unfamiliar research topics it is very hard to understand the papers. But if you keep reading the paper (and related papers) you will finally understand it.

Sorry! 

One more question.

Why  the accuracy is different  if I train svm using one against rest algorithm?

What do you exactly mean with accuracy? Obviously the decision values will differ, because the hyperplanes are totally different. The resulting label should not differ too much but this depends on the dataset entirely. The multi-class svm should perform slightly better because you get more dicison functions (for the price of higher computational costs) with a better generalization especially for imbalanced data.

Whatever last accuracy inbuilt svm_predict function is showing is the overall accuracy of the classifier? am I right?

Actually libsvm is using one-to-one algorithm for multiclass problem. I am using one-against -rest algorithm and use libsvm svmtrain and svmpredict functions. So,I got 4 values for accuracy. I think these are the accuracy of each class. Now, to find overall accuracy of the classifier what I should do?

Your are right. You get the accuracy for each class and this is your result. As far as I know you usually do not combine the accuracy of all classes. I recommend you to calculate the confusion matrix too. At the beginning it's easier to interpret the results.

https://en.wikipedia.org/wiki/Confusion_matrix

you can try

>> mex -setup

and see ....

if give you this message ..

MEX configured to use 'Microsoft Visual C++ 2012 (C)' for C language compilation.

Warning: The MATLAB C and Fortran API has changed to support MATLAB

variables with more than 2^32-1 elements. You will be required

to update your code to utilize the new API.

You can find more information about this at:

http://www.mathworks.com/help/matlab/matlab_external/upgrading-mex-files-to-use-64-bit-api.html.

you need that compile