My svmlight output for binary classification (+1 for true and -1 for false classes) is like
+1 0.001
-1 -0.34
+1 0.9
+1 055
-1 -0.2
I tried generating roc in R using ROCR package. But I found in all examples of ROCR TPR ,FPR and cutoff values were used. TPR and FPR can be calculated from the above data but will get only one value and also w.r to cutoff I am confused. In this case cutoff is -1 to +1 right?
Can anyone help me how to give the values from the above data for drawing ROC in ROCR package.
I can't comment on R or ROCR, but it's best to explain what's actually going on in an ROC curve.
First, sort the predictions based on the prediction score:
true prediction score
+1 0.9
+1 0.55
+1 0.001
-1 -0.2
-1 -0.34
Then, apply a threshold to the scores to bin the predictions, so that TPR and FPR can be calculated. Say, we set a threshold of 0.3:
true prediction score prediction
+1 0.9 1
+1 0.55 1
+1 0.001 0
-1 -0.2 0
-1 -0.34 0
This gives TP=2, FP=0, FN=1, TN=2, TPR=2/3, FPR=0/2
Then, for every possible value of the cut-off or threshold, compute TPR and FPR.
You'll notice that the TPR and FPR won't change for threshold values between (here) 0.56 and 0.89, between 0.01 and 0.54, etc, because it does not affect any of the sample predictions.
Therefore, you need to calculate only the FPR and TPR for threshold values of 0.9, 0.55, 0.001, -0.2, and -0.34.
Each of these threshold value will give you another data point (TPR/FPR) to include in the ROC plot.
true prediction score ~threshold TP FP FN TN TPR FPR precision recall
+1 0.9 0.89999 1 0 2 2 1/3 0/2 100% 33%
+1 0.55 0.54999 2 0 1 2 2/3 0/2 100% 66%
+1 0.001 0.00099 3 0 0 2 3/3 0/2 100% 100%
-1 -0.2 -0.2001 3 1 0 1 3/3 1/2 75% 100%
-1 -0.34 -0.3401 3 2 0 0 3/3 2/2 60% 100%
The ROC for your data will be the perfect curve: it goes straight from the origin to the 100% sensitivity (TPR) to 100% specificity (or 0% '1-specificity' (FPR)).
It was really helpful. Thanx a lot for step by step explanation.
Now with this I think I can draw ROC in excel itself.
But then one doubt remains - how to calculate AUC.
Use some method to get your data as a dataframe. I defined it directly as a 2 column table (you can use also read.csv() to import a tabble) . 1 point is added at the end to escape from the trivial case (AUC=1) :
library(ROCR)
d
Thanx for the reply. But have a doubt.
This is the code which I used in R to generate ROC and AUC. I got the curve.
dat
Hello,
1) First of all there are syntax errors. You can't access columns "TRP" and "FRP" using "dat.TRP" and "dat.FRP". You need instead "dat$TRP" and "dat$FRP";
2) More importanly, the 2 columns that "prediction()" needs are: a) prediction column (which is some score or ranking, e.g probability) ; b) "labels" - these are the (true) classification values which in your case are -1 or 1. Thus the TPR/FPR have no place here at all.
3) The ROC curve is drawn by detecting ALL possible cut-off points automatically. You do NOT need to supply them.
I suggest you try again the code I posted (I don't think you can get anything from the your code - I tried) and read the help for function "prediction". Hope this helps!
1) Actually, you can draw the ROC curve from your data "dat" just by using:
> plot(dat$FPR, dat$TPR, type="o")
but computing the AUC requires more steps. Then why just not use the original data (from your 1st post) in the "prediction" and "performance" functions?
2) The reason you're getting the same AUC with 2 different datasets might be that the columns in your 1st example have different lenghts (you probably missed to type a value) and the dataset "dat" is not created. So, if you run your code a couple of times and use the same name "dat" for the data frames only one (the 2nd) will be created and that's why you'll get just one value for AUC (for the dataset that was created out of the 2) while thiniking you have 2 different sets.
3) To overlay 2 ROC graphs you might use par(new=T) :
> plot(dat$FPR, dat$TPR, type="o", xlab="", ylab= "")
> dat1 par(new=T)
> plot(dat1$FPR, dat1$TPR, type="o", col="red", xlab="FPR", ylab="TPR")
Thanx for the help Mr.Iordan . Actually with lot of browsing I corrected the errors and could also manage more than one curve in ROC. Got the plot of multiple curves. But AUC in R was giving problem. So finally I managed with excel which has a formula to calculate AUC.
For AUC in R I tried RocAUC, pAUC but nothing worked. I was giving for each dataset different names.
Thanx Mr Nazar for suggestion. Have installed LibSVM but did not use it.
Hi Hemalatha,
Could you tell me how did you use SVMLight to get result as
I converted my training data in SVM format, trained it using svm_train. To test used svm_classifier.
I am also doing my test in that way. My training data's target is just 1 or -1, so the output file generated by SVM is also -1 or 1. I can't get the floating numbers as yours. Could you tell me more detail? And do you know what is the meaning of these floating numbers? Are you using a continuous target value? Thanks!
My output file is like:
1.7024454
1.3008905
0.16843932
-0.042523802
-1.9269312
-1.4695678
-1.6725906
I did'nt do any settings. Depending on my testing set I used to calculate accuracy.
I did ROC curve manually. Any method is available in SVMlight have no idea.
Can we do cross validation in SVMlight???
Here is a possibility to plot the ROC curve and calculate the AUC (Area Under Curve).
Assuming the INPUT_FILE is the file you gave to test your SVM. It looks like:
+1 0.001
-1 -0.34
+1 0.9
+1 055
-1 -0.2
And the resulting OUTPUT_FILE looks like:
1.7024454
1.3008905
0.16843932
-0.042523802
-1.9269312
-1.4695678
-1.6725906
Then you can use the following commands in R (install the ROCR package first):
R
library(ROCR)
labels
- Badges
- Science topic