For a predictive model the AUC gives an estimation of the overall threshold independent performance. Sensitivity and specificity on the other hand are threshold dependent. Hence if you need to know the sens and spec, then you'll need to choose a threshold and then compute the measures.

Hi Daniel, 

It is sometimes handy to use str function from R to extract the sensitivity and specificity because some R objects are nested inside a object (eg. list inside a list). For more info,  Please see the video from Roger Peng https://www.youtube.com/watch?v=_eOxjvsGcus

I am assuming that you are comfortable with loading R packages and calling the function. And for reproducibility, please see the folioing code. In the following, there were two packages that we use, RWeka and pROC by calling library function. Please note, I just library function instead of require function. See http://yihui.name/en/2014/07/library-vs-require/

We use popular data which is the iris dataset which suited your condition (i just assume), as it is a multi class (setosa, versicolor and virginica). J48 function from RWeka, an interface for Weka, very popular machine learning software. I have also copy and paste from results of str() function at the very end. Just for info, the line started with greater than sign (i.e., >) are the command in the R console. 

> library(RWeka)

> library(pROC)

> data(iris)

> fit predictor daniela str(daniela)

List of 7

$ response : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

$ predictor: Named num [1:150] 0 0 0 0 0 0 0 0 0 0 ...

..- attr(*, "names")= chr [1:150] "1" "2" "3" "4" ...

$ percent : logi FALSE

$ call : language multiclass.roc.default(response = iris$Species, predictor = predictor)

$ levels : chr [1:3] "setosa" "versicolor" "virginica"

$ rocs :List of 3

..$ :List of 14

.. ..$ percent : logi FALSE

.. ..$ sensitivities : num [1:5] 1 1 0.06 0.02 0

.. ..$ specificities : num [1:5] 0 1 1 1 1

.. ..$ thresholds : num [1:5] -Inf 0.0104 0.1771 0.6558 Inf

.. ..$ direction : chr "

Hi Saw, thank you for your answer but I am still confused. Using your code we just calculate the probabilities for the multiclass variable independent of my response - that I want to analyze with ROC. Let's say, how can I find the results using your code for Petal.length if I am assuming it as my response variable? I couldn't identify this in the code (in the output of str)

Hi Leon, yeah that is the point, how to get the sens and spec for this multicalss case, do you know how to do that?

Hi Daniela,  Because the iris data set has 3 class, the pari-wise comparison was used. Please see the documentation of  multiclass.roc function. And oh, for more info, please read the Advance R book from Hadley, which teaches about subsetting a list from a list. 

Hadley book. It is free. 

http://adv-r.had.co.nz

The documentation of the multiclass.roc function. 

http://www.inside-r.org/packages/cran/pROC/docs/multiclass.roc

The paper that was used to implement the multi class.roc function. 

http://link.springer.com/article/10.1023%2FA%3A1010920819831

And also see the documentation on the CRAN for the package

https://cran.r-project.org/web/packages/pROC/pROC.pdf

And also for the actual paper for pROC package. 

http://www.biomedcentral.com/1471-2105/12/77

If you find any bugs for the code,  or question about this function report that to the package maintainer. 

https://github.com/xrobin/pROC/issues

And for extracting the sensitivity, specificity and cut-off from the multi class.roc function, please see the following code. 

> data(iris)

> library(pROC)

> ok class_1 class_2 class_3 wants wants

sensitivity specificity cutt_off

[1,] 1.00 0.00 -Inf

[2,] 1.00 0.02 1.05

[3,] 1.00 0.04 1.15

[4,] 1.00 0.08 1.25

[5,] 1.00 0.22 1.35

[6,] 1.00 0.48 1.45

[7,] 1.00 0.74 1.55

[8,] 1.00 0.88 1.65

[9,] 1.00 0.96 1.80

[10,] 1.00 1.00 2.45

[11,] 0.98 1.00 3.15

[12,] 0.94 1.00 3.40

[13,] 0.90 1.00 3.55

[14,] 0.88 1.00 3.65

[15,] 0.86 1.00 3.75

[16,] 0.84 1.00 3.85

[17,] 0.78 1.00 3.95

[18,] 0.68 1.00 4.05

[19,] 0.62 1.00 4.15

[20,] 0.54 1.00 4.25

[21,] 0.50 1.00 4.35

[22,] 0.42 1.00 4.45

[23,] 0.28 1.00 4.55

[24,] 0.22 1.00 4.65

[25,] 0.12 1.00 4.75

[26,] 0.08 1.00 4.85

[27,] 0.04 1.00 4.95

[28,] 0.02 1.00 5.05

[29,] 0.00 1.00 Inf

> str(class_1)

List of 14

$ percent : logi FALSE

$ sensitivities : num [1:29] 1 1 1 1 1 1 1 1 1 1 ...

$ specificities : num [1:29] 0 0.02 0.04 0.08 0.22 0.48 0.74 0.88 0.96 1 ...

$ thresholds : num [1:29] -Inf 1.05 1.15 1.25 1.35 ...

$ direction : chr "

Thank you Saw, I have already researched about pROC and multiclass.roc and I know that they can not provide what I wish. 

As you said they compute a pairwise comparison of all possible combinations (of the level) and then multiclass.roc delivers a summarized AUC (it is something next to a overall mean of all the pairwise cases). 

If I would use the code you presented I would have to somehow define the overall sensitivity and specificity for a "multiclass ROC" by averaging the results from the pairwise comparison and I am not sure if this is the right way to do it, I could not find any reference on the paper used to implement multiclass.roc.

The same way I get a AUC (already summarized) for multiclass.roc I want to get the Cut-Off, Sensitivity and Specificity - summarized.