What is a ROC curve in a multiclass problem?

The best answer could be to treat the multiclass as a binary classification problem that is consider one vs all and calculate the operating points for each class and then average it out for the entire classifier. I am not sure how good of an idea this gives regarding the performance of the classifier, but this is the approach I have seen in many papers.

Also, it is possible to plot multiclass ROC curve using pROC library in R through the multiclass.roc function.

May I have a question about the multiclass classification AUC;

if I have three class this means I have also 3 AUC, so my question is which one  can represent my model(system) , or can I take the average of the three AUCs?

Best wishes

Yes Dheeb, you can take the average of the three AUCs. Alternatively, using the levels argument in the multiclass.roc function in pROC library, all levels are used and combined to compute the multiclass AUC. See David J. Hand and Robert J. Till (2001). A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning 45(2), p. 171–186.

http://link.springer.com/article/10.1023%2FA%3A1010920819831

Regards 

To complement the answers already here, you might want to have a look at this answer in Stack Overflow.

There are ways of visualizing the performance of multi-label classification tasks other than with ROC curves (e.g. as in attached publication).

http://stats.stackexchange.com/a/2155

Article Formulation and comparison of multi-class ROC surfaces

try it multiclass.roc() in R

Weka has this function. You can also try multiclass.roc () function in R

Here is the process-

Article A Simple Generalisation of the Area Under the ROC Curve for ...

If you use the Orange Machine Learning Software, you will be able to generate multiclass ROCs. However, they will not exist on the same graph.

Thanks people for the interesting and illuminating discussions above. In practice, the situation as follows often comes up:

• The goal is to identify the best prediction under a competitive multi-class setting, i.e., each class is competing to provide the best answer
  • We allow for the possibility of "no decision".

Suppose Softmax classification is used. How do we come up with a ROC?

An approach that almost no one uses (because they don't know it), but I consider far superior, would be to compute the consequential information, which is an information measure that captures all the performance issues in a single scalar (and bigger is always better). If you are interested, you can find the development for the binary case in a paper of mine published in IEEE Access, where it is referred to as "separation", and I would be happy to provide the yet to be published notes of how to extend it to the classification problem. The problem with both ROC curves and confusion matrices is they are ambiguous measures of performance; this is not.

In multi-class classification problem, you either formulate the problem as one-vs-all, where you will have a ROC curve for each class. or you formulate it as one-vs-one, where you will have a ROC curve for all class-pair combinations.

Confusion matrix is nice, but it is not statistically significant as it is a point estimate, except if you will plot the whole Precision-Recall curve and calculate its area.

I have a three class problem. My label for the three classes are 0, 1, and 2 for class 1, class 2, and class 3, respectively. I used one-vs-all approach to compute the sensitivities ans specificities. And once I know these two values, I can compute the area under the curve.

But the problem is, since I am using one-vs-all approach, I have three sensitivities, specificities, and AUC. How can I determine the overall values of these parameters for a classifier?

I've just written a blog post that discusses plotting ROC curves in the multi-class setting. It is available here:

https://www.datascienceblog.net/post/machine-learning/performance-measures-multi-class-problems/

What happens in a multi-class "ranking" classifier? I have a classifier that will identify an input of data as one of five available classes by just looking for the higher score (a score is generated for each class). How can I vary a classifier threshold in such a situation? Is it possible?

You can perform micro or macro averaging of TP and FN ratios across different classes. Sklearn has a great explanation on that.

https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html

Dear Eric Nunes

- For multi-class problem, you can do vROC (Volume ROC) instead of ROC. Indeed, ROC is just for binary classification task.

- If you want to only use ROC, you can evaluate your model by getting AUC for bi-ROC (between each two-classes) and then average all of them.

- Rather than ROC, there are many evaluation metrics for evaluation such as: Sensitivity, Specificity, Jaccard, F1 score, Overall accuracy, and MCC. You can find the definition of these metrics in my research papers here on ResearchGate.

Thanks!

Both PR and ROC is used for binary classification but if for multiple class, then consider one the positive class and rest of all lies in negative class. so, on this strategy we incorporate mulit class problem.

Hi Mugahed A. Al-antari ,

Would you be able to point me in the direction where I can learn about vROC please for multi class analysis.

BTW, have you had any experience with the Gerrity score for confusion matrices?

Thank you,

Don

this link may help

https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html

https://stackoverflow.com/questions/34169774/plot-roc-for-multiclass-roc-in-proc-package

Thanks very much Seyma

Dear Eric Nunes ,

Great discussion point. Following the updates!

Two questions!!

Can we plot multi-class ROC curves in SPSS?

Any insights into sample size requirements for multi-class ROC curves?

Thanks in advance 😊

How to plot multiclass.roc like this.

> library(ROCR)

> library(pROC)

> p roc.multi

Offcourse , you can construct a ROC for multi-class classification, as evident as Confusion matrix for it.