the over all accuracy is the first 1 one you calculate

= (TP+TN)/(TP+TN+FP+FN)=95.60%

TP and TN here are the same = 11472 because both are the sum of all true classified examples, regardless their classes 

false positives, which are items incorrectly labeled as belonging to the class=528

false negatives, which are items which were not labeled as belonging to the positive class but should have been=528

However, you may do the accuracy for each class:

class N

TP 1971

TN 9501

FP 37

FN 29

Acc 99.43%

precision=TP / (TP + FP) 98.16%

sensitivity = TP / (TP + FN) 98.55%

specificity = TN / (FP + TN) 99.61%

F-score = 2*TP /(2*TP + FP + FN) 98.35%

class L

TP 1940

TN 9532

FP 83

FN 60

Acc 98.77%

precision 95.90%

sensitivity = TP / (TP + FN) 97.00%

specificity = TN / (FP + TN) 99.14%

F-score = 2*TP /(2*TP + FP + FN) 96.45%

class R

TP 1891

TN 9581

FP 195

FN 109

Acc 97.42%

precision 90.65%

sensitivity = TP / (TP + FN) 94.55%

specificity = TN / (FP + TN) 98.01%

F-score = 2*TP /(2*TP + FP + FN) 92.56%

class A

TP 1786

TN 9686

FP 137

FN 214

Acc 97.03%

precision 92.88%

sensitivity = TP / (TP + FN) 89.30%

specificity = TN / (FP + TN) 98.61%

F-score = 2*TP /(2*TP + FP + FN) 91.05%

class P

TP 1958

TN 9514

FP 36

FN 42

Acc 99.32%

precision 98.19%

sensitivity = TP / (TP + FN) 97.90%

specificity = TN / (FP + TN) 99.62%

F-score = 2*TP /(2*TP + FP + FN) 98.05%

class V

TP 1926

TN 9546

FP 40

FN 74

Acc 99.02%

precision 97.97%

sensitivity = TP / (TP + FN) 96.30%

specificity = TN / (FP + TN) 99.58%

F-score = 2*TP /(2*TP + FP + FN) 97.13%

avrg acc of all classes 98.50%

in this case you get your second measurement which you stated in your question.

I think the first accuracy is the most appropriate, however, you may write down the second accuracy as not the accuracy of the system but rather the average of the accuracies for each class.

an excel file is attached.

Thanks, dear Saman.

I couldn't understand ur method for sensitivity , ... calculation. If possible, plz help me to  calculate one of them,at least. 

Is there a MATLAB function to calculate them automatically? I used 'classperf', but get  different results with ur method!

the over all accuracy is the first 1 one you calculate

= (TP+TN)/(TP+TN+FP+FN)=95.60%

TP and TN here are the same = 11472 because both are the sum of all true classified examples, regardless their classes 

false positives, which are items incorrectly labeled as belonging to the class=528

false negatives, which are items which were not labeled as belonging to the positive class but should have been=528

However, you may do the accuracy for each class:

class N

TP 1971

TN 9501

FP 37

FN 29

Acc 99.43%

precision=TP / (TP + FP) 98.16%

sensitivity = TP / (TP + FN) 98.55%

specificity = TN / (FP + TN) 99.61%

F-score = 2*TP /(2*TP + FP + FN) 98.35%

class L

TP 1940

TN 9532

FP 83

FN 60

Acc 98.77%

precision 95.90%

sensitivity = TP / (TP + FN) 97.00%

specificity = TN / (FP + TN) 99.14%

F-score = 2*TP /(2*TP + FP + FN) 96.45%

class R

TP 1891

TN 9581

FP 195

FN 109

Acc 97.42%

precision 90.65%

sensitivity = TP / (TP + FN) 94.55%

specificity = TN / (FP + TN) 98.01%

F-score = 2*TP /(2*TP + FP + FN) 92.56%

class A

TP 1786

TN 9686

FP 137

FN 214

Acc 97.03%

precision 92.88%

sensitivity = TP / (TP + FN) 89.30%

specificity = TN / (FP + TN) 98.61%

F-score = 2*TP /(2*TP + FP + FN) 91.05%

class P

TP 1958

TN 9514

FP 36

FN 42

Acc 99.32%

precision 98.19%

sensitivity = TP / (TP + FN) 97.90%

specificity = TN / (FP + TN) 99.62%

F-score = 2*TP /(2*TP + FP + FN) 98.05%

class V

TP 1926

TN 9546

FP 40

FN 74

Acc 99.02%

precision 97.97%

sensitivity = TP / (TP + FN) 96.30%

specificity = TN / (FP + TN) 99.58%

F-score = 2*TP /(2*TP + FP + FN) 97.13%

avrg acc of all classes 98.50%

in this case you get your second measurement which you stated in your question.

I think the first accuracy is the most appropriate, however, you may write down the second accuracy as not the accuracy of the system but rather the average of the accuracies for each class.

an excel file is attached.

Hi Akbar,

Python has a very useful library for assessment.

"from sklearn.metrics"

Confusion matrix is included.

for further information, please check following link:

http://scikit-learn.org/stable/

Thank you all for your help.

Dear Ahmad Hassanat, thank you for your complete guidance. But, in the first method that you have mentioned, all performance parameters (accuracy,sensitivity,...) will be equal with each other. In all papers that i have read, this parameters are different with each other. Also, at second method, we will have some samples that do not belong to non of TP, TN, FP, FN (TN+TP+FP+FN

Dear Akbar

"first method that you have mentioned, all performance parameters accuracy,sensitivity,...) will be equal with each other" that because TP, TN, FP, and FN normally come with binary classifiers, Yes/No, Go/nogo, etc. hence you have more than 2 classes, correctly classifying class 1 is TP and correctly classifying other classes is TN, same for the others.

the negative and positive are not defined well here, i.e. the negative of class 1 is not the positive of class 2 and so on, therefore, I calculated the summation of the FP for all classes, and the summation of the FN for all classes, that why they are equal,

since FP = FN  ==> all the measures are equal (for the over all accuracy)

I am  not aware of those Matlab functions, however, my advice is to use the first accuracy "95.60%" as the accuracy of your system and if you want to use the precision and recall measures, you have to use them w.r.t each class as clarified in the above linked excel sheet.

Dear Dr.Hassanat

I would like to thank you for your explanation

I have question about the deference between the classification correct rate and accuracy. in Weka the used the classification correct rate as the accuracy but the authors use the TP + TN/ All samples , as accuracy . my question is can you use classification rate as accuracy.

Hi

I think there are some of wrong with results for Dr. Ahmad Hassanat "Sorry Dr". The value of all (TN) is not right, may be he took some of numbers from confusion matrix as mistake. The results of this confusion matrix should be as following:

          TP         TN          FP           FN          Accuracy

N     1971     9963        37           29             99.45

L      1940     9917        83           60             98.80833333

R     1891     9805      195         109            97.46666667

A     1786      9863      137         214            97.075

P     1958      9964        36           42             99.35

V     1926      9960        40           74             99.05

If need more on the computation I suggest you consult neurosolution documentation

Dear Akbar,

Sensitivity and specificity are only defined for binary classification problems, i.e.: when the confusion matrix is actually a contingency table. Both in two- and multi-class problems accuracy is the sum of the main diagonal normalised by the total number of instances to classify. Other options for multi-class problems are multi-class ROC graphs - see for instance T. Fawcett, 'An introduction to ROC analysis', Pattern Recognition Letters, 27:861-874, 2006

Best,

f.

this should help you - > http://www.marcovanetti.com/pages/cfmatrix/

automatically gives you accuracy after you enter your confusion matrix details.

 hi,

I have a confusion matrix like below

> head(table)

thrsh tp fp fn tn

1 1 0 1 7080 8155368

2 1 0 1 7080 8155368

3 1 0 2 7080 8155367

4 1 0 3 7080 8155366

5 1 0 4 7080 8155365

6 1 0 5 7080 8155364

how i can calculate true positive rates (TPR) at relatively low value of false positive

rate (FPR = 0.03)????

for one row i can calculate based on formula but i got confused for my table how i can calculate

thank you for your help

Dear Dr.Hassanat,

According to your definition Acc= (TP+TN)/(TP+TN+FP+FN). Whenever TP=0, even though accuracy is high. How can we judge the performance in this situation. At the same time precsion= TP/(TP+FP) is zero. Anybody please explain this situation.

Dear Rajesh Kandala

any system with TP =0 is a very weak system, and that why we always recommend to calculate precision and recall, and not only accuracy, because a weak system with 0 precision might give you 90% accuracy, and this is misleading for the users of the system.

for example  if we got:

               client          impostor

client         0                 10

impostor   0                 90

TP 0

TN 90

FP 0

FN 10

precision=TP / (TP + FP) =0,00%

specificity = TN / (FP + TN) =100,00%

accuracy = (TP+TN)/(TP+TN+FP+FN) =90,00%

F-score = 2*TP /(2*TP + FP + FN) =0%

this is normally happens when we have an unbalanced training data, with a specific class (impostor) has much larger number of examples than the other class (client) , according to such training data I can right the following "intelligent" system (2 lines of code):

-----------------------------

Input feature vector

Output "impostor"

-----------------------------

and I get 90% of accuracy using the same data. but is this an intelligent system?

Thank you, Ahmad Hassanat 

In addition to what Dr Hassanat explained, the accuracy itself is trustable as long as there is sort of imbalanced data problem . Otherwise, accuracy is biased to one of the classes and the precision and recall need to be considered to evaluate the classification performance. 

Dear Kalyan

1) Let A be the expert segmented image and assume it is the ground truth segmentation.

2) Let B the resultant image 

3) Let C=A-B

4) False rejection (FN)=count white pixels in C 

5) Let D=B-A

6) False acceptance rate (FP) =count white pixels in D 

7) true positive (TP)= count white in (A & B) 

8) true negative (TN)= count white in (not A & not B)

Now carry on the equations:

precision=TP / (TP + FP) 

sensitivity = TP / (TP + FN) 

specificity = TN / (FP + TN) 

F-score = 2*TP /(2*TP + FP + FN) 

etc.

what a thorough answer, Ahmad. Some kind of acknowledgement is very much in order.

Thanks Hongbo

I learnt a lot from you my friend.

kindly help me in calculating accuracy in weka tool for kdd cup data set...thank you all

Dear ProfessorS,

I am also facing a problem while calculating "Recall". I need your kind guidance in this regards.

Problem is:

If I have two ground truth classes let say

w1={0, 1, 2, 3, 4, 5, 6, 7, 8, 9} 

and 

w2={a, b, c, d, e, f, g, h}

and my clustering algorithm gives me 4 clusters, let suppose

CL1={0, 1, 2, 3, a}

CL2={4, 5, 6, 7, 8}

CL3={b, c, d, e, 9}

CL4={f, g, h}

how can I calculated the recall? What will be the values of True Positive, True Negative, False Positive and False Negative?

Hi everyone,

I have problem with confused matrix in papers.

How is created/computed confused matricies (percentage by class) and overall accuracies in papers about the action recognition on the MSR Action 3D dataset? Especially in papers:

• Vemulapalli et al. - [correction] Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group
• Du et al. - Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition
• Evangelidis - Skeletal Quads: Human Action Recognition Using Joint Quadruples

I try in my case compute Accuracy (ACC) and F1 as stated in https://en.wikipedia.org/wiki/Confusion_matrix. I also try compute principle TP/ALL.

I think, the autors of the papers don't uses TP/ALL for compute overlay accuracy because no multiple of numbers 11 or 15 (in Action Set 3 - cross subject) is not an integer or number close to the integer.

@ Wathiq Laftah Al-Yaseen , calculations on the top by Ahmad Hassanat are fine. The table he is using is not visible thats why calculations doesnt look accurate. Click on the image above to look at table:

https://www.researchgate.net/profile/Akbar_Esmaeelzadeh/post/Can_someone_help_me_to_calculate_accuracy_sensitivity_of_a_66_confusion_matrix/attachment/59d6206b6cda7b8083a19b81/AS%3A273652065472520%401442255144350/image/Untitled-1.jpg

Dear @Ahmad Hassanat,

Thank you very much for the explanation. However, I would need to cite the thing you have explained. Could you refer to any publications regarding the method to compute accuracy per-class?

Thanks in advance,

Hello @Azkario Rizky Pratama,

Re citations: I recommend reading about Performance Measures: Multinomial Targets in the book titled, Fundamentals of Machine Learning for Predictive Data Analytics Algorithms, Worked Examples, and Case Studies, by Kelleher, J., Mac Namee, B., & D'Arcy, A.

The authors thoroughly explain the process.

Hello Sir,

I am working on weka but few problem arises like few classifiers are disabled .So that I am not getting results TP Rate,FP Rate , AUC ,ROC CURVE and others . KIndly tell me why it is happend . Suggest the process so that I can sort out my problem. Thanks

Thanks a million for such a thorough explanation. Very useful!

Ahmad Hassanat The mean accuracy which is defined as " average accuracy is the average of each accuracy per class (sum of accuracy for each class predicted/number of class) " in https://social.msdn.microsoft.com/Forums/en-US/eb31a252-8d84-4873-ad78-ceffc20b03b6/what-is-the-difference-between-overall-accuracy-and-average-accuracy?forum=MachineLearning

According to this definition Sensitivity will be the mean accuracy?

Dear @ Adnan Awan

Precision/recall are defined for binary classification, not multi-classes, but I do not know why some reviewers normally ask for these measures when dealing with multi-classes problems, therefore, we have to do these calculations for each class separately, then we average these measures, to get the average of precision and the average of recall, exactly like what Weka does, if you see the classification results at the end, you will see the tp, tn, p, r and more for each class then they average these at the end.

I hope I answered your question.

This paper explains many classification measures in details with examples and it also explain a similar example to yours.

Article Classification Assessment Methods: a detailed tutorial

Presentation Classification assessment methods: a detailed tutorial

Hi! I am facing a similar problem. I am comparing different algorithms for object recognition and I am not sure about TN/FN/......

I am gonna attach a small example for one dataset with the results for all four algorithms. Any help / advice would be really appreciated especially concerning the right metric. Thanks a lot!

Here is a simple Mat file for calculation.

@ Ozal Yildirim I need it.

http://www.cs.cornell.edu/courses/cs678/2006sp/performance_measures.4up.pdf

https://machinelearningmastery.com/confusion-matrix-machine-learning/

Hi Akbar,

Unlike to what Francesco Bianconi explained, you can actually have sensitivity and specificity regardless of the number of classes. The only difference here is that as Ahmad Hassanat showed, you will get one specificity and sensitivity and accuracy and F1-score for each of the classes. If you want to report, you can report the average of these values.

Hi

I thought this movie "Machine learning fundamentals: Sensitivity and specificity" was very helpful in explaining the calculation of sensitivity and specificity from a confusion matrix with more than two classes.

https://www.youtube.com/watch?v=sunUKFXMHGk&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=4

If you are dealing with a multiclass problem then

Article Multivariate comparison of classification performance measures

might be worth to look at. In the supplementary material is a matlab file which calculates all performance measures described in the paper given the confusion matrix.

If you want do to it by hand the paper describes all the formulars needed

Although a few suggested averaging your evaluation measures, there was no mention of mIcro- and mAcro-averaging. When dealing with a multi-class (or multi-label) problem, you could average in two different ways. Alternatively, you could compute your evaluation measures (precision, recall, accuracy, f-measure, AUC, etc.) on a per-instance basis, which is equivalent to mAcro-averaging.

Here's a paper describing (with formulae) all three methods: https://ieeexplore.ieee.org/abstract/document/6471714

Additionally, I should note that in your example case, the classes are fairly balanced (the number of instances in each class is the same). However, when dealing with class-imbalance cases, you will see that mIcro- and mAcro-average will produce different results. It helps to report both. Also, examining your evaluation measures on a per-class basis could reveal useful insights.

Here's a recent paper that evaluates different classification performance measures in multi-class and class-imbalanced situations: Article Multivariate comparison of classification performance measures

Read the book

Congalton & Green, Assessing the Accuracy of Remotely Sensed Data. Principles and Practices. CRC Press

@Ozal , I've edited the code of your confmat file cause it looks to me as it's mistaking the calculation of true negatives. Here the updated version. Feedbacks are welcome.

Refer these links, that evaluates different classification performance measures

https://machinelearningmastery.com/confusion-matrix-machine-learning/

https://www.researchgate.net/deref/https%3A%2F%2Fieeexplore.ieee.org%2Fabstract%2Fdocument%2F6471714

https://www.researchgate.net/publication/321711344_Multivariate_comparison_of_classification_performance_measures

The following video would help you:

https://www.youtube.com/watch?v=FAr2GmWNbT0

Accuracy is the ratio of correct classification count and total (correct and incorrect), Refer to following link for further information..

https://classeval.wordpress.com/introduction/basic-evaluation-measures/