I am running tests on BERT transformers and using the evaluate Python library. On their site, it says: "computed with Accuracy = (TP + TN) / (TP + TN + FP + FN) Where: TP: True positive TN: True negative FP: False positive FN: False negative", which seems to indicate that it is the macro-accuracy.
The evaluate_metric/accuracy function in Hugging Face's library can be used to calculate either overall accuracy or macro-accuracy, depending on how it is used and configured. The definition of accuracy that you provided in your question refers to overall accuracy, which is calculated by taking the total number of true positive (TP) and true negative (TN) predictions made by the model and dividing it by the total number of predictions made (TP + TN + FP + FN).
On the other hand, macro-accuracy is calculated by averaging the accuracy of the model across all classes, rather than calculating the overall accuracy for the entire dataset. In other words, macro-accuracy calculates the accuracy of the model for each class and then takes the average of these accuracy scores, while overall accuracy calculates the percentage of correct predictions for the entire dataset.
Therefore, to calculate macro-accuracy using the evaluate_metric/accuracy function, you would need to provide it with the predicted and true labels for each class separately, and then average the accuracy scores for each class.
I hope this helps clarify the difference between overall accuracy and macro-accuracy, and how the evaluate_metric/accuracy function can be used to calculate either one. Let me know if you have any further questions.
After some investigations and inquiries, I found the answer to the question.
The simple answer is that this is the way one computes macro accuracy:
from sklearn.metrics import balanced_accuracy_score
balanced_accuracy_score(y_true, y_pred)
The longer answer is the one I got from Jacob Tyo, one of the authors of "On the State of the Art in Authorship Attribution and Authorship Verification". He explained how one can compute micro- and macro-accuracy. The following image contains the mathematical breakdown:
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Meta-Analysis