The code is:
import evaluate
metric = evaluate.load("accuracy")
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
return metric.compute(predictions=predictions, references=labels)
from transformers import TrainingArguments, Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=test_dataset,
compute_metrics=compute_metrics,
)
Well, here is an efficient pseudo-code of how to compute macro-accuracy for BERT multi-classification text:
// Calculate macro-accuracy
1. Initialize macro_accuracy to 0
2. For each class in the multi-class classification text:
a. Calculate confusion matrix and accuracy score
b. Add the accuracy score to macro_accuracy
3. Divide macro_accuracy by the number of classes
4. Print macro_accuracy.
------------------------------------------------------
You can use the following code example for computing macro-accuracy for BERT multi-classification text:
From sklearn.metrics
import confusion_matrix, accuracy_score
import numpy as np
predicted = model.predict(X_test)
cm = confusion_matrix(y_test, predicted)
accuracy = accuracy_score(y_test, predicted)
# Calculate macro-accuracy
macro_accuracy = np.mean(np.diag(cm))
print("Macro-accuracy:", macro_accuracy).
Thank you very much for the reply. I was able to implement the required code...
----------------------------------
predicted=trainer.predict(test_dataset)
c=0
y=[]
for v in predicted.predictions:
c+=1
max = v[0]
index = 0
for i in range(1,len(v)):
if v[i] > max:
max = v[. I]
index = I
y.append(index)
from sklearn.metrics import confusion_matrix, accuracy_score
cm=confusion_matrix(test_dataset['label'], y)
print(cm)
accuracy = accuracy_score(test_dataset['label'], y)
print("accuracy:", accuracy)
# Calculate macro-accuracy
macro_accuracy = np.mean(np.diag(cm))
print("Macro-accuracy:", macro_accuracy)
----------------------------------
However, I got the following results:
[[100 1 2 5 1 11 0 0 0 1]
[ 0 235 5 24 2 0 1 0 54 6]
[ 0 3 290 7 0 0 5 3 10 7]
[ 0 0 0 138 0 9 0 0 0 2]
[ 1 0 0 0 39 0 0 1 14 2]
[ 0 2 0 3 0 192 0 0 0 8]
[ 0 0 0 0 3 0 22 4 3 0]
[ 0 0 0 0 1 0 5 95 30 3]
[ 0 5 0 0 4 0 0 0 177 2]
[ 0 0 0 8 4 7 0 0 2 478]]
accuracy: 0.8669612174766814
Macro-accuracy: 176.6
It seems to me the result is not ok. The value should be somewhere up to 1.
After some investigations and inquiries, I found the answer to the question.
The simple answer is that this is the way one computes macro accuracy:
from sklearn.metrics import balanced_accuracy_score
balanced_accuracy_score(y_true, y_pred)
The longer answer is the one I got from Jacob Tyo, one of the authors of "On the State of the Art in Authorship Attribution and Authorship Verification". He explained how one can compute micro- and macro-accuracy. The following image contains the mathematical breakdown:
- Badges
- Science topic