Hi Saeed,

I believe two concepts are being mixed here.

Assuming that you are estimating a single variable, during a classification task, the results are typically qualitative or categorical. In other words, if you are trying to classify certain data instances into two groups, your dependent variable (answer) should be like a label (e.g. group A = 0; group B = 1). Therefore, regardless of the specific number/result that your model is providing for a data point, it should be converted into one of your labels after applying an appropriate threshold (for the example mentioned, there could be a threshold of 0.5, and for any answers above 0.5, the result would become a 1, whereas for answers below 0.5, the result would become a 0). By doing this, you will always reach an accuracy rate (100% in your case) after determining the correct hits and dividing by total data points evaluated.

On the other hand, MAE directly compares the estimated value with the real value, so labels are not involved. In other words, for the previous case, for example, if a value of 0.9 is provided by a model and it is converted to 1, it might be correctly classified (100% accuracy), but when you calculate its MAE it is still greater than 0 (0.1 in this case).

Hope this helps you!

John

Agreed with John.

John C Cancilla Thank you for your valuable answer. That means that this situation is about the probabilistic classifiers, right? Because for some classifiers like SVM there is no threshold that you mentioned.

A question still remains for me. I appreciate any idea about it.

I doubled the training dataset size (with augmentation) and the accuracy was improved from 95.5 to 100% with an MAE error. I kept increasing the size of training set until the MAE error became 0. The question:

- Is it a good idea to decrease MAE error with augmentation or I should work on the threshold mentioned above?

Thank you so much.

The SVM provides a certain class estimate. There are several examples where you can give probability estimates given an SVM. For example, have a look to the SKlearn-Description of probabilistic estimates given the SVM.

http://scikit-learn.org/stable/modules/svm.html

Also have a look at the contribution of Wu et al.

• Wu, Lin and Weng, “Probability estimates for multi-class classification by pairwise coupling”, JMLR 5:975-1005, 2004.

This paper is talked of in the SVM user guide of the scikit-learn module.

Hope this helps,

Patrick

Further more you can rely on the decision function of the SVM. Consider the documentation of sklearn again.

Best regards,

Patrick

Hi Saeed,

In my opinion, if you are able to fulfill your goals without using augmentation (artificially preparing new samples), I would go for it. If your database is big enough, I would try to avoid creating new samples from your old ones that may be increasing your accuracy due to overfitting (may not be the case, but when you use augmentation, you are basically inputting very similar data points multiple times possibly "encouraging" an overfit system).

Nevertheless, be thorough with your validation methods to see if your algorithm can generalize well. Make sure you keep samples outside your training dataset to validate, including all augmented versions of those samples. Otherwise, if you test or validate your model with augmented data that is contained in your training set, you may see an artificial increase in accuracy.

Best of luck,

John

John C Cancilla Thanks a lot for your answer.

Of course I am not using the training samples for the test process. I am using cross-validation option because my training set is very small. The augmentation method is also reliable and I am applying it to avoid the over-fitting problem. However, I have tested the method with splitting the data to train and test set (60%-40%) and gotten almost the same result.