Imbalanced dataset sometimes provide you with good accuracy score but mostly it is misleading. Firstly try to keep the ratio of both files near about like 60:40 then check out your classifiers for the same. In my opinion your classifier if it is giving good results with 60 being malicious and 40 being good files, should also work well otherways round. But first try to work out with near about ratio.

Please look at the class

Machine learning

by professor

Yasser

from

Caltech university

on

YouTube

For imbalanced Dataset one cannot be just sure by seeing the accuracy. It is because the model might be just predicting the majority class.

Some other metrics to consider are :

1. Precision

2. F1 score

3 Ranking metrics like ROC and AUC

4. Precision recall ROC curves

( https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/ )

This blog by Jason Brownlee clearly states you need of a metric based on your specific type of output in a flow diagram like manner:

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

You can also use techniques like :

1. Undersampling

2. Oversampling

3. SMOTE

4. Axis rotation

These are some techniques to bring balance to the imbalanced dataset. I have also used it in my recent research paper:

Article OSX malware detection : Challenges and solutions

I found SMOTE ( with its various variations ) and Axis rotation to be most helpful with smaller or medium-sized datasets.

Also, see this link for more details :

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Image credit : https://machinelearningmastery.com/ @ Jason Brownlee

Dear Petra Vukmirovic ,

Look the link, maybe useful.

https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Regards,

Shafagat

One way to approach this problem is to force a 50-50 balance on the training and testing data, as Ashay Singh mentioned. Then you can use accuracy or any other measure to evaluate the performance. However, when you deploy your model, you should always correct the model's final prediction using the actual class prior (actual proportion of malicious files), because you can't assume the equal probability of getting a sample of either class when there are actually going to be more of one than the other.

Another way to do it is to use cost-sensitive classification approaches and increasing the cost of misclassifying malicious files vs the others.

For evaluating performance, I recommend using balanced accuracy, which was designed to handle class imbalance situations. Balanced accuracy is basically a mean of recall values across all classes. In binary classification, it is (true_positive_rate + true_negative_rate)/2. Here's a paper that details balanced accuracy: https://ong-home.my/papers/brodersen10post-balacc.pdf.

Since in real applications you would expect much fewer malicious files, then maybe some form of outlier detection is a better approach than classification.

Dear Petra Vukmirovic,

One of the most widespread problems affecting Machine Learning models is data imbalance, when, as you said, the sample size of one class or some classes is far greater than other classes. The negative impact data imbalance has on the traditional learning models is worst in combination with other dataset issues (small disjuncts, presence of outliers and insufficiently number of training observations).

The two most common approaches to deal with imbalanced datasets used to improve the performance of Machine Learning classifier models are methods based on data and based on algorithm.

- Methods based on data typically adopt the under-sampling or over-sampling technique to resample the initial dataset. This type of processing intends to change the distribution of the imbalanced dataset so as to improve the classification performance of the model (be careful on resampling only the training dataset!)

- Methods based on algorithms try to improve the performance of the ML models. They include many approaches such as the use of ensemble learning models, single positive class learning, one-class learning, etc.

Cheers!

I think you have to depend on more than one evaluation metric for evaluating the classification results. Moreover, performing cross validation on the data to detect the behavior of different classifiers on the data to decide which classifier to use..

e.g. Accuracy (ACC), Error Rate (ERR), Precision (P), Recall (R) or True Positive Rate (TPR) or Sensitivity (SN), F1-Score (F1), Area Under the Curve (AUC), Specificity (TNR), Geometric-mean (GM), Miss Rate (FNR), Fall-Out Rate (FPR), False Discovery Rate (FDR), False Omission Rate (FOR), etc.

As others have mentioned above, class imbalance can often be dealt with by:

Using a combination of these two techniques could be useful if you simulated new samples and then evaluated the effect using an F-score metric for model tuning.

Best of luck!

A quick note about the F-score, which measures the tradeoff between precision and recall. It only considers the positive class in the calculation. So you have to be careful about how you interpret it.

If your dataset has very few positive instances, and you want your classifier to be sensitive to positive instances, then F-score is one of the better metrics. However, if your minority class is negative, I don't think F-score will work that well. Please correct me if I am wrong.

Tom Arjannikov you are correct. This is commonly handled by always making the minority class the "Positive" class, but that is an excellent reminder! It is also effective when the class sizes are reasonably balanced, making it a great metric for either case.

Fix the other hyper parameters and compare the results by using

1) 800 malicious and 800 good files VS. 2) 1200 malicious and 400 good files

that keeps the total number of training examples on the two sets, so the 'only' difference are the class imbalance.

I think you can change the beta value in the f score and adjust it to give more weights either to precision or recall to overcome this problem . Tom Arjannikov

Khitam Muhammed Mesri, that's right, but f-score does not account for true negatives, which means it treats negative class differently from positive class. However, it is somewhat arbitrary which class you call "positive". So, in binary classification, you can compute two f-scores, one for positive class and one for negative class. You will notice that when the classifier has different rates of success with one class from another, there will be a difference between the two f-scores.

I finally got it . Thanks for making it clear for me Tom Arjannikov