There are a few ways you can deal with imbalanced datasets. Undersampling involves removal of some of data your majority class to result in a balanced distribution of all classes. However if you have a small dataset size this may not be ideal! There is also a method of oversampling where you increase the numbers of your minority class (by duplication of the data of the minority class - your under-represented class) However this method is not ideal as you are duplicating data so may effect your models.

Additionally if you have a small dataset and do not want to remove data then maybe the application of misclassification costs to your SVM models. Basically you apply higher misclassfication costs to your minority class and should essentially balance out your classes. In other words your model is better at classifying data that it has more of so if you apply a higher misclassfication cost to the minority class it will avoid misclassifying your minority class due to higher cost you have applied to it. As models want to perform with the least cost so if you apply higher costs to classes you want to improve the prediction of it will avoid this misclassification.

I am pretty sure you can apply this to SVM models however the main bulk of my research is using these methods using decision trees but even so it may help you with further ideas! You can compare the methods with and without misclassfication costs for example and see if SVM can cope with the imbalanced data.

Please check out my publications on my page for more information as these go into more detail regarding the imbalanced data problem.

Danielle

An empirical test would be, to use cross-validation. Split your set in j-folds, use j-1 folds for training and the remaining fold for testing. This is repeated j times with different js as test sample. The performance can be averaged over all j folds. The variacne of this aveaging can tell you something about the imfluence of imbalancing.

(cf. A survey of cross-validation procedures for model selection, Statist. Surv. Volume 4 (2010), 1-274).

Absolutely i use 10-fold cross validation as model selection method. If the model was useless in imbalancing problem, the question is how i make the model able to deal with such problem.

You may use oversampling or cost-sensitive learning

There are a few ways you can deal with imbalanced datasets. Undersampling involves removal of some of data your majority class to result in a balanced distribution of all classes. However if you have a small dataset size this may not be ideal! There is also a method of oversampling where you increase the numbers of your minority class (by duplication of the data of the minority class - your under-represented class) However this method is not ideal as you are duplicating data so may effect your models.

Additionally if you have a small dataset and do not want to remove data then maybe the application of misclassification costs to your SVM models. Basically you apply higher misclassfication costs to your minority class and should essentially balance out your classes. In other words your model is better at classifying data that it has more of so if you apply a higher misclassfication cost to the minority class it will avoid misclassifying your minority class due to higher cost you have applied to it. As models want to perform with the least cost so if you apply higher costs to classes you want to improve the prediction of it will avoid this misclassification.

I am pretty sure you can apply this to SVM models however the main bulk of my research is using these methods using decision trees but even so it may help you with further ideas! You can compare the methods with and without misclassfication costs for example and see if SVM can cope with the imbalanced data.

Please check out my publications on my page for more information as these go into more detail regarding the imbalanced data problem.

Danielle

maybe Semi_supervised methods be able to solve your problems. These methods try to synthesis some unreal samples based on the existing samples. in this way the amount of existing samples increases. for example we can add some new genes to a pool of genes just by using sequence alignment techniques performed on a gene dataset.

I am agree with @Danielle Newby in one point, when taking more cost for the minority class to avoid its classification, because every change in dataset natural distribution may lead to fatal errors in some application related to medical and biology. i am not agree with deleting or increasing one or more class samples. and we must be sure in choosing dataset sample, i know there is formal standards to collect datasets. every change must be in model and learning algorithm.

@Danielle Newby what if we have multi-class in the data set. how can we assign costs to classes within the model. suppose we have more than one minority class which.

@Saeed Reza Kheradpisheh are you sure that semi-supervised methods "try to synthesis some unreal samples based on the existing samples"?. this task seems related to preprocessing data or so on. may you provide me some references ?

Hi Osama,

With multi class problems you can still apply missclassification costs for multiple minority classes, however this is with my work using multi class models using decision trees. If you can apply missclassification costs for your method i do not see why not for multi class. It also might depend on the software you use if it lets you

You can go for discretization using a threshold value.

Hi Osama,

recently, i have done a facial expression classification project. The goal of the project is to determine the percentage of each expression in query. I have observed that unbalancing one class affect the accuracy of whole algorithm. I used a heuristic method. In my proposed method, i cascade two classifier. in the first stage, i used LDA to define whether its belong to the big unbalanced data or not. if yes, the answer is reached. if not, i passed the query to the multi-class SVM which is used to distinguish between other 6 classes.

I think you can utilize same strategy for your unbalanced data. at first, divide your class into groups to expiate unbalancing. Note that if you use same classifier in different stages, you can't obtain better result in contrast with situation when you use one stage multi-class classifier.

reagrds

I have done some work on the imbalance of data using KDD Cup 1999 Data set. the paper is titled as "Effect of Feature Selection, SMOTE and under Sampling on Class Imbalance Classification" . The outcome of this work is that for the class imbalance data set the under-sampling technique is more effective than SMOTE in detecting minor classes. this conclusion was drawn by evaluating several classifiers. May be this helps to answer ur question.

Hi Ehsan,

i get your idea but this step don't suffer from overload? i understand as you said to make high level classifier to determine that the current test instance is related to big unbalanced data or not. what if we have high error rate in this step i think the error will propagated in the testing step. if you have any reference or published work send me please.

Cheers

Hi Danielle,

I am using Matlab toolboxes and LIBSVM library. I got it in SVM training phase we apply multiple costs for each separated SVM classifier depends to its classes. then we can fuse results together. here i am not sure about generalization and over-fitting model. I must to test this method in order to take more confident answer according to results.

Osama

What do you mean by overload? if you mean computational complexity, i want to tell you it's not suffer from overload even its faster and i got better result and lower error rate. just recall viola jones cascade classifier. I use that idea to cascade some weak classifier to construct a powerful but simple structured classifier.

I agree with your error propagation idea, but you can just try it and report us your result. In my project, I got extremely good result but as we know the results depend on the application, data variation, outliers and etc.

I did all that work for a course project and me with my partner decided to publish it. If you need more information do not hesitate to ask me more questions.

As already suggested, there are different ways to dial with the problem, inluding undersampling of the majority classes, overasampling of the minority classes, different missclassifcation costs per class, etc. Undersampling is a safe apporach, but it comes at the drawback of throughing some of the data as well as being unsuitable for small datasets. Assigning different costs in each class is very intuitive on certain problems and quite straightforward. In my experience it might be all you need if you have a moderately unbalanced dataset (e.g., some of the classes are a few times more frequent than others). If we are talking about classes that have orders of magnitude less samples than the dominant ones, the effect of different miscalssifcation costs might no be sufficient though. For such cases I would recomend to use SMOTE, you can find some details here:

http://www3.nd.edu/~dial/papers/JAIR02.pdf

Depending on your platform (Matlab, python, etc) there might also be some open source implementations of the technique

Osama:

First you need to know whether the dataset you are dealing with is naturally imbalanced or it is resulted from an "improper sampling." Moreover, prior to any analysis, you would need to know your type of cost assignment, either the classical overall accuracy or some adjusted costs. In many cases, specifically in Bioinformatics, the datsets are imbalanced, not due to the nature of subgroups. It happens often because of our lack of knowledge regarding the "true underlying population statistics." If we have the population statistics, we could have re-sampled "randomly" the dataset, so that it becomes consistent with the truth.

If you believe that your dataset is supposed to be like that, then you would better go with oversampling or reverse, as suggested by the others. Nonetheless, again, please be aware of improper stratified sampling. We have recently published a paper in Bioinformatics entitled "Effect of Separate Sampling on Classification Accuracy:" http://bioinformatics.oxfordjournals.org/content/30/2/242.short. In that publication, we introduced the notion of "minimax sampling ratio." It is tested for most of the popular classification rules, e.g. Linear SVM, RBF SVM, LDA, QDA, or Decision Tree, etc.

Finally, in case you do not have access to the full text, I would be glad to send you the pdf file in a message.

I think you can use decision tree with cost function.

Hi osama

i have the same problem on classifying mode of transportation using mobile data.

Some classes have about 7000 sample in the training set while the other have about 500 values. I have tried several approaches including oversampling and down-sampling but they provide less accuracy than expected. The approach that gives fairs results is a mix of downsampling and oversampling. In my case i adjust the size of all classes to 2000 sample.

Many times undersampling works better than oversampling.

Conference Paper Cluster Based Under-Sampling for Unbalanced Cardiovascular Data

Article Addressing the Class Imbalance Problem in Medical Datasets