It depends on the type of data you are going to classify. Depending whether you have distinct or continues features.

For further reading, I recommend to have a look on the following:

1- https://towardsdatascience.com/7-data-types-a-better-way-to-think-about-data-types-for-machine-learning-939fae99a689

2- https://towardsdatascience.com/probabilistic-machine-learning-series-post-1-c8809652dd60

Simplest approach - to use Logistic Regression (its output, in differ of SVM, NN, etc. is directly probabilities of classes) and some expressions for conditional (Bayes) probabilities. But if P(A) = 0, I prefer to build two algorithms - one for 3 classes (with A) and second for two classes (without A).

The Bayer classifier might be one of the most suitable because it uses the prior probability of each class as a parameter, using which information already known about the classes can be incorporated

The best option in this case would be a Bayes classifier.

In Bayes' classifier, the class assignment for an observation is done by the combination of the Bayes' rule and the maximum a posteriori decision rule as follows:

y = argmax k = P(Ck) x (Likelihood of the observation given class Ck)

where P(Ck) is the class' prior probability and is typically calculated in either of the following ways:

• assuming equiprobable classes: prior = 1 / number of classes
• estimating from the training data: P(Ck) = number of observations in class k / total number of observations

However, one can choose a different prior, and here your prior information about a certain class will come into play.

Hope it helps!

Mohamed Elhadad

Amir Hossein Poorjam

Ranjan Parekh

Sergey Porotsky

Dear all, Thanks for your great answer, let me tell you more about the question.

1- I want to apply this approach for SVM, RF, and other machine learning methods, But I can not incorporate the information in the model.

2- I wanna apply this on test data

Bests and thanks for your consideration

For classification tasks using Machine Learning (ML) algorithms, categorization or classification is usually based on the patterns of relationship learnt from the various categories or classes present in the training set. Thus, if a new entry or record is shown to the ML classifier after training; the classifier tries to classify this new record into one of the classes or categories it has learnt from the training set based on a certain degree/percentage/probability of confidence. All classifications are based on the number of classes learnt during training ONLY. So if a new class is introduced after training (or during validation/testing), it is still classified as one of the classes contained in the training set.

My opinion on handling unforeseeable classes is to transform your problem into a regression task, and apply ML regression models to your problem. This way, the model will output the results as a function of confidence level (i.e. probability degree/percentage). Thereafter, you can use the (output) confidence level to make appropriate classification or categorization.

Bonaventure Molokwu

Many thanks

I would like to suggest you to follow this Thesis on Bayesian classification for matrix valued data:https://stats.stackexchange.com/questions/35185/dimensionality-reduction-svd-or-pca-on-a-large-sparse-matrix?rq=1 . For PCA you may follow: https://stats.stackexchange.com/questions/35185/dimensionality-reduction-svd-or-pca-on-a-large-sparse-matrix?rq=1

Hasan Misaii

Classifiers such as SVM and RF are discriminative. Unlike generative classifiers, incorporating the class priors into discriminative classifiers is not straight forward. The attached paper (or similar ones) explains how to incorporate the class priors into such classifiers.

One of the easiest way to implement this in non-generative methods such as RFs, NNs, SVMs..is by re-weighting of the training data distribution. So you want to impose constraint P(A=0), you go and place weight of 0 on all the observations from class A..hence the observation is no longer used for training. If you want to have P(A=0.1), you randomly select as many observations from class A such that they consist of 10% of the training data. Hope that helps, happy to clarify.