Hello, everyone!
In recent time I am learning an online course called "learning from data". The lecturer showed the below learning diagram you can find from this website: http://work.caltech.edu/slides/slides02.pdf. It's very interesting and impressive for me.
My question is based on the diagram.
As the lecturers said, the ultimate goal of machine learning is to find an optimal hypothesis from data that can replace target function which is impossible to learn. For a regression problem in machine learning, except for data quality issues such as outliers and missing values, imbalanced data is an import issue as well. Considering the fact that data quality is one of the most important aspects that can influence the final hypothesis that we may find, I am thinking that if it is helpful for us to learn probability distribution of input data when we apply machine learning methods like ANN for solving the issue: imbalanced data. Of course, it is based on the assumption that probability distribution of input data is learnable.
1. So the first question: is probability distribution of input data learnable?
2. The second question: Is it possible for us to obtain any insight from probability distribution of input data to solve the issue: imbalanced data?
Notes: Supposed input data x is generated by a unknown distribution P(x), written as x∼P(x).
Thanks for your answers!
To learn a general distribution is impossible, statistics or modern machine learning uniformly provide a "variational approach". The variational approach is an analog of projection in geometry or linear algebra. You start with a model, i.e. a specific mathematical form of function/distribution and try to "fit the data" by tuning the parameters. In another words, you project the target distribution (with an arbitrary form, can never be learned with certainty) into one that can be described by your variational framework. This projection is done by minimizing some sort of distance(not necessarily a metric). As in geometry you minimizing the Euclidean metric. In general variational approach you have to minimize the KL-distance denoted by KL(p(x)||q(x)) where p or q marks the target distribution or your model respectively. Let p be the target distribution, then the distance is approximated by the law of big numbers. Let q be the target distribution, then you can compute this value precisely w.r.t. a set of parameters and optimize. You may refer to any chapter about Probabilistic Graphical Models.
Hope it would be helpful. : )
You can find more video notes to help you. https://work.caltech.edu/telecourse.html
P(X) is indeed relevant, whether for classification or for regression; more specifically, P(X|Y) is crucial, e.g. for a Bayes classifier. As for P(X), the estimation of P(X|Y) is very inaccurate if the input dimension is large, as mentioned in Simon Barillet's answer. For moderate input dimensionality, Naïve Bayes may circumvent this problem. Anyway, all that is irrelevant for the class imbalance problem.
Variational autoencoders have been very popular a couple of years ago, but more efficient solutions have been found for large input dimensions and very large amounts of data.
- Badges
- Science topic