Suppose we want to train a MLP neural network in order to learn the underlying probability density function (pdf) of some real-valued data. That is, the network must learn p(x), where x is a continuous random variable (possibly high dimensional vector). My proposed solution is as follows.
First a nonparametric density estimation method, such as Parzen (kernel) method, is used, and then its result is fed as training data to the neural network. More precisely, at each data point the value of pdf is obtained via the nonparametric estimation, and these point/value pairs are used to train the network in a supervised manner (i.e. using backpropagation).
Is this method okay in theory, or I have missed something?
Regards.
.
same as Marko above ; once you have a non parametric density estimation, what is the point to train a NN to learn it ?
if you want the density ... well you have it from your non parametric estimation or am i missing something ?
or is it that you expect your NN to properly interpolate / smooth the imperfections of your initial density estimation which might occur because of the high dimension ? not very likely ...
.
Actually, In my project I need to obtain a parametric model of the distribution, because of the application. It is not possible to store all of the data points. So I thought it might be a good idea to first create a nonparametric model, then convert it to a parametric one.
You could use kernel density estimation; it is a model with parameters (less than a neural network) and focused on the task you want to solve. In addition you can find R software for this task. Of course, if dimension increase you will need to increase a lot the sample size to obtain an accurate approximation to the pdf, and this cannot be avoided using NNs (or any other universal approximator).
Dear Amirhosein Ahmadian, I'm following this discussion and I'm wondering what you mean by: 'I need to obtain a parametric model of the distribution, because of the application'. If what you need is a parametric pdf model, please note that it is not what you get by training an ANN, for the ANN parameters have not direct meaning in terms of probability. By contrast, in many interesting ways, you can regard your Parzen as a model with meaningful 'parameters', if you take the Parzen kernel centers as free parameters (i.e. let the centers be adjustable in agreement with the goals in your application, not constants as in usual Parzen.).
Besides, if Parzen is cumbersome, because there are too much kernels (therefore the number of 'adjustable parameters=kernel centers' is too high), you may cluster your training data into a convenient number of representative centroids (e.g. through K-means or any other clustering method) and then use these centroids to adjust your Parzen. In fact, this path will probably lead you to something between Parzen and Gaussian Mixture Model, both in terms of complexity and performance.
- Badges
- Science topic