In a probability density function in a continuous space, probabilities between 0 and 1 are replaced by probability densities. So if the standard deviation or variance is very small, the densities can be bigger than 1. A delta Dirac function on a singe point needs to have infinite probability density on that point {with dimension 0}, to that the probability mass integrates to 1.

Good luck with your research!

Probabilities should be [0 ,1]. Maybe you're looking at a value from the population?

Thank you Dariusz for your consideration

I'm not sure what is your point: "looking at a value from the population".

I've created a Gaussian Mixture Model using some training data and then I've used this model in order to decide about new samples but the probabilities for some new samples are bigger than 1. I've tried it with different components in matlab even with different data sets and the output was the same.

THanks

So if I understand correctly, you've trained a GMM to act as a classifier to decide the correct class of a new sample? And one of the outputs of the classification is a probability value that occasionally exceeds 1?

If this is the case, then the only thing I can think of is that somehow something isn't being initialized correctly. That is, the value that is supposed to contain your probability sometimes has a garbage value. Perhaps you need pre-allocate and zero-out your probability variables somewhere in your code.

Yes you are right I`m doing a classification task with GMM.

About the initialization, during the training phase it gives me a warning :: " Some posterior are ziro..."

But when I use the other data sets without any warning it has the same problem.

Thank you so much for your beneficial tips.

Probabilities should always be lower or equal than 1. Very small would be OK, but greater than one is not a probability, I suspect that you have a problem while calculating those values (a division by a parameter that tends to zero) and then you get the extreme values that are up to the numerical resolution of your software.

Hope that helps

Dear Reyes

Thank you so much for your consideration and advantageous comment.

I checked it. In the calculation of p(x), the Netlab source code has used the likelihood and in the likelihood calculation it uses Product of Cholesky decomposition of covariance matrix in its denominator. The problem is that product of the matrix is pretty small and tends to zero. But I'm wondering how should I cope with this. Maybe I should use log(likelihood) or something like that.

Thanks

In a probability density function in a continuous space, probabilities between 0 and 1 are replaced by probability densities. So if the standard deviation or variance is very small, the densities can be bigger than 1. A delta Dirac function on a singe point needs to have infinite probability density on that point {with dimension 0}, to that the probability mass integrates to 1.

Good luck with your research!

Florin is absolutely correct. A probability density function is non-negative and its full integral is one, so its integral over any region is less than one. However, there is no upper bound to the value of the probability density function. Furthermore, the numerical value of the density function depends on the scale or unit of measure, which usually is arbitrary and irrelevant to the actual problem being studied. This is not an issue particular to GMM or even to single multivariate normal distributions. It is true for any continous probability density. A trivial fix to having a numercial value of the density greater than one, if you actually need a fix, is to choose a unit of measurement such that the standard deviation in each dimension is always greater than one. You can achieve this, for example, by scaling your data to be a specified multiple of the (unscaled) standard deviation.

The fact that some of your values are very large and some of them are very small is a different issue. Again, this is not a property specific to GMM, but tends to happen with high-dimensional multivariant Gaussians in general. It is due to several causes. In part it is due to an aspect of the "curse of dimensionality". In a high-dimensional space, the "tail" of the distribution is spread over a very large volume of n-space, so the probability density function decreases very rapidly an observed values with real data will exhibit a much wider range of values than for single variant distributions. This situation will not change with the unit-of-measurement scaling I mentioned in my previous answer. They all will be scaled proportionately. However, scaling will lessen the extent to which some dimensions dominate or overwhelm the contribution from other dimensions.

Thank you so much Florin, your comment is perfect and leads me to the right way.

Dear Baker, I appreciate your comment they are pretty advantageous, practical and clear. Thank you again Sir.

Your comments are vary interesting and I'm intrigued about mathematical and statistical detail of GMM.

s there any mathematical proof or source which indicates that probabilities greater than 1 in GMM are OK? I am suppose to talk about the exact reason of these values.

Thank you

It is a bit harder to prove it for the Gaussian distribution which GMM use, but for just a uniform interval on one continuous variable x element of [0, 0.5] then at each point between 0.0 and 0.5 the probability density needs to be 2. That is because the integral over the whole space should be equal to 1

It should be said that with a GMM the prior on the probabilities of the different mixture models (which you need to choose beforehand) should be between 0.0 and 1.0, because this is a probability distribution over a small set.

The continuous densities come from the generative continuous probability distributions that use a variance or covariance matrix. If these have very small values, the width of the distribution is small, and therefore you also get probability density values higher than 1. If you put the variances and covariances to lower numbers, you will not get these probability densities higher than 1.

Good luck!

Sorry for my mistake. I mean: If you put the variances and covariances to lower numbers", I meant higher numbers since they represent the width.

Sajjad asked "[i]s there any mathematical proof or source which indicates that probabilities greater than 1 in GMM are OK? I am suppose to talk about the exact reason of these values. "

The word "OK" is not precise. If the question is whether probability density values greater than 1.0 are correct values, then you can easily construct a proof by example: For example, just take a simple, single univariate Gaussian distribution with a very small standard deviation. The probability density function near the mean will be greater than one. In fact it can be arbitrarily large, diverging to infinity as the standard deviation goes to zero.

However, if the question "are the values greater than 1 OK" as used in a particular application, then the answer depends on the application and the software. For example, if the numerical values of the probability density function are used with software that expects probability values rather than probability density values, then obviously things can go wrong. In fact they can go wrong even before the probability density function values great greater than 1 because, in any case, they don't match the properties expected by the software. In particular, this can b a problem if you plug these values into a canned software package, where you might not be able to determine what assumptions are made by the software for numbers that are called "probabilities".

Even if you only use the probability density function with your own software where you are careful to properly treat them as density function values, there can be practical problems in computations even though they are mathematically correct. In particular in high-dimensional multivatiate Gaussians, which I believe is your case, it can cause some values to be very large and some to be very small, as you have observed. This circumstance, in turn can cause problems with overflow, underflow or round off error. All of these uses affect computations on computers even if the mathematics (with infinite precision numbers) is perfectly correct.

So mathematically, having probability density values greater than 1 is certainly "OK" in the sense of being the correct value in certain cases. As a practical matter, however, whether it is OK depends on how the numbers are used.

Dear Baker

After reading your comment I think, in my case these large values are OK. Because I've used the GMM as a classifier and I've used the values in order to decide about samples. So I've simply set a threshold and take decisions based on this threshold. In addition, I've got acceptable accuracy near 90%. And as you said my feature space is 30D which is high. Thanks for your great comments.

Dear Marco, Yes you are right and as i said I checked it . I've used full covariance matrix and the covariance matrix value is very small which bring about large probability values. Thanks for your consideration and your supportive replies.

Thank you all professors

I encountered a similar problem on my dataset. First I used a dataset to train the GMM model and then tested it on a different dataset. While calulating the loglikelihood probabilities on new test vectors, the value of loglikelihood probability at some points reached infinity. The reason being zero probability density value for those feature vectors. However, I observed that increasing the training dataset size resolved the issue. Still, I want to achieve the results with the same dataset size. Please help me on this issue.