No problem. Well, let's see what this really is about.

Probably the simplest example is fitting a simple univariate Gaussian distribution to some data. The free parameters are the mean m and the standard deviation sigma. One just writes the likelihood function of the samples, takes the logarithm of it and maximizes by equating the derivative to zero. This leads to the well known formulae as the estimated mean is the average of all samples, and the estimated sigma parameter is the empirical standard deviation of the samples. So far so good.

If we have a multivariate Gaussian, one can do the same, although it will be a bit more complicated as matrix inversion needs to be solved. Anyways, the steps are the same.

When it comes the a Guassian mixture, it has as many free parameters as all of its components plus the weight of each component. For this problem, there is no closed form solution, however, one can apply the expectation maximization (EM) technique to iteratively approach and converge to the maximum a-posteriori estimation of parameters. This is what is usually implemented under the hood in Gaussian mixture models.

Now, the thing that all this works for the Gaussian is by chance, or nature, or God. For a more general distribution, for example, like the Gamma or Beta distributions, one can hardly derive any formula to fit its parameters to the data, not even in the univariate case. This is just an example, I didn't check, there might be some closed formula for the Gamma distribution, but what I want to illustrate is that it is not natural that for a complex, analytically described distribution one easily estimate the parameters. In many cases it leads to unsolvable equations. And it becomes even more tricky with multivariate distributions, and even more tricky with mixtures of multivariate distributions.

What I want to say, is that what works with the Guassian is due to its special properties, generally it does not work with other distributions, not even in the univariate case.

However, there is a solution, which is called probabilistic programming, closely related to Bayesian statistics. This is not an analytic but a purely probabilistic approach to fit the parameters of any distributions, and if you have enough data compared to the number of free parameters to be fitted, it works fine, and theoretically, it converges.

This probabilistic programming needs a different mindset. Unlike in frequentist statistics (what I described above), in this case you won't get a single point estimate that "hey, this is the mean of the distribution". Rather, you get a distribution for each free parameter, like the mean of a Gaussian. So what you get from a probabilistic programming framework is that the mean of the Gaussian generating your samples is 1 with 10% probability, 2 with 45% probability, 3 with 35% probability and 4 with 10% probability. You get a distribution like this for each parameter -- to be exact, you get a joint distribution. And you can take the most likely parameter combination (for example, 2 as it has 45% probability), this is related to the frequentist choice - the maximum a-posteriori, but it is even better to take the weighted mean and compute the expected value of this parameter, which will be something like 2.4.

This is only an illustration, but this approach works with any distribution with any complexity, and is based on Monte Carlo simulation. So this can be used to fit the parameters of any distribution of any complexity to any dataset. The thing that Gaussian Mixture Model works is very unique. In Python, the most widely used package is pymc3. You can find further description here https://docs.pymc.io/notebooks/getting_started.html

No. The Gaussian Mixture model assumes the data to follow a Gaussian Mixture distribution, which is a mixture of individual multivariate Gaussians. Thus, if your data generator is surely producing a mixture of Gaussians, the GMM model should be the right choice, given you have enough samples.

For other probabilistic-like clustering techniques I recommend checking fuzzy-clustering.

For implementations, sklearn in Python has GaussianMixtureModels, I used it multiple times, works fine.

György Kovács Thanks for answering. Do you know what would be the alternatives to Mixture model but with Non-Gaussian distributions ?

No problem. Well, let's see what this really is about.

Probably the simplest example is fitting a simple univariate Gaussian distribution to some data. The free parameters are the mean m and the standard deviation sigma. One just writes the likelihood function of the samples, takes the logarithm of it and maximizes by equating the derivative to zero. This leads to the well known formulae as the estimated mean is the average of all samples, and the estimated sigma parameter is the empirical standard deviation of the samples. So far so good.

If we have a multivariate Gaussian, one can do the same, although it will be a bit more complicated as matrix inversion needs to be solved. Anyways, the steps are the same.

When it comes the a Guassian mixture, it has as many free parameters as all of its components plus the weight of each component. For this problem, there is no closed form solution, however, one can apply the expectation maximization (EM) technique to iteratively approach and converge to the maximum a-posteriori estimation of parameters. This is what is usually implemented under the hood in Gaussian mixture models.

Now, the thing that all this works for the Gaussian is by chance, or nature, or God. For a more general distribution, for example, like the Gamma or Beta distributions, one can hardly derive any formula to fit its parameters to the data, not even in the univariate case. This is just an example, I didn't check, there might be some closed formula for the Gamma distribution, but what I want to illustrate is that it is not natural that for a complex, analytically described distribution one easily estimate the parameters. In many cases it leads to unsolvable equations. And it becomes even more tricky with multivariate distributions, and even more tricky with mixtures of multivariate distributions.

What I want to say, is that what works with the Guassian is due to its special properties, generally it does not work with other distributions, not even in the univariate case.

However, there is a solution, which is called probabilistic programming, closely related to Bayesian statistics. This is not an analytic but a purely probabilistic approach to fit the parameters of any distributions, and if you have enough data compared to the number of free parameters to be fitted, it works fine, and theoretically, it converges.

This probabilistic programming needs a different mindset. Unlike in frequentist statistics (what I described above), in this case you won't get a single point estimate that "hey, this is the mean of the distribution". Rather, you get a distribution for each free parameter, like the mean of a Gaussian. So what you get from a probabilistic programming framework is that the mean of the Gaussian generating your samples is 1 with 10% probability, 2 with 45% probability, 3 with 35% probability and 4 with 10% probability. You get a distribution like this for each parameter -- to be exact, you get a joint distribution. And you can take the most likely parameter combination (for example, 2 as it has 45% probability), this is related to the frequentist choice - the maximum a-posteriori, but it is even better to take the weighted mean and compute the expected value of this parameter, which will be something like 2.4.

This is only an illustration, but this approach works with any distribution with any complexity, and is based on Monte Carlo simulation. So this can be used to fit the parameters of any distribution of any complexity to any dataset. The thing that Gaussian Mixture Model works is very unique. In Python, the most widely used package is pymc3. You can find further description here https://docs.pymc.io/notebooks/getting_started.html