Dear Mr. Ahmed,

sorry that I could not directly answer your question right now. I think this is interesting, however, and I would like to discuss. I would like to ask you some details, because I do not know what kind of data you fitted.

You used the word "accuracy", does it mean you already have a theoretical result? Also, I am wondering how did you define the "fixed number" of components. Does it based on theory?

Sincerely,

Kuan-Wei

Hi Kuan-wei,

I am using GMM to make a Universal background model (UBM) (general model of all classes of certain brain signals). This UBM is then used to create adapted models from training data of a single class. The means of the adapted model are then used as features and fed to the SVM for classification (My paper https://www.researchgate.net/publication/277727924_Grading_Hypoxic-Ischemic_Encephalopathy_Severity_in_Neonatal_EEG_Using_GMM_Supervectors_and_the_Support_Vector_Machine ). This approach is well documented in

 https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/0508_WCampbell.pdf

Accuracy:

I am using leave one out file to estimate the classification accuracy.  Just to clarify that in my case when each file is classified, the result is a single instance which is the class of the file. So if I have 64 files and 50 are correctly classified then accuracy is 78%.

By fixed number of components I mean, that  during the training phase in each iteration of Leave one out,  I don't use model selection to give me the best number of GMM components. Instead, I fix the number of components to say 16 and then train each model for unseen test file with same number of components. 

Does it make sense ?

Now my question was, if such was the use GMM, then what could be the best criteria to perform model selection for GMM. Should I use BIC values to select the number of components or should I replicate the classification stage completely (where we use means of adapted model to be classified by SVM).

Attached presentation has some figures that may explain it more easily

Article Grading Hypoxic-Ischemic Encephalopathy Severity in Neonatal...

Hi Rehan :)

Interesting question. Recently I have been using Bayesian nonparametric methods for this kind of work. The general idea of these methods is that the number of components will be determined from data, so your input to specify the number of parameters is reduced. 

The obvious model class to consider is the Dirichlet process GMM, but you may also want to consider the Indian Buffet and Chinese Restaurant processes. (See links below.)

Cheers, 

Niall

https://en.wikipedia.org/wiki/Dirichlet_process#Use_in_Dirichlet_mixture_models

http://www.jmlr.org/papers/v12/griffiths11a.html

https://en.wikipedia.org/wiki/Chinese_restaurant_process

Thanks a lot Nial, this Dirichlet process stuff is really worth exploring. Just read the intro of the paper you sent and it does make sense. Very well explained.  I think there is an implementation of DPGMM on scikit. Will try it and let you know. 

Dear Rehan,

Thanks for your explanation. I am really interested in this topic, but I am just a student in Statistics without PhD. degree, so I might be wrong. If there is anything wrong in my saying, please let me know.

I have read your article, your field is outside my specialization. However, as we all know that the GMM method is mainly for dealing with over-identification problem, and the BIC criterion is for dealing with overfitting problem and trying to avoid large models. Due to these reasons, I would like to ask if there is other advantage to choose GMM method?

In your article, in my humble opinion, I think that you need neither GMM nor BIC. That is the reason why you choose the variables could perform better than BIC. Furthermore, just for curiosity, I would like to ask how do you decide the variables used  manually. Is that only based on GMM-UBM method?

And I agree with Dr. Nial, choosing other model specifications might be a good idea.

I hope that does any help for you.

Sincerely,

Kuan-Wei

Hi Kuan, 

Thanks for your interest :)

The reason we use GMM is because we want to capture the dynamics of different grades of EEG. The idea here is to create a general model first that will represent entire EEG space. This will include even the artifacts and all the other activities that actually may not are of interest but are present in the EEG signals. Now when this model is adapted by the data of a certain grade of EEG then it will result in a more tighter model that will not only be specific to that grade but also to the general EEG.

To elaborate is usefulness a bit more. Lets say an artifactatous epoch comes in at classification time. Then if we compare it to the models that only represent single classes of EEG, the likelihood of that epoch to all these models will be very low. Because none of the model saw such data at training time.

Now with the proposed approach such an epoch will adapt the general EEG model in a way that it will move the means where its closest to the EEG grade and also the means of certain artifactous space as well which was modelled at the training time. Now the likelihood will be higher to the model of a certain grade. This is because all the grade/class specific models are adapted from a general EEG model which has seen such kind of artifactous data.  This approach has its roots in speaker identification area which is a very similar but mature field.

We compared the results with other classification approaches before publishing them and this approach yielded by far the highest accuracy.

Regards,

Rehan

In my opinion if you want to improve overall accuracy you should optimize whole process (with SVM classifcation). More GMM components might give better (lower) BIC but this will create larger supervetors and that will affect SVM performance. So maybe holistic approach to analyze whole system is better in your case.

Are you tunning SVM parameters for diffrent supervector sizes?

Hi Stanislaw, 

Thanks for your reply. I pretty much agree with you. I also think that higher dimensional supervector probably doesn't result in better SVM model and thats the reason of getting better accuracy through lower GMM components. 

Yes I do model selection for every SVM model as well.  

Regards,

Rehan

use nonparametric Bayesian models to learn the optimum complexity directly 

Hi  Ahmed.

i have  the same problem.

i have found a paper who proposed a new method to estimate the number of components.

The proposed method seems more important than the usual method such like :AIC,BIc.., especially :estimating the number of components without any priori ! , and more efficient that is obvious by the curves.

Article Estimating the Number of Components in Gaussian Mixture Mode...

refer to this blog https://towardsdatascience.com/gaussian-mixture-model-clusterization-how-to-select-the-number-of-components-clusters-553bef45f6e4