If one computes multiple topic models on the same corpus, is there a measure that allows one to choose which model is the best one. I did look at various internal coherence metrics, including Cumass, as well as some external consistency measures, Cuci, but I found that with low alpha, those measures may be higher than with high alpha, yet be associated with clearly inferior topic models.
You can consider to incorporate multiple topic models into a hybrid model based on learning-to-rank. Thus, the united model may incorporate advantages of each topic model and outperform each single model.
Incorporating multiple topic models is not appropriate since our objective is to compare algorithms for topic model solutions (for example LDA vs LSA vs CTM vs ...). We also want to do this in an unsupervised context. This is the reason why I am looking for some metrics to optimize each model (allowing us to select the best LDA possible to compare against the best LSA possible, etc.
How about optimizing the hyper-parameters? Have you implemented your own LDA or you are using some available package?
Hi Ashkan, I am using Mallet. We can adjust the alpha, but how can we decide which value of alpha is the best one. In other words, how do we optimize the hyper-parameters (in a unsupervised context).
Optimizing Lda is a big research concern,because enough exploration of both types of inference techniques including various Gibbs sampling and variational inference are done in last two decades since 2003,various packages in R like textmineR, Text2vec includes these implementations,But interpretation of generated topics in various applications are still not up to the mark or comparable to human.
So instead of hyper parameter,much work needs on inference techniques.
Meta-heuristic Algorithms (e.g GA) can be used to tuning the LDA parameters
There are several already existing code for that. Just search google for any of them
- Badges
- Science topic