I have a big dataset (repository) which contains text data. The repository contains a data related to many different domains so that I can search about any domain or general topic.
when I search in my repository about general topic (e.g; nano-technology), I would like to get all sub topics related to that topic. Someone else also wants to search about another topic in the same repository and get the subtopics. What I’m going to do is: developing a tool which is able to identify the topics for a data which returned by search query. each time I search, I get different data=>dynamic
I have tried to apply clustering algorithm like k-means, and k-means with Canopy, as well as the topic modeling (LDA). But unfortunately for both the flat clustering like k-means and term-based clustering like LDA, there is no specific way to automatically set the right K (#Clusters, or #Topics).
Is there anyone has any idea how to identify topics in dynamic text? I mean how can I bring the clustering algorithms into real world to work in real application?
Thanks in advance.
You may like to read about text mining and specifically how to calculate the tf–idf frequency–inverse document frequency), which is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. So, you can a have ranking score(s) for all the documents in your corpus relevant to your query search: "nano-technology" for example.
To learn more about tf-idf score, you may like to read:
http://en.wikipedia.org/wiki/Tf%E2%80%93idf
http://pyevolve.sourceforge.net/wordpress/?p=1589
For implementation of tf-idf in Python, you may like to check this URL:
http://aimotion.blogspot.ca/2011/12/machine-learning-with-python-meeting-tf.html
http://en.wikipedia.org/wiki/Tf%E2%80%93idf
http://pyevolve.sourceforge.net/wordpress/?p=1589
http://aimotion.blogspot.ca/2011/12/machine-learning-with-python-meeting-tf.html
Hi Noha,
thank you.
I know much more about tf-idf, pre-processing filtering, nlp, ... so on. The question is about identification of topics in the results of search query? How Do I know previously number of topics/clusters?
I would like to use the clustering algorithms/topic modeling in real application to deal with a dynamic data? "the input parameters of algorithm should be changed according to the nature of dataset".
any Idea??
Thanks;
Dear Donni,
I understand you have two separate problems. The first is that your data is too large or your data just keep comming (dynamic). Secondly you want to determine number of clusters.
To address the first problem, I think you should have a look at the stream/ online text processing with LDA.
To address the second problem, checkout hierarchical LDA.
Following links might help you out.
http://www.cs.princeton.edu/~blei/topicmodeling.html
https://github.com/jessykate/streamLDA
Kind Regards
Abdul Wahid
Hallo Abdul,
Thank you. Of course I have a large dataset(big Repository), but I want to identify the topics only in the data which I search about. so different search==>different topics.
Do you think that the estimation of topic numbers by hierarchical LDA is good in practice?
I'm searching about topic identification method that really helps me to use it in the real application.
Best Regards,
Donni
Sorry, I misunderstood your question.
You may like to read about "subtractive clustering" that determine dynamically the number of clusters. I attach a paper that uses it:
I hope this may help,
Noha
Article Clustering: A neural network approach
It is an interesting problem...
Do you know what are the potential topics that might show in your datastream? Is it always texts about cars or is it cars today and bicycles tomorrow?
Do your data come with any metadata, so you potentially could train a classifier?
Found this interesting article on "Real-time topic modeling of micro-blogs' - http://www.oracle.com/technetwork/articles/java/micro-1925135.html
Hope you will find this useful
- Badges
- Science topic
- Similar topics
- Engineering
- Computer Engineering