Hi,
Clustering terms or clustering document existing on document collection is mainly based on the similarity based on the term-weighting scheme used.
Please, What is the shortage and limitation in clustering techniques in Information Retrieval when using TF-IDF and its variations?
Thanks & Best wishes,
Osman
.
https://stackoverflow.com/questions/17537722/better-text-documents-clustering-than-tf-idf-and-cosine-similarity
http://ieeexplore.ieee.org/document/6258608/
.
IDF and its variations are based on semantic independence between the terms on the same document for those terms that repeated in the same frequency. However, if there are two terms repeated with the same frequencies in the documents, they should have semantic relation between them. This issue can affect on the clustering performance.
The stackoverflow section above misses the point. TF/IDF LSA and many similar techniques rely on statistical properties of language. These properties only emerge in the large - so an argument based on keyword matching over two documents is a spurious argument for these techniques.
So really one of the key shortages and limitation of TF/IDF and the many improved variants is that they will only work well if you have a large document collection to deal with.
- Badges
- Science topic