Hi,

TF-IDF and other mathematical term-weighting schemes are based on Luhn idea for measuring the information content of the terms in the documents. The improvement of these mathematical term-weighting schemes is the probabilistic and language models. However, probabilistic and language models are based on relevance judgement values in IR or classes specifications in text classification. On the other hand, relevance judgement values and class-assignment can not be gathered accurately without term-weighting scheme at the beginning before learning them. Thus, the need for TF-IDF or statistical term-weighting scheme at the beginning is a critical issue.

Thanks 

Osman

Let’s consider a corpus of 100 documents, we will look into the TF-IDF & term distribution scores for different cases:

A term with low occurrence  (e.g. Hamlet, to be coherent with your question) should have a high score.

Doc 1: TF-IDF =  7*log(100/2) = 11.89, while the term occurrence = 7/(7+3)=0.7

Doc 2: TF-IDF =  3*log(100/2) =  5.10, while the term occurrence = 3/(7+3)=0.3

TF-IDF = 50*log(100/100) = 0, while the term occurrence = 50/(50*100) = 0.01

TF-IDF = 20*log(100/50) = 6, while the term occurrence = 20/(20*50) = 0.02

The above example illustrates that the advantage of TF-IDF is twofold:

However, if you just want to group documents on a specific term or expression, you might as well just use a regex on each text for speed.

I wasn’t entirely sure what you meant with “… for feature selection”, so I hope this helps.

Hi Bart de Clerck,

I think you should read about TF-IDF and other term-weighting schemes.

Thanks

Osman

Thank you, Osman and Bart. 

I understand how TFIDF works. but feature selection process implies ranking the features according to its ability to identify the documents of interest. This means that a word "Hamlet" with scores tfidf=11 and distribution=0.7 will achieve the highest scores identifying Shakespear document.

Meanwhile, the term "the" achieves scores of tfidf=0 and distribution = 0.01, which will allocate them at the lowest possible ranking for all terms on both cases. 

So what is the advantage of TFIDF over frequency distribution? what I can see so far, it makes the high scores higher and low scores lower. 

By feature selection, I mean selecting keywords that Identify the class the most, which is usually formulated as a ranking problem. this ranking considers some score that is generated by correlation metric or similarity metric or some scoring technique such as TFIDF or K-Best

IDF discriminates the index terms in the context of document collection that may be has the same topic domain. Thus, some terms may repeated frequently in all or most of the documents. Consequently, these terms are not good discriminator for the document. But TF-IDF does not consider the variations in documents and queries length that affect on cosine similarity and other similarity measures. Also, some terms may be repeated one time in most of the documents in the collection but it also may be repeated more frequently in the rest of the documents in the document collection which can be consider elite documents for that terms. TF-IDF treats these terms similar to the other non-signinficant terms. This may affect in IR effectiveness

 @ Osman,  can you please explain your comments by example to emphasise the difference?   

Please, check Information Retrieval: A Survey report in google. There is a discussion about this issues with examples. Also, you can check my paper for alternative term weighting (term frequency-Average term occurrence for information retrieval ) ( TF-ATO with DA)

@ Osman, I have checked the two documents you mentioned and none of them answered my question. 

What is the advantage of TF-IDF over term-weight-percentage (term distribution)?