Hi,
I need to classify a collection of documents into predefined subjects. The classification is based on TF-IDF. How can I determine whether unigrams or bigrams or trigrams...or n-grams would be most suited for this? Is there any formal or standard way to determine this?
Also, how to determine the most appropriate number of features I should consider?
Any help would be highly appreciated.
Manjula.
depends on your corpus.
you can use this as a feature selection and select the best using cross validation accuracy.
Hi Manjula,
Generally, it depends on the task, so I recommend reading previous work in your area to figure this issue out.
On the other hand, applying features selection can be useful after knowing the required tokenizer (i.e., unigrams, bigram, etc.). In addition, obtaining the "accuracy" is NOT enough to evaluate certain prediction model, you need use another measurements, such as ROC curve, for evaluation purpose.
HTH.
Samer
Dear Vikas and Samer,
Thank you very much for your answers. It seems both of you suggest that the best fitting tokenizer, size of feature space, etc. will depend on the corpus we use, nature of the documents we classify, etc. Hence, the best way to know the most suitable model will be classifying a set of test documents and inspecting the accuracy, ROC curve, etc. Am I correct?
Hi Manjula,
Generally, extract text features, preprocess them. After that, apply classification algorithms, and evaluate them using accuracy, ROC curve, etc.
HTH.
Samer
The best approach depends on the type of data (what the text is is about), how much you have, if you have already labelled data to train a model (without this you'll need to use unsupervised approaches and have a strategy for evaluation) and if you want to use the model on new, unseen data later on.
For your evaluation, be sure that the categorisation of test data is done independently of the model (ie: don't look at model output and say "yes, looks good", instead categorise some test data FIRST, then see if the model agrees).
Be sure to use good experimental design here so as to be sure to detect overfitting. That is:
This approach is important to get a realistic estimate of the quality of your model on new data. Even when you only want to classify data you already have, use this approach to estimate the quality of your final model, as overfitting may be a problem even then (it'll add a hidden bias to your categorisations), and it can be hard to detect post-factum.
Information Gain is the easiest way to assess which features help your model best, but you will need to build those features first to get the evaluation.
All your ideas are very helpful to improve my work and thanks a for the contributions.
@Ian:
Could you please clarify me the importance of evaluating the system again (using ~30% of data initially not used) after selecting the best model? Why is it not enough to conclude the performance of the best model based on the results obtained in Step 2?
Thanks.
- Badges
- Science topic
- Similar topics
- Correction