I am, familiar with Ordinary Least Squares Regression (OLS), which produces an equation for forecasting  a  dependent variable, Y(est),from given X values. A residual is simply Y(given) - Yest). The sum of residuals =0. In Marketing, one typically wants to target areas in which there is a large protential to increase sales. So, select the areas where Y(observed) - Y(est) has highest  negative value. This tells us: There is a great "Opportuniy Loss" in these areas, and one should invest in increasing sales figures.

Additional baseline methods for text classification

1- Random guess classification

2- Rule based classification

3- Decision Tree 

Dear Anwar,

Thank you for your answer.

Could you explain why the three methods mentioned by you are regarded as baseline methods for text classification?

Could you support your claims by related papers or other evidence?

Best!

Yaakov

Dear Yaakov,

Before answering your question let me clarify my view on how to select the baseline method for text classification. I think the decision on baseline method selection should be taken in light of your research hypothesis about the classification methods. In other word, I think you won’t find a consensus on a common baseline classification method. The reasons why I regarded the above three methods as baseline classifiers can be explained as follows:

1- Random guess classifier is regarded as a baseline classifier because by intuition any other informed classifier outperform this classifier

2- The rule based classifier is regarded as a baseline classifier because in text classification there two paradigms: rule-base (old) and machine learning (current). Due to the impressive level of effectiveness achieved by machine learning paradigm, it becomes the current trend in text classification. Therefore, if your classifier falls in the machine learning paradigm, it is almost intuitively that it outperform the rule-based classifier.

3- Finally I regarded decision tree as a baseline classifier because it was mentioned in the following most popular reference in text classification “Machine Learning in Automated Text Categorization“ by Sebastiani. It was mentioned that decision tree was used as a baseline classifier in many research ,but definitely there must be some intuition behind that.

R. G. Rossi, R. M. Marcacini, S. O. Rezende, Benchmarking text collections for classification and clustering tasks, Tech. Rep. 395, Institute

of Mathematics and Computer Sciences - University of Sao Paulo (2013).

http://sites.labic.icmc.usp.br/ragero/docs/TR_395.pdf

Dear Rafael,

Can you summarize here what are your recommended baseline results for text classification?

Best regards,

Yaakov

I use algorithms from different paradigms:

- Multinomial Naive Bayes and Bayesian Logistic Regression (Probabilistic)

- Knn (Instance-based)

- C4.5 and Ripper (Symbolic)

 - SVM (Statistical)

My criteria is as follows:

IF I want to add complexity to my model, this MUST worth the effort.

In other words, a baseline solution could be the following scenario: take a state-of-the-art binary/multi-class classifier and test it using a BoW feature set. I think this is the simplest solution to the text categorization problem, it performs well, and any addition to the BoW that enhances results means that it provides useful info.

You can add some "typical" features ... features that are usually combined with BoW such as number of tokens ...

As for learners, since you are using the "cost-efficiency" (in terms of complexity and implementation) criteria, you can use SVM or MaxEnt, which are widely known to work well on text-classification.

Other aspects depend on your task.

In the best case, any machine-learning classifier can perform as best as an experienced human. However, even two humans can not agree with each other in some text classification task, or even the same one may change one's decision at different time. 

Therefore, if two human raters disagree in d% of all documents, the best machine-learning classifier can not perform better than 100%-d% correct classification, because if the machine classifier 100% agrees with one rater, it will at the same time disagree with the other rater in d% documents.

This is a hypothesis on the performance upper bound of a machine-learning classifier. I haven't published this work. But some articles have touched the idea:

Yuen-Hsien Tseng and William John Teahan, "Verifying a Chinese Collection for Text Categorization," Proceedings of the 27th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '04, July 25 - 29 Sheffield, U.K., 2004, pp.556-557.

The above article mentions about how to detect agreement in the training documents automatically. It can be considered as a feasible way of knowing the agreement rates among the human classifier.

So, if your classifier's performance is distant from this rater agreement rate, then there may be some room to make improvement. On the other hand, if it's performance is close enough to this human agreement rate, then the machine could be said to perform  well enough.

Conference Paper Verifying a Chinese collection for text categorization