well precision is essentially the amount of noise that each of your algorithm generates while recall is the exactness of the result. So this essentially would mean that the algorithms generates similar amount of noise to the amount of correctly generated data labels. So if I am not mistaken, this means, that the better algorithms, match more target samples but also generates proportionally more noise. This would mean that you have increasing recall but similar precision

Yes, that must be the case., more false positives with increased recall. Not such a strange phenomonon is it? Do you also have a NO-classifier? Same there?

Do I understand correctly that your database consists for 50% of YES and 50% of NO instances? So recall of BoW performs below the base rate of change?

Let us consider how the accuracy can be improved:

1.- Label a false NO as YES

2.- Label a false YES as NO

In both cases the accuracy increases in 1/N (assuming you have N samples), but the precision and the recall change in different ways in each scenario. In the first case, the precision is untouched but the recall is incremented (in particular, it is incremented the inverse of the number of positive examples), while in the second case the recall is the same but the precision is increased.

I guess that it depends on the particular cases, but it seems that the improvement of your metric (which is, in a 60% yes-centered) is bigger in the first scenario and, maybe, this guides the optimization algorithm towards a solution where almost all the positive examples are recovered.

Another issue you should check is how the search heuristic you use moves from one (or several) solution to another (others).

Martin is right, the proportion of true positive and false positive documents remains stable.

Precision = true positive / (true positive + false positive)

Recall = true positive / (true positive + false negative)

In your case, for each true positive you have one false positive; therefore, precision remain stable.

For each true positive you do not add false negatives (it would be more precise to say either that you add a fraction of false negative or that the proportion of true positive and false negative is greater than 1).

@borga I don't agree. In scenario 1 where you label a false no as yes, both recall and precision increase if nothing else changes. You add an extra positive to your return set so noise reduces.

In scenario 2 where you label a false yes as no, the recall stays the same (obviously) and the precision increases again.

So what must happen in Myathams case is not only labbeling false no as yes but labelling true no as yes.

And i meant to say CHANCE ofcourse, not CHANGE in my first reply.

I think Borja is stating the point. Your objective function is not fair in terms of precission and recall. It is better for the convergence of your algorithms to learn labeling a false NO as YES than doing the oposite. Changing just one label has a greater efect in the first case, and this is the case that makes the recall to increase faster than the precission (the other case leaves the recall untouched and increases the precision). You can see a numeric example in the attached figure.

Sorry. Here you have the figure

It looks like for your problem identifying true negatives is as important as identifying true positives. However, the Precision (P) and Recall (R) formulas don't care at all about true negatives (P is not looking at the negatives at all) and your recall is going up just because you have less false negatives. So, in order to differentiate between your classifiers, you can either just use Accuracy, or compute another set of P and R values for negatives: look at the negatives like they are positives and vice-versa.

My interpretation of your results suggest that your classifier is roughly getting right the same proportion of YES but the amount of false negatives found is smaller. Basically, the classifier is not getting any better at classifying correctly a YES but is smarter at wrongly classifying a NO as a YES.

Another way to understand better what is happening will be to check the F-measure, recall and precision for the NO class.

Radboud, you are right, in the first scenario both incrase. It can be easily seen fron the equations (i pretended to add them, but finally i removed them and, together with them, the change in the precision). As I have previously said (and this correction reinfoces it), I would say that in the first scenario the improvement is bigger. I also agree with Dan and Julian, if you want to equilibrate the prediction in both clases you should use a symmetric objective function.

Thanks for all for their very useful comments. In fact, some researchers in textual entailment area look at f-score while the others look at accuracy. So, in my test I choose to optimise a mix of f-score and accuracy since optimise these factors depends on applications.

I done some experiments before on optimise f-score only and optimise accuracy only. Below is the results for these runs in terms of P, R and F for YES class only and the accuracy.

optimise (f-score)

---------------------------------

p:58.11 r:92.31 f:71.32 acc:63.00

optimise (accuracy)

---------------------------------

p:64.94 r:66.89 f:65.90 acc:65.50

The results you have shown here suggest that increasing the recall is the best way of optimising the f score. If you want to have a trade off between precision and recall, you may try to use a kind of symmetric score, such as f score for yes + f score for no.

Given that it seems that the two metric (accuracy anf f score for yes) lead to different areas in the search space, an interesting thing to explore would be multiobjetive optimization heuristics. In other words, try to optimize simultaneously both metrics. In principle, although one would expect that both objectives arer not opposed, given the sults you have obtained, the MO could be worth a try

Coming back to your question ( to tell a coherent story about why precision and recall behaves like they do), let me insist with another numerical example.

Your systems learn searching their parameter spaces that are mapped to the space of all possible responses. Each point in this space has values for precision and recall that can be plotted ordered by F-score (I am using a 10Yes+10NO example in the attached figure, that is, 110 different responses where F-score is computable). If you also plot the difference between both values, you can discover that the high F-score area is populated by more configurations showing higher recall than precision (and viceversa) (the moving average clarifies this appreciation). This is clearly linked to the fact that learning to increase recall moves the system further in the F-score axis than learning to increase precision. Looking from this side, you can see again that your values are coherent with the a-priori expectation defined by the F(yes) part of your mixture function.

So, I think we all are coming to coherent "stories" about your values and the nature of the objective function. Another question is if this is the function representing the behavior you want for your system, or if you are willing something different. (and, of course, if your learning algorithms are doing their best given that function).

The world of objective functions and evaluation is tricky and there is no ground truth out there. If you want a perfect recall system, it is easy, build the system that returns all the items :-) (there is not a symmetrical saying for precision. F is a mix of different nature entities)

Maytham,

Perhaps your metric for evaluating the performance of your classifier should take into account whether a YES is a good thing or a bad thing in your domain of application. If a YES is a good thing like, for example, whether or not a product on an assembly line is within specification, then precision is important so that defective units are not shipped to customers; and recall is a secondary issue of reducing cost by accepting as many good units as possible. Conversely, if YES is a bad thing like, for example, whether a product is defective, then recall is important as you do not want defective units to ship to customers; and precision is a secondary issue of reducing cost by rejecting as few good units as possible.

If a YES is not good or bad, but just an arbitrary label, then the f-score (in my understanding of it) measures how close the performance of the classifier is to the ideal of perfect precision and recall. You would have to look at how the f-score changes with changes in precision and recall (partial derivatives?) to see the relationships among the three metrics - these relationships (covariances?) are not linear. I would suggest plotting a contour map of f-score as a function of precision and recall as a means of getting an intuitive insight into these relationships.

(I admit I didn't read all comments, so this might have been said already).

Provided you have confidence scores (not only binary decisions) it might be interesting not to look only at the F-score but compare the ROC curves, in order to see what's going on in more detail

In order to have a better explanation, perhaps you should take a closer look to your independent variables and the way they are used by different classifiers.