Hello James,

I used this test for the comparison of two classification algorithms. Based on McNemar’s test results you can decide which algorithm performs (statistically) better regarding accuracies, which are used in the formula. For the value X, calculated by McNemar’s test, holds: if X=3.84 (or lower), when algorithms have the same error (with probability ~95%). Otherwise, if X>3.84, when the performance of one algorithm is higher.

McNemar’s test is used for testing generalisation ability, e.g. of a new designed classifier. Further, it is well-suitable for the situation, where training set and validation set are predefined.

to my knowledge the formula is: (|N12 - N21|-1)^2/(N12 + N21)

Thanks Helene, you are right that another form exists, most of the reference works I have seen give the formula I stated.

For the benefit of readers, here is a reference presenting the formula in the question:

http://www.statisticssolutions.com/non-parametric-analysis-mcnemars-test/

and one for Helene's:

https://www.graphpad.com/quickcalcs/McNemarEx.cfm

Whichever form it is presented in would make no difference to the question as stated as it would have the same implications. I am not interested in this thread about relative merits of the two forms.

Apologies, Helene I wasn't shown your first answer initially and missed it. I may be a bit slow, but could you explain which terms in the formula capture accuracy?

Testing generalisation, I can understand that but I think it doesn't full explain what exactly the test is measuring. To me the test tells us whether or not two tests disagee on different samples. I think it is testing against the null hypothesis that the two tests detect positives and negatives the same samples. When it fails the null hypothesis this means that there is likely to be a meaningful difference in which samples are assigned positive and negative.

It is possible for tests to have identical accuracy but still be significant in McNemar's if one is weighted towards reporting false positives and one towards false negatives.

Hello James,

for comparing performances of two classifiers (let say A and B) it is in the formula above: N12 – number of examples misclassified by A, but not by B; N21 number of examples misclassified by B, but not by A. Null hypothesis assumes that A and B have the same error rate if N12=N21.

In my tests I used McNemar on the same sample. As I understand you have different samples and one single classifier. The question in your case would be: how perform a classifier on different samples.

Thanks for the reply Helene. I'll address the last point first so you know the context matches your typical situation before continuing with my queries that arise from your response.

McNemar's test must always be used on paired outcomes (so performed on the same samples for classifier comparison), so I am not describing a different scenario. It would be impossible to get any values for the contingency table if the classifiers were applied to exclusive samples.

The only way I can see N12 representing the 'number of examples misclassified by A, but not by B ' as you describe is, if instead of the usual contingency table, you transform it into a truth matrix with true/false for A as columns and true/false for B. I've never encountered this approach in my reading but I can't claim to be extensively read on McNemar's test and would appreciate some useful references on this. In that case I would understand the null hypothesis would be that there is no difference in the likelihood of A being false dependent on B being correct compared with the likelihood of B being false dependent on A being correct. That still doesn't sound like an 'accuracy' test to me (it would ignore all the false predictions both tests record) but still boils down to the same issue of skewness in disagreement.

The approach I've always saw was to use the usual contingency table of predicted class for A vs predicted class for B. When comparing two classifiers (I would never describe the ground truth as a 'classifier' so I assume you are talking about two classifiers, neither of which is considered ground truth. So apologies if this is a source of confusion) my understanding is that the contingency table contains zero information on misclassifications. If I have misunderstood I am afraid you will need to explain to me how I can derive that information from solely the cells N12 and N21 of the contingency table.

The elements used in cells b and c of the standard contingency table comparing two classifiers are just looking at disagreements with no judgement on which is correct. So N12 is not indicating the number 'misclassified by A, but not by B', but as I stated the ' number of samples classified by method A as belonging to class 1 that are classified as class 2 by method B '.

In a situation where the differences between classifiers is completely random then one would expect on average half of N12 to be correctly classified by A and half by B, the same goes for N21. Based on the N12 and N21 alone I cannot see how we an infer the balance of 'correctness' between the two tests.

I'm afraid I am not able to see how your explanations can be derived from the formula, so you'll need to be more explicit in connecting your statements to the elements of the formula and outlining your assumptions to help me grasp it.

My explanations are based on:

Alpaydın, E.: Introduction to Machine Learning. The MIT Press, Cambridge, 2 edn. (2010)