I am applying MCNEMAR test to Compare 2 Machine Learning Algorithms.
Alpha Value Set to 0.01.
If P-Value is greater than Alpha Value,
1st: What is this mean that I am accepting Null Hypothesis?
2nd: Is this mean that my results of both algorithms is wrong?
3rd: What I have to do to overcome this or I can put this test in my documents?
Brief Detail Needed.
Thanks
1st: What is this mean that I am accepting Null Hypothesis?
No. Hypotheses are not accepted. You just failed to reject it. That means: the observed data are not sufficiently incompatible with the Null hypothesis. This does not mean that they are supporting the Null hypothesis. If I tell you that I am not sufficiently sure that it's now raining outside, then this is not any confirmation that the sun is shining.
2nd: Is this mean that my results of both algorithms is wrong?
No. It only means: the observed data are not sufficiently incompatible with the Null hypothesis. See above.
3rd: What I have to do to overcome this or I can put this test in my documents?
If your aim is to reject the null, use larger sample sizes. If your aim is to show that the algorithms are comparable, then find a smart way of a meaningful comparison, quantify/estimate the difference in some relevant feature and interpret this w.r.t. the subject matter.
Amir Aziz in addition to Jochen Wilhelm 's excellent answer, do not take p-values as the definitive way of deciding on differences, they have a very specific interpretation. The effect size is equally relevant, or 1 algorithm can score poorly on a small but critical subset of your data.
To estimate how well multiple decision makers perform (human or AI) you can also look at https://en.wikipedia.org/wiki/Inter-rater_reliability
I would recommend "The essential guide to effect sizes" for a very accessible discussion on this and related topics, such as understanding not statistical power and design, not just p-values, when you do test/formulate hypothesis.
Article The Essential Guide to Effect Sizes: Statistical Power, Meta...
I agree with Jochen; You reject the null or fail to reject the null hypothesis
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python