Deal fellow researchers:
I am looking for references to explain how to interpret if the Cohen's kappa statistics is different on training and validation datasets. Some of the machine learning models had almost similar Kappa-value, however, some of them deviated a lot.
I will appreciate for sharing published works on this issue.
Thank you and best regards,
Sushant
Dear Sushant K. Singh , this article here might be helpful. It explains quite neatly why Cohen's Kappa is important when you are dealing with highly imbalanced datasets (resp. why measures like classical accuracy might not capture the performance increase of your model very well):
https://www.knime.com/blog/cohens-kappa-an-overview
Taking this into consideration, my first guess would be that your training and validation datasets might sometimes be unequally imbalanced.
If you are looking for a more quotable reference, you might also want to check out these:
https://ieeexplore.ieee.org/abstract/document/5584447
Article Comparison of classification accuracy using Cohen’s Weighted Kappa
Good luck with your research!
Dear Annika Hamachers , thank you very much for sharing such wonderful references. Let me go through them. I guess, I kind of got an idea about the potential reasons for higher Kappa score on the validation data. Let me confirm.
Appreciate your kind support for this!
Thank you and best regards,
Sushant
Dear Luca,
Thank you very much for your kind response and for sharing the reference. I will read the article.
Best regards,
Sushant
Maybe these jobs can help you
Regards Reynald
https://www.sciencedirect.com/science/article/abs/pii/S0022030221001983?dgcid=rss_sd_all
Article A machine learning approach to monitoring and forecasting sp...
Article Is Natural Language Processing the Cheap Charlie of Analyzin...
Dear Reinaldo Requeiro ,
Thank you very much for sharing references, and I will go through them.
Best regards,
Sushant
It’s simple. Train and test means, agreement or disagreement between the two. I will either agree or disagree to something. If Kappa value in MSA is higher it’s better. More than 90% means accuracy is excellent. Anything less than 60% should be taken as a weak model. It means the model is not being able to differentiate strongly. Overall the difference between train and test should not exceed 10%
Hi Anuraj Nayarisseri ,
Thank you for your kind response. Could you please share a reference for the 10% rule?
Best regards,
Sushant
- Badges
- Science topic