Dear Team,
I am using Kolmogorov-Smirnov Test to check the model performance.
I have built 3 models and the accuracy is around 70%- 76% for the models (the higher the better).
Is there any threshold or a limit that we consider as a good fit for the model.
Please suggest.
There is no particular threshold or limit that is a standard "good fit" level for agreement between your three models by using Kolmogorov-Smirnov (if I understand your question correctly). As I understand it, the Kolmogorov-Smirnov test compares the cumulative distribution of events (such as number of accidents per hour over a 24-hour period) with some theoretical or known distribution, to see if the new distribution is different. The difference is called D and is tested statistically. You don't provide enough information to understand your experimental situation, but it sounds like you might want to use the kappa statistic or inter-rater reliability, where 70-76% agreement would be considered "good", although this is only a rough guideline and entirely depends on your experimental design and other factors.
As you use the (Kolmogorov) statistics in a goodness-of-fit sense, you cannot get a judgement of the size of the sample. With small samples, any model may fit, with large samples, you might get significant deviations from the given model. All in all, an "accuracy" of 70% cannot be interpreted as absolute value as it depends on sample size.
It might be a good idea to compare your models with alternative models with different inherent properties and calculate the "accuracy". If such alternative models still have a good value, you cannot discriminate between these models and would need more data.
If very similar models have large differences in accuracy, then the sample size is large so that even small changes in the model would lead to large changes in the accuracy.
A few information about the "good" value for K-S test was wrote here: (https://qualityamerica.com/LSS-Knowledge-Center/statisticalinference/k-s_statistic.php) "The K-S statistic reported is alpha, where alpha is the reject level for the hypothesis that the fitted curve is the same as the empirical curve. K-S should be a high value (Max =1.0) when the fit is good and a low value (Min = 0.0) when the fit is not good. When the K-S value goes below 0.05, you will be informed that the Lack of fit is significant." I'm trying to get a limit value, but it's not very easy. At the moment, I performed the following steps to calculate a Pearson's R value to decide if my set of values are normal or non-normal. Using Origin prgogram: (i) you should create a new book or worksheet and add your values on column; (ii) then click Plot->Statistics->Q-Q plot; (iii) finally, Analysis->Fitting->Linear FIt. You will able to evaluate the correlation parameter and decide about the normality of your distribution. This article can help: Asghar Ghasemi, Saleh Zahediasl. Normality Tests for Statistical Analysis: A Guide for Non-Statisticians. Int J Endocrinol Metab. 2012;10(2).
I am getting two different answers here.
Jerry Miller said that the the KS value ".... the Kolmogorov-Smirnov test compares the cumulative distribution of events with some theoretical or known distribution, to see if the new distribution is different. The difference is called D and is tested statistically...."to which I agree. The KS value should be as low as possible to have the same CDF of both empirical and theoretical distribution as I have read/understood from other literature.
But Ivan Pires de Oliveira says KS value should be as high as possible.
Kindly help.
- Badges
- Science topic
- Similar topics
- Philosophy
- Moral Philosophy
- Free Will