Recently, I have completed a research work on the application of spectral imaging for classification of browning level of development in button mushroom. The paper is currently under evaluation for possible publication in a scholarly journal. In the 2nd round of review process, one of the referees asked me to make some modifications on the paper as bellow:
"Many studies show that ks algorithm is not a suitable method for sample partition because this method leads to different distribution of calibration and test samples.Compared with KS algorithm, the random sample partition is a more ideal sample partition method, hence, the manuscript should use random method to create calibration and testing dataset. More importantly, this random sample partition method needs to be repeated several times, and the statistic method(such as T-test) should be used for model evaluation."
Personally, I am not in agreement with this comment. As I know, most of researchers prefer to use Kennard-Stone (KS) algorithm instead of random selection approach for dataset splitting. Please correct me, If I am wrong. What do you think about pros and cons of KS vs. random selection? Should I ago according to the reviewer comment and consequently repeat the analysis based on random partitioning?
The problem with KS is that you build only one training/test pair, unlike random partitioning and cross validation, which can be bootstrapped to build confidence intervals as well.
The Kennard–Stone algorithm allows to select samples with a uniform distribution over the predictor space, so based on your problem random sample selection much more matching, thanks you
Just as an update, please refer to the following article to see how to use statistical t-test technique to validate results of classification on randomly selected samples:
http://link.springer.com/article/10.1007/s12161-017-0845-y
http://link.springer.com/article/10.1007/s12161-017-0845-y
Article Non-destructive Identifying Level of Browning Development in...
Hi,
The question is quite old. But I wanted to share just in case someone will search for the answer here.
Here is a paper of comparing Iterative random vs. Kennard-Stone for FTIR spectra.Conference Paper Iterative random vs. Kennard-Stone sampling for IR spectrum-...
In short, the paper's conclusion (IRS-iterative random sampling, KS - Kennard-Stone):
> In conclusion, IRS is more reliable than KS in sampling representative training set.
In addition, I would also warn that it is important to control repeated spectra of the same sample (in lab meaning) to not appear in both train and test sets at the same time. Because that makes results skewed: test set is implicitly included in the train set.
The KS algorithm is expected to maximize the consistency between calibration and validation errors, because the validation samples are extracted in order to represent faithfully the samples' universe. However, if you want to check if your model has a really good predictive power, you should extract random samples from this universe. The validation dataset may be less representative of the samples' universe and the validation becomes more demanding. Furthermore, repeating the random sampling allows to have a mean and standard deviation of the prediction errors. So, to summarize, IRS sampling is more realistic; KS is more "idealistic"
Relevant: Article Kennard-Stone method outperforms the Random Sampling in the ...
- Badges
- Science topic
- Similar topics
- Linguistics
- Composition Studies
- Publication