We know that we need to split our data in training and testing subsets, so we can have have the estimation performance of our model on new data. We do this when training a model that have to "learn" like neural networks for example. But do we need to split our data when performing simple statistical comparisons? For example, in many clinical papers they plot the ROC curve based on only one feature and they report the classification accuracy according the ROC which was build using the complete dataset, is this correct?
Performance measures of classifiers that are estimated from the same set of data are always biased. An unbiased estimate can be obtained only on new data. As J.A. said, splitting a coordinately acquired data set into a training and a test set reduces the bias but will not completely remove it.
So, as so often in statistics, there is no "correct" vs. "incorrect", there are only different shades of "usefulness", depending on the aims and on the context. Sometimes you only have biased estimates, and you may discuss the possible bias. There is also the possibility that the (potential) bias might not be relevant for your problem.
Edit: typo
I would say technically yes, from a practical perspective not really. What you want to know is the classification accuracy of unknown samples (samples that were not used in the construction of the classification model). To achieve this, some form of cross validation can be used (or double cross validation if meta parameter need to be optimised). The classification results of unknown samples will give an idea of performance for new samples.
However keep in mind, that there also will be sample bias. Typically samples are collected simultaneously and will be alike to a certain extent. If you start classifying samples collected at a later instance, it may be much harder to classify them correctly than the crossvalidation initially may have indicated.
Performance measures of classifiers that are estimated from the same set of data are always biased. An unbiased estimate can be obtained only on new data. As J.A. said, splitting a coordinately acquired data set into a training and a test set reduces the bias but will not completely remove it.
So, as so often in statistics, there is no "correct" vs. "incorrect", there are only different shades of "usefulness", depending on the aims and on the context. Sometimes you only have biased estimates, and you may discuss the possible bias. There is also the possibility that the (potential) bias might not be relevant for your problem.
Edit: typo
An approach worthy of consideration is cross validation. This is particularly useful when the amount of data available is limited (as you correctly note). The general approach used often in model building using data mining techniques (ie., classification trees, etc.), is to apply a "10 times 10-fold cross validation." This basically means that you would run the training model on 90% of the data and test it on the remaining 10%. Then you would repeat this for each of the 10 holdout samples and collect all the goodness of fit measures. Then, you would repeat this all 10 times for a total of 100 times. A good model should provide good "goodness of fit" statistics across the test samples.
For good coverage of CV on a book that is online, see
http://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf
It is not good idea that you divide your data into the training and test set because you cannot obtaine the 95% CI.
See my papers using " the "k-fold cross validation "method after 2014.
- Badges
- Science topic