I am a bit confused as to the usage of permutation tests to detect / disprove over-fitting, such as this case is, in machine learning.
Say I have a dataset, and they're divided into training / test set (10% and 90% proportion). There's only two labels (0 and 1) and a value for each data.
I illustrate the training dataset below:
Label Value
0 11
0 35
0 48
0 56
1 67
1 88
1 44
I would normally run a machine learning (ML) method to generate a solution and an outcome.
This solution will then be used in the test set, and the outcome observed.
I should expect similar performance in the test set as compared to the training set.
To measure overfitting, my null hypothesis is that there is no inherent relationship between label and value and I should get the same outcome in the test set if such labels are random. I guess what I'm trying to prove is that if I shuffle the label, I will not get similar performance when applying such solution to a test set.
I was told to use Y-scrambling at the moment. What I understand so far is as below:
1. Scramble the Y-vector (the label) N times to generate N number of permuted training set.
2. Run these training sets into the ML to get its own permuted solution.
3. Apply the solution to the original test set and observe its outcome, hence individual outcomes based on permuted training sets.
I now have the following information:
a. Outcome of the original test-set when using the ML solution provided by the original training set.
b. N outcomes when using the ML solution provided by the N permuted training sets.
My confusion is from this point on, what am I measuring? If there's a y-scrambling plot of X and Y, what would X and Y be? I've been reading about correlation coefficient and statistical test, and measuring differences of mean, but it would greatly help if someone can point up a bit more in detail what to do to represent this overfitting measurement. Thanks so much.
one of the most effective and quite easy ways to take overfitting under control is to add a "validation" phase between training and test, based on cross validation (CV). The easiest technique is the "k-fold" CV, in which you leave k training samples at any time in a cyclic training process and test on them the current training session. By alternating the k training samples removed from the training set, you can evaluate each training session. At the end I suggest you to take as final training weights those obtained in the worst case, i.e. giving the worst training performance. Doing so, you should be sure to avoid overfiting in your experiment.
In addition, you could perform a statistical evaluation of both training and test results. Whenever training performances are evidently better than test ones, it is a good indication that there is no overfitting in your training.
For a simple explanation of k-fold CV see http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29
Y scrambling (also called "target shuffling") is sort of like a statistical significance test - you can use it to estimate the probability that your ML solution's performance could have occurred by chance. If the probability estimate is low enough, you can have some confidence that your model hasn't been over-fit to a spurious relationship in your data. It's a good tool for preliminary testing of your model, before you move on to prospective testing.
Check out this blog post for a description of the method:
http://www.plottingsuccess.com/3-predictive-model-accuracy-tests-0114/
FWIW, I have (many years ago) found the ML literature to be confused about overfitting which ALWAYS occurs when samples are used and which is exacerbated by issues like small or unrepresentative samples, measurement error, small effect sizes (the predictors are only weakly related to the outcome variable). The number one source of overfitting in regression analysis is including too many low-quality predictors.
If I understand that you have a single predictor, then if you are randomly assigning cases to your test and training samples, I don't see what else you could do to reduce overfitting.
I don't think a significance test of a trivial null like random chance is very compelling. Massimo's suggestion of K-fold CV would be far more compelling.
Finally, I hope it's a typo in your question that you are using 10% for training and 90% for testing. Usually the bulk of the cases are used for training and fewer for testing (as in k-fold CV).
Some good points Alan.
I should clarify: Y scrambling is not something you'd do instead of cross-validation. It's a test you apply to the result of cross-validation. In some situations a cross-validated estimate of prediction performance is not as good as it seems, so I think the idea of Y scrambling is to put the estimate in context of what might be expected under the null hypothesis.
I generally don't use Y scrambling myself (I usually have other prediction methods I can compare to, which I think is better than comparing to the null hypothesis). But I think the concept makes sense, at least.
- Badges
- Science topic