I want to work on a machine learning and pattern recognition task, but the size of data set is small and there are only 43 samples for both training and testing purposes. How about the bagging method?
How should I deal with this problem? Is there any way to find the reasonable size of data set or is it only based on trial and error?
Thanks in advance.
You have not mentioned how many features you have. Just wanted to remind that if you have like 200 features and 43 samples you are in trouble, and careful feature selection is the only way to avoid overfitting. If your number of features are ok then above suggestions should help.
Sajjad:
Check two options...
1. Boostrapping: http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
2. Montecarlo cross validation: http://www.sciencedirect.com/science/article/pii/S0169743900001222
Fernando
Dear Sajjad,
Possibility #1: Perform a leave-one-out cross validation. More specifically, for each sample, hold that sample out as the test case. Run the system with all the other real samples as the training set and see the system's recommendation for the test case.
Possibility #2: Artificially increase the number of the samples via duplication (optional by making slight modifications).
Best regards,
Yaakov
When only limited training data is available, supervised ensemble methods such as Random Forest commonly perform the best. However, if the unlabelled data is available, you can take advantage of it to improve the predictive performance. Hence, you can reformulate the problem as semi-supervised learning. This paper may help you: http://pages.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf
Hi Sajjad,
As people have mentioned before you have a few options:
1) cross validation using leave one out, leave many out etc more details in the link below:
http://en.wikipedia.org/wiki/Cross-validation_(statistics)
With such a small sample set leave one out cross validation would be the best option here in this section however other methods as shown below..
2) bootstrapping
http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
Someone has already posted this link above however i found this useful link:
http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-12.html
3) Increase sample size - this is a tricky one as this can be done depending on the distribution of your dependent variable in your dataset. You can use for example SMOTE (a type of oversampling technique) however there are other oversampling techniques. As well as already what has been mentioned, semi supervised learning. Finally, other ways to obtain data would be the typical literature search or maybe there is a industry collaborator you can obtain some data from.
In an ideal situation an external validation set (data that is completely separate from the training of the model) would show the predictability of any model built however when you have such a small dataset doing this would waste vital data therefore the internal validation methods stated in points 1, 2 above are needed. However, be careful as just because the internal validation statistics might be high they too (like the training set) might be over optimistic of the actual model predictive performance. Just something to bear in mind!
Hope the above helps,
Danielle
Hi, First, having 43 samples is not too small. Second, for better training use leave one out (LOO) cross validation, as mentioned before. I had data set of that size and LOO produced good results.
P.S.: ANN can even learn XOR function with just four data set points! So 43 is big enough!
I also support leave-one-out cross validation. I agree with Mahmoud that 43 samples is not too small to achieve something, but with such a small dataset, unless the problem is trivial, you will need to maximise that availability of the training data.
I want to show respectfully my disagree about LOOCV.
Check this interesting paper.
http://www.academia.edu/7388093/Measuring_the_prediction_error._A_comparison_of_cross-validation_bootstrap_and_covariance_penalty_methods
Hi Sajjad,
I recommend you to use leave -one -out method which is dedicated for small samples for training and testing on the same sample.
You have not mentioned how many features you have. Just wanted to remind that if you have like 200 features and 43 samples you are in trouble, and careful feature selection is the only way to avoid overfitting. If your number of features are ok then above suggestions should help.
I tend to agree with Valeri. From my observations on many datasets, the ratio between no of samples and features should be more than 10:1 to get sufficiently decent results. But this ratio also varies with applications, IMHO it cannot be generalized. Assuming you satisfy this requirement, the above suggestions such as LOO is popularly applied. K-fold CV is a good one.
I would recommend using Monte-Carlo resampling (subsampling) or nested resampling, maybe in combination with a regularization method (Elastic Net, Ridge). Anyway make as much use of your data for training as possible. You could also perform a PCA during training in order to possibly end up with a smaller number of predictors (at the cost of interpretability).
Hi,
You can use distant supervision. Here you can find a simple tutorial (with implementation level clarity):
https://youtu.be/uj71XNRsGiE
Dear Sajjad, this question can be attributed to small sample size problems (SSS). In medical field, it's well known as 'large p small n' dataset problems. Some sample generation technologies have been proposed, such as virtual sample generation (VSG), and data augmentation.
You can use adversarial concept to augment your data. More details are in Zhu, Wentao, and Xiaohui Xie. "Adversarial deep structural networks for mammographic mass segmentation." arXiv preprint arXiv:1612.05970 (2016). https://arxiv.org/pdf/1612.05970.pdf, IEEE ISBI 2018.
- Badges
- Science topic