I would like to forecast the flashover parameters (surface resistance, inception voltage and flashover voltage) of polymeric insulators using four input features. But the size of the samples is 44. How to deal with this lower sample in the machine learning context.
Dear Jawad Ahmad ,
Look the link, maybe useful.
https://towardsdatascience.com/breaking-the-curse-of-small-data-sets-in-machine-learning-part-2-894aa45277f4
Regards,
Shafagat
Dear Jawad Ahmad ,
Look the link, maybe useful.
https://towardsdatascience.com/breaking-the-curse-of-small-data-sets-in-machine-learning-part-2-894aa45277f4
Regards,
Shafagat
In case of images you can use data augmentation in which you need to flip the images into various angles to increase the number of images. You can also use one shot learning technique in which there is less number of training data.
You can also increase the dataset by repeating the values more than once. This might be helpful.
Dear Jawad,
When dealing with small datasets in machine learning, it is important to have strong priors and use domain knowledge which can help models to overcome the limitations of training on few samples. As pointed out by Shafagat, there some techniques which could be worth trying, most of them also included and described in the following article:
https://www.industryweek.com/technology-and-iiot/digital-tools/article/21122846/making-ai-work-with-small-data
As a starting model, I would suggest to try simple linear or regularized linear models. Other complex models, such as neural networks-based ones, have higher complexity, i.e., more parameters to learn, thus requiring a generous amount of data to train them or, otherwise, they may easily end up facing overfitting issues (not generalising well in new unseen samples).
Best wishes,
Daniel
Hi,
Have a look here, might be useful
https://www.kaggle.com/rafjaa/dealing-with-very-small-datasets
Jawad Ahmad in small dataset, any classification or clustering algorithm will be perform accurate (correctly classified instances will be better than large dataset) as well as good to utilize and understood the resource and particular classifier or cluster. Moreover, best for developing own improvised classification algorithm exatcly what i'm doing now.
The accuracy of the output depends on the number of samples in the data set.
You can use Generative Adversarial Network (GAN)/ DCGAN/ cycleGAN/WesGAN. The output of G network can be used for oversampling your data.
Smaller data can be used in ML Testing but doesnot assure any proper outcome. There are many online tools to work on it.
Hi Jawad Ahmad
Have a look at these articles.
https://towardsdatascience.com/breaking-the-curse-of-small-datasets-in-machine-learning-part-1-36f28b0c044d
https://towardsdatascience.com/breaking-the-curse-of-small-data-sets-in-machine-learning-part-2-894aa45277f4
Regards
Hi
Please check the following work. It describes the process in details. I believe it will help you.
https://www.nature.com/articles/s41524-018-0081-z
Regards
Hi,
Please check the link given below. I hope it will help you.
https://www.quora.com/What-are-some-machine-learning-algorithms-that-can-learn-well-even-with-a-small-data-set
Thank you,
Arnab Barua
The sample size cries for a Design of Experiments techniques! I can not explain the full concept in a few words but I can give you some terms to search for.
So maybe it is a good idea to try a 2nd order polynomial model first. In literature it is called general linear model (GLM).
When you observe high VIF values while applying such a model you have to augment your design using the I-Optimal approach with some more observations the make you data collection orthogonal to your four independent influencing variables (the design factors of your product).
If new runs are not possible try to make orthogonal (stratified) subsets of your 44 runs using a bootstrap approach. Then you get bootstraped estimators for the model beta values.
Please see this link
https://www.kaggle.com/rafjaa/dealing-with-very-small-datasets
- Badges
- Science topic