Dear Hooman Abdollahi,

I suggest you to see links and attached files on topic.

https://towardsdatascience.com/cross-validation-430d9a5fee22

https://www.emerald.com/insight/content/doi/10.1108/BEPAM-08-2014-0035/full/html?casa_token=woAds-

https://www.eng.auburn.edu/~aesmith/files/ieeesmc.pdf

https://shodhganga.inflibnet.ac.in/jspui/bitstream/10603/227187/14/chapter%205-results%20and%20discussion.pdf

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Article A survey on addressing high-class imbalance in big data

Best regards

Here "sometimes" is the crucial word. It depends on your problem, the number of classes and the distribution of your dataset to find a good balance. If you have A LOT of very similar and equally distributed targets in your data 98% could be enough.

With the 2000 observations, you have what would give you only ~40 test cases. I don't think (without knowing the specific problem) they will be enough to represent all the variety in your dataset to give you a good metric for testing.

The most important question you have to ask yourself if the test data represents the dataset as a whole AND if your metrics give you valid answers for the problem you are asking.

Hooman Abdollahi. I agree with Michal Rapczynski . The split between training and test depends on size of your whole data, as well as distribution of response or class variable. If you have a million images, you can very well take 90% or even more data to train because even at 10% test, you have substantial data to validate your model. But an important thing to note that you are forecasting (time series problem) and not predicting, so your data must not be shuffled or randomized to preserve the time sequence. You need to look for trend in the data and then capture test set which covers the whole trend similar to what you see in training.