It depends on what you are trying to achieve, and the characteristics of your data.

With respect to splitting the data into training/test (or train/validation/test) vs K-fold cross validation, it depends on the amount of data that you have, and how well this data represents the distribution of the data where you want to apply the model. In an ideal world, you would want to have an independent test set to verify the performance of your model. Sometimes the dataset is not big enough to be split into a training and test set with those characteristics, so people use cross-validation to use as much data as possible for both, training and testing.

Now, usually there are three questions that you need to answer in a machine learning problem. (1) How do I select the best learner/hyperparameters? (For example, should I use SVM with linear kernel, and C=0.1, or Logistic Regression with regularization parameter = 0.5?) (2) What is the expected performance of the best learner? (3) What is the final model that I will actually use?

For answering (1), it is enough to use either of the methods that you described: A simple training/test split, or K-fold cross validation. Simply train all your learner/hyperparameters combinations, and select the one with the best performance. This will allow you to know what is the model that you want to use, but the accuracy that you obtained using this procedure is NOT the expected accuracy of your model.

For getting the actual accuracy (question 2), you need to first create an independent test set, and then use the remaining data to repeat the process that you did for answering question 1. The procedure for question 1 will give you the best model using the training data. Now you use that model with the test set. That is the expected accuracy of your model. There are several ways of doing this: One is: First divide the data into train/test. Then apply K-fold cross validation in the train set. Other options are nested cross validation, divide the data into train/validation/test, etc... It all depends on the amount of data that you have available.

For answering question 3, you take the model/hyperparameters selected when you answered question 1.Then, train that model using all the data available in your dataset, without any splitting. This is the model that you will use for new incoming data.

Hello Mohamed,

If you are planning to tune the hyperparameters, the second method is the correct way. When tuning the parameters, your model fits on the whole data used on CV process. So you need external data to evaluate the real performance. However, if you have a static configuration and you are going to train the model without any change (one time), you can use the CV results for evaluation.

Good Luck

Hi,

I want to add that this selection depends on the nature of your data. If you have fully independent data, e.g. random image samples that you found somewhere in the internet, you should use K-fold cross validation. If your data has an inherent correlation, like some measurements from subject 1, some from subject 2 and so on, you should split the data that you create independent data from different subjects. Otherwise you verify your machine learning algorithm with known data and you cannot say anything about the generalization of your system.

Best regards

Simon

Hi Mohamed,

Here is another paper discussing assessments metrics for learning from imbalanced data. I hope it helps.

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5128907&casa_token=3RlXWOGnh7AAAAAA:7ZbjIccFtUbcneX2Fjje6yeIDivGtOgA7r71fiFy3fJliNDg6-X7MxDiCBTWdiBHeEafUpF-EQ

Best,

Hasan

It depends on what you are trying to achieve, and the characteristics of your data.

With respect to splitting the data into training/test (or train/validation/test) vs K-fold cross validation, it depends on the amount of data that you have, and how well this data represents the distribution of the data where you want to apply the model. In an ideal world, you would want to have an independent test set to verify the performance of your model. Sometimes the dataset is not big enough to be split into a training and test set with those characteristics, so people use cross-validation to use as much data as possible for both, training and testing.

Now, usually there are three questions that you need to answer in a machine learning problem. (1) How do I select the best learner/hyperparameters? (For example, should I use SVM with linear kernel, and C=0.1, or Logistic Regression with regularization parameter = 0.5?) (2) What is the expected performance of the best learner? (3) What is the final model that I will actually use?

For answering (1), it is enough to use either of the methods that you described: A simple training/test split, or K-fold cross validation. Simply train all your learner/hyperparameters combinations, and select the one with the best performance. This will allow you to know what is the model that you want to use, but the accuracy that you obtained using this procedure is NOT the expected accuracy of your model.

For getting the actual accuracy (question 2), you need to first create an independent test set, and then use the remaining data to repeat the process that you did for answering question 1. The procedure for question 1 will give you the best model using the training data. Now you use that model with the test set. That is the expected accuracy of your model. There are several ways of doing this: One is: First divide the data into train/test. Then apply K-fold cross validation in the train set. Other options are nested cross validation, divide the data into train/validation/test, etc... It all depends on the amount of data that you have available.

For answering question 3, you take the model/hyperparameters selected when you answered question 1.Then, train that model using all the data available in your dataset, without any splitting. This is the model that you will use for new incoming data.

It all depends on the application ..if your application demands to find some trends or patterns presence in it then it's advisable to perform only 2 fold cross validation or simply divide the data into test and train set. ...otherwise you are supposed to go with k-fold cross validation..but I think if you want to find unbiased result the you should havery separate test set and on the remaining set you can perform k-fold cross validation.

You need to reserve part of data for training and the remaining for testing purposes is a usual procedure.

splitting the data into training and testing datasets then evaluate the model on the basis of any validation method is basic procedure to get efficient result. i suggest you to try all validation techniques to find better accuracy.

Actually, there are many factors that could help to better understand the situation, but just to explain a little bit, when you use K-folds cross-validation, K-1 folds are used as training data and the remaining fold as a test data. If you try to optimize hyperparameters, I suggest splitting your entire data into 3 subsets, a train data which will be used to build your model, a test data to test your model performances, then you can apply K-folds cross-validation into train/test subsets solely either to optimize hyperparameters OR to have a clear observation about your model with various subsets of data. After that, you will use data from the validation subset that has never been seen from your model, and then you can clearly check the real performances of your model in a pseudo-real-life scenario. As you are dealing with an unbalanced dataset, choosing appropriate metrics such as F1-score, ROC-AUC and PR-AUC is preferable (avoid using accuracy).

If there is sufficient data, then k-fold cross validation might be preferable because splitting in training testing sets could be biased / customized

Roberto Vega gave a detailed articulation. To add to that, Percentage splitting (training, testing & validation ) is most suitable when working with large amounts of data since you can have sufficient data for training, testing and validation. Sometimes the data is just enough to allow training and testing. In case of limited amount of data, one would opt for Cross validation. The two approaches can be combined when working with more data. Note that cross validation might make use of a lot of computing time, especially if the folds are finer. I dont understand what Ranjan Parekh is implying though.....

Hi,

You can try both options: (a) dividing in K-folds or (b) dividing in training and test. It depents mostly on your data characteristics. From my exprience with imbalance data I prefer option (a), specially when you applied resampling techniques (oversampling, undersampling or hybrid).

To emphasize Roberto Vega's answer, if you are doing parameter estimation, feature selection, etc, it is very imp to have a separate test dataset that remains untouched until testing is done. It is not uncommon to see researchers incorrectly using the whole dataset during training and over-estimating the classifier performance.

For imbalanced datasets, you have two possibilities:

1. If you apply a processing technique as undersimpling or eversampling, you use all dataset to obtain a balanced one. Then, when you apply a learning algorithm to this balanced dataset, you apply K-cross validation for evaluating the performances of obtained model.

2. If you apply directly a learning algorithm, you use initially K-cross validation.

Good luck.

Both are right, however if you do sampling of the dataset before model training then option 1 is better...which keeps the definition of k fold cross validation justified...

I would suggest k-fold cross in your case. You have sufficient data for that.