Hi Joseph Jose , below are my 2 cents regards your doubts:

1. Is training accuracy and testing accuracy defined for K-fold cross validation method or is it only defined for Hold-out cross validation(where data is split into train and test data) ?

Ans: For K-fold CV, training is done on the remaining K-1 folds, and tested ONLY on the Hold-out set, hence the accuracy corresponds to the Hold-out set only.

2. In training accuracy if model is tested on training sets will not the training accuracy 100% always?

Ans: Not necessarily. It depends on the complexity of the pattern to be captured in your dataset (for Regression), or the complexity of the decision boundary (for Classification). So for simple datasets, if you apply a complex ML algo or even ensemble methods, you will easily get 100% accuracy in the training set.

Conversely, if you have a dataset with lots of outliers, anomalies, missing values, categorical and numerical data mixed, and having quite different ranges, lots of correlated variables, or highly unbalanced dataset, even if you apply RandomForeest or XGBoost, you may not get 100% accuracy (or R^2 of 1) even in the training set.

Hope this helps. Cheers!

I can only add to Prashant Sahu answer that averaging accuracy from folds makes sense only if the tool is verified in folds separately.

Best regards

Hubert

Hi Prashant Sahu thank you for spending time answering my question. i still need some clarity recently i communicated a Machine learning paper, it was regarding fault classification (3 classes of fault) using SVM. i used 10 fold cross validation. One of the reviewer is asking for training accuracy and testing accuracy How will i provide this for 10 fold cross validation ?? from your answer i guess i will have to provide the accuracy of 10 sets used in 10 fold cross validation separately. please correct me if i am wrong and what about training accuracy? TESTING ACCURACY = accuracy of (TEST1+TEST2....TEST10)??

The sum of accuracies divided by 10 (number of folds). Then, you present averaged accuracy from CV. You are also able to discuss the variation of the accuracies across the folds - it shouldn't be high (if the model is reliable).

Dear Hubert Anysz ,

so accuracy will vary according to how i split the data ?

It can happen like that. However, to provide a real, reliable result we shouldn't manipulate the splitting into folds. The manipulation should be limited to providing near to equal representation of specific cases among folds and among training and testing subsets.

Joseph Jose , thanks for coming back :-)

One of the reviewer is asking for training accuracy and testing accuracy How will i provide this for 10 fold cross validation ??

Ans: Use a for loop( to run 10 times), within it do the train_test_split with test-size of 0.2 maybe (DO NOT specify the random_state). Now you have a train-set & test-set separately. Now You can do the classical training & test, and finally collect the results of the 10 iterations of the loop. You can make a box plot of the train & test accuracies, and/or report the average train & test accuracy alongwith standard_deviation. The stdev will give you the precision.

This method will be similar to K-Fold with CV=5. If you want 10-fold CV (not a good choice), use 0.1 in the test_size above.

2) Use the cross-validate function from scikit-learn, and use cv=5 or 10, with return_train_score=True. This function also to returns the train & test scores, fit times and score times.

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate

Hope this solves your problem now.

Prashant.

Prashant Sahu ,

thank you for the answer, it really helped me.

Cross-Validation (k-fold, nested cv LOOCV, etc.).

Random train/test splits, are both methods to estimate the model skills under evaliuation.

Cross-validation is a sampling procedure used to evaluate machine learning models on data sample (especially, limited data ).

Random train/test splits: is used to split dart to train , validation, test (if you have big data)

I would recommend that the cross-validation approach could be used as a tool to help you estimate the skill of models. you calculate these estimates so you can compare models and tuning hyperparameters. After that, you have to choose a model and its config based on the estimated cross-validation result with the best hyperparameters, then you can train the chosen model and test it on hold-out test set.

I think the reviewer do not know the conventional data processing/analyzing. In order to generalize your model over new/test data for accuracy, you do not need to count or consider the performance of your training data. A simple generalization is split your data at hand. Reserve particular data for training and the remaining for testing/cross validation purposes.

1 - Testing accuracy is definitely not computed by cross-validation. Test and validation (whether hold-out validation or cross-validation) have different purposes.

The purpose of validation is to select the model complexity that is most appropriate given the available data. The complexity of a model is governed by its "hyperparameters" such as the number of hidden neurons for a neural network, the degree of a polynomial kernel of an SVM, or the width of a Gaussian kernel of an SVM, the value of the regularization constant of a soft-margin SVM, or whatever hyperparameter(s) is required for the chosen type of model. Hence, the purpose of validation is to find the (hopefully) "optimal" values of these hyperparameters, given the data used during the validation phase.

The purpose of testing is to estimate, in a statistically significant way, the generalization ability of a model that uses the values of the "optimal" hyperparameters found at the end of the validation phase. Testing consists in applying the model having these hyperparameters to data of a data set (rightfully called "test set") that have never been used for any of the previous design step, i.e. neither for feature selection nor for model complexity selection. The accuracy - or whatever performance index is chosen - obtained on the data of the test set is the only statistically significant estimate of the performance of the model. If the model has been designed correctly, the test accuracy and the cross-validation (or hold-out) accuracy should be on the same order of magnitude. If the test accuracy is significantly worse than the validation accuracy, the model is overfitted to the training/validation data, hence is worthless.

Therefore, when designing a model by machine learning, first split the available data set into two disjoint sets: a training/validation set and a test set.

Validation can be performed by several methods, the popular being hold-out validation and cross-validation. Hold-out validation consists in (i) splitting the training/validation set into a training subset and a validation subset, (ii) training a model on the training subset, and (iii) applying the resulting model to the validation subset.

Cross-validation consists in splitting the training/validation set into K disjoint subsets, iteratively training a model on the data of K-1 subsets and validating on the data of the K-th subset (more details can be found in any good textbook or good on-line course on machine learning).

As a general rule, hold-out should be preferred for "large" data sets (thousands of examples), and cross-validation for datasets of moderate size (a few hundred examples); for small data sets, leave-one-out (i.e. K-fold cross-validation where K = number of examples of the training/validation set) is preferred. Whatever the validation method, it provides a score of the model ("hold-out score" or "cross-validation score", e.g. the cross-validation accuracy).

For each possible set of hyperparameter values (e.g. for each number of hidden neurons, in a reasonable range, for a neural network model, or for each value of the Gaussian kernel width, in a reasonable range, for an SVM model with Gaussian kernel), perform validation as described above; select the set of hyperparameters that provides the best validation score. That set of hyperparameters is the (hopefully) optimal set given the data of the training/validation set.

At this point, a model using these hyperparameters should be trained on all the data of the training/validation set (if cross-validation was used) or of the training subset (if hold-out validation was used). As mentioned above, that model should be applied to the data of the test set, which have been used neither for feature selection nor for tweaking the hyperparameters. The performance (e.g. accuracy) on the test data is the only statistically valid estimation of the generalization ability of the model.

To summarize, a manuscript that stands a chance of not being rejected from any good machine learning journal should (i) explain clearly the feature selection method, (ii) explain clearly the method used for finding the hyperparameters (model selection method), (iii) provide the validation score (e.g. the validation accuracy), AND (iv) provide the test score (e.g. the test accuracy).

2 - If the model is tested on the training set - a meaningless procedure, clearly - the test accuracy will obviously be equal to the training accuracy, but there is no reason why the training accuracy would be 100%, especially if you use real-life data. Reporting 100% accuracy on real data arouses very strong suspicions that the model is hopelessly overfitted to the training data.