In case of ANN I divided data set on three groups: training:validating:testing in a proportion 70:15:15 or 60:20:20. The validating is not used for training, but just as a protection from overfitting. During finding the weights of ANN when MSE of training set is decreasing and MSE of validating too - the training is continued. When MSE of the trainig set is still decreasing but MSE of the validating is starting to rise - that is the moment of the end of training (not to overfit).

Matlab ANN Toolbox and Statistica make it automaticly. In a simple ANN excel file of Augshuman Saha it has to be done by hand (the stop of process).

Many examples can be found where 100 rows of the database were enough for successful ANN use. The size of the database should be correlated with the number of connections between neurons (the number of the wieghts to be found by the algorithm in ANN). It can be found in the literature, that the number of rows should be approx. 10 times higher than the number of the weights, but my calculations (and found in the literature too) shows, that even if the relation is not kept (there is more weights) the predictions or classifications are successful too (with low errors).

best regards

Hubert

First, thanks for very interesting question.

For imbalanced data sets such metric as accuracy is essentially non-correct - it may be, e.g., 99% if we assign all objects for majority class.

About values of AUC, TPR, FPR, etc.

1) What is your positive class - minority or majority ?

2) What is your degree of dis-balance ?

3) What did you use as Validation Set - set after SMOTE using or part initial data set?

Regards, Sergey.

Greeting Segey,

Thanks for your response.

I first solved the problem of imbalanced data using SMOTE method.

then after, i used accuracy , AUC and other metrics to evaluate the model performance.

1)my positive class is minority.

2)the degree of unbalance is 10% vs. 90%

3)i used 10-fold cross validation.

Thanks

Tusneem

I don't agree with your reviewer. In my opinion, using of 10-fold CV can provide correct estimations to avoid over-fitting. Certainly, amount of positive objects on each validation fold will be small (6...9), but it isn't problem. For example, for using of LOO (Leave-One-Out) CV for balanced data sets this amount often is zero -:). Sorry, once more question (it is very interesting for me !) :

What did you use as Validation Set on each of 10 runs of CV - extended set after SMOTE performing (large data set, not real/initial) or part of initial data set (real, size of 6...9 objects)

The measure of overfitting can be then the difference for the test data of MSE for overfitting case minus MSE for optimally trained tool.

It seems to me that what you did is pretty standard

I would just repeat cross validation ten times.

a dataset of size 750x5 is definitely not small, you can look for papers on that but generally small sample size in machine learning problems is below 100 (https://www.nature.com/articles/s41524-018-0081-z).

You can always use balanced accuracy instead of accuracy, but generally the AUC is robust against class imbalance (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-119 )

The reviewer's objection - as you presented - is not really relevant, because overfitting concerns the training not the testing: if your model overfits the training data then automatically will perform poorly on unseen data.

Maybe the reviewer was concerned with overfitting after oversampling?

PS: Random Forest allows undersampling at each tree level, so you do not really need to oversample: in your data balanced randomForest is going to sample 750*0.1 samples from each of the classes

For class balance issue, you also have to be careful how you do the final test. If your original data is imbalanced and you believe that it is representative of the world at large. Then you have to keep that in mind during the testing step and actually test your final model on imbalanced data. Otherwise you wouldn't know how your model would actually perform. Here, I agree with Sergey Porotsky that overall accuracy is not a good measure of your model's performance. As an alternative, you could compute the error rate individually for each class, you would want similar error rate across all classes. Better yet, examine the confusion matrix and compute other performance measures.

Additionally, I would refrain from using synthetic data during the final test, since it is possible that some of the synthetically generated data points would never occur in the wild and your algorithm could be learning wrong. Besides, since you're using cross-validation, you will end up testing every one of your available data points.

Also, because you're using cross-validation, then by "final test" I mean the test at the end of each cross-validation step, the one you sum and divide by the number of folds. By the way, another helpful evaluation measure is the variance in your performance metrics across all of the folds. If variance is high, then your model is not doing well on some regions in the feature space.

When you don't have much data and computation time allows for this, leave-one-out cross-validation is probably the best way to go.