I was performing a binary classification problem with 15000 RGB images using a scratch build CNN model. While it comes to evaluate the model, I can do it in two ways:
1. Splitting data Train and Test and use 10 fold cross-validation for the training data. Later with the best model, I would use the unseen Test data. In this way I got appx. 91.5% avg. accuracy for both test and validation.
2. Just use 10 fold cross-validation and got 92.5% avg accuracy(slightly better result than the previous one.)
Which option would be the best for reporting the performance of my model in the research article?
TIA
Regardless you should actually not be using accuracy as your primary evaluation metric in a ML classification problem, especially a deep learning problem. A lot of examples exist out there that use accuracy and while they are plenty they are not always correct in doing so. For your situation given classification that is binary you should be using AUC, PRAUC, F1 score, Binary Class cross-entrapy, F Beta score, Precision, or Recall. These would be more relevant to your problem and give you a better idea of if you are hitting what you are looking for. You should in reality be using two, one metric to confirm if the model is classifiying correctly, and another to ensure you are indeed getting what you are classifying for. So, a combination of metrics is the best bet. The use of accuracy is highly misleading in any classification problem and should not be given much thought or credit for model performance. Hopefully this helps a bit. Good luck.
Great question. I would suggest following published guidelines such as CLAIM (Article Checklist for Artificial Intelligence in Medical Imaging (CL...
).Item 35. Report the final model’s performance on the test partition. Benchmark the performance of the AI model against current standards.
Item 36. For classification tasks, include estimates of accuracy and their precision, such as 95% confidence intervals. Apply appropriate methodology such as receiver operating characteristic analysis and/or calibration curves. When the direct calculation of confidence intervals is not possible, report nonparametric estimates from bootstrap samples.
I suggest you follow https://www.machinecurve.com/index.php/2020/02/18/how-to-use-k-fold-cross-validation-with-keras/ and https://datascience.stackexchange.com/questions/47797/using-cross-validation-technique-for-a-cnn-model
Alexjandro Daviano thank you for your answer. Actually the accuracy or the performance metrices was not my focus right now. My question was is it necessary to use unseen test data to proof that, the model is working good in unseen data or the just k-fold is enough.
Muhammad Ali Edward Peake Thanks for sharing. I will check them out.
In that case, kfold is never enough alone. So yes, always use a train, validation, test method to ensure a model is ready for a production like task. Especially if the dataset is very large, because in that case CV may not be the best approach. But with CV you always need to check the model against a test set at the minimum to ensure that you have not overfit the model an that your results are not a fluke and did not by chance perform well on the selected sample data. So, always use at least a test set. An when you can a good approach in the future when you are comparing models is to do CV on the train set and actually evaluate the model on the validation set to compare to the other models and see how well it did. Then once the best model is chosen evaluate that on the test set. This ensures you have a robust model and that its ready for a production environment to actually be used in real-world tasks. Hope this helps.
- Badges
- Science topic