Currently, I have trained a Mask-RCNN model on a dataset(train and Val)and chosen the final model by considering the minimum validation loss. Now I need to evaluate the model performance. So to do that I need to know how to perform k-fold cross-validation. According to my knowledge, I know during the k-fold cross validation if I chose the k as 10 then there will be (k-1)train folds and 1 test fold during an iteration. Then I read that we need to take the average value of all iterations while calculating the evaluation metrics like map(Mean average precision), recall, and precision.
I am having the following doubts.
1) if I am performing the k-fold cross validation should I stick with the same parameters that I use during the training as the number of epochs, steps per epoch, etc. Let's say I got the final model during the training at the 10th epoch so should I use this as the number of epoch counts during the k-fold validation?.
2) When it comes to the calculation of validation metrics, should I need to calculate the above mentioned metrics at the end of each iteration which is at the 10th epoch and take the average of them as the final value for each metric(In here since I have chosen the epoch count as 10, an iteration will be over after 10 epochs)?
3) So if the epoch count is 10 does the whole process take 50 epochs to complete(Explanation:1 iteration = 10 epochs ,5 iterations = 50 epochs)?
4)Then in a case of epoch count became 100 (If the final model got during the 100th epoch) , does the process take 500 epochs to complete(Explanation:1 iteration = 100 epochs,5 iterations = 500 epochs)?
So, first of all note that the purpose of cross validation is to get a better overview of the performance of the model and possibly choose the best architecture among the candidates (and its combination with grid search). That being said, here are the answers
1. You need to just specify an architecture if you don't want to do grid search, meaning the hyperparameters are the same. The algorithm will them fit a model to the (k-1) folds and then evaluates the performance on the left fold.
2. Please note that you don't need to care about epoch. Yes, of course if you say you have specified the number of epochs, then the last epoch should be used. However, if you have early stopping, then the situation may be different. What I am trying to say is that you should evaluate based on the 'trained network', which may have been trained in less than the epochs you specify. Then after evaluation on each left folds, calculate the mean.
3. As I mentioned above, you may have less than this amount, if you specify early stopping. Otherwise, yes, you would have 50 epochs.
4. This is similar to three and the explanations above.
Mostafa Kiani Shahvandi Yes of course here I am considering the optimal epoch count during the training , which is not the last epoch .
Ex: if epoch count is 200 , if the optimal epoch is 40 I should choose epoch count as 40 for k-fold validation .I am I correct ? Then I need to calculate the evaluation metrics at the end of each iteration which is the 40th epoch and should get the average of all iterations? I am I correct ? So if k =10 then I may have 400 epochs during the cross validation.Is there any way of speeding things up ? .Because this process cost 10 times as training the model.
Dear Hasindu Yahamapth Dias Dahanayake
I suggest the following links, have a look:
1. https://www.machinecurve.com/index.php/2020/02/18/how-to-use-k-fold-cross-validation-with-keras/
2. https://github.com/quanghuy0497/Mask_R-CNN
3. https://github.com/matterport/Mask_RCNN/issues/281
Kind Regards
Qamar Ul Islam
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training