I am currently preparing a paper for a conference, describing a deep learning model for Human Activity Recognition.
The model is tested on several known datasets.
Is a resulting confusion matrix supposed to formed on a single random training/test split, since this information is lacking on most of the papers I have read?
What I am currently doing is a stratified 10-fold cross-validation, for each training/test split reinitializing the model, training it on the training set and calculating a confusion matrix on the test-set and then adding all 10 confusion matrices together. The idea here being that this would result in a general accuracy over all test sets.
I am aware that cross-validation is usually used for finding optimal hyperparameters, but lets say I have found them in this scenario and I'm not changing them.
I would also like to include a plot of the training accuracy/loss for each batch/epoch, which would show the training process, but this would take too much space in the paper because I would have 10 plots.
A common approach is also to work with 3 sets - training, validation and test. You evaluate for the best model using cross-validation on the training and validation pairs and then do your final analysis on the test set.
If you have tested the model on well-known datasets then you could look into the previous papers using the same data for similar results and see their experimentation. For example, you could see the batch normalization and group normalization papers and see how they are similarly evaluated.
Otherwise what Helge Hecht has said works quite fine!
Hi Plamen,
A confusion matrix is a matrix, where rows are the true class and columns are the predicted class.
As suggested by Helge Hecht , during cross validations, after training the model, you can predict the validation data and compare it with the original classes which are already known to you.
For example, in R, it can be easily calculated as
> table(originalClassVector, predictedClassVector)
where the vectors are sample matched and are of equal length. The original vector contains the true classes, and predicted one contains the predicted labels.
I hope this helps.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training