In use, the weights obtained are considered with the greatest accuracy when the data set is divided, because the weights are initialized at random and the training data is swapped at random considering the small batch size.
In the literature,
1) Some articles only report the maximum accuracy achieved with the test set.
2) Other papers report the average of the maximum accuracy over the test set over a number of trains.
3) We believe that these approaches are not fair because we do not have an insight into the overall behavior of the dataset.
Question: What is the best way to express accuracy?
Typical model training and testing is performed at least 3 times and thus the average test measurements are recorded (along with the standard deviation to show the level of variation).
Therefore you can see "discrete" values, regardless of the average of x trains/test runs, in the records.
Typically, you will be actually significantly overfit if you have a 100 percent train precision. In principle, if you have more data, however you would be in a position to identify it correctly 97 percent, most definitely very fine, based on the application (there are exceptions such as where the information is very unbalanced, 97 percent precision will not be adequate to inform).
You can mess the cross validation in several ways, but most refer to time-ordered data that is not possibly an image-recognition problem.
You still have opportunities to mess with it. For instance, take your dataset and replicate each picture and then split the train randomly. If you had to overfit deliberately, so that your neural net was actually "remembering" you would then be very heavily checked (because there are many instances in your research range as well) but your model would not generalize to new datasets obviously, it might not be bad, because you would not know.
So this is a positive thing, like I said, unless you "screwed up your cross-validation" It doesn't seem like you did, but I have so many different ways to list them all. After four years as a data scientist, I also often see other users.
Finally, simply because you get a "good" 97 percent ranking, it doesn't mean you should do the highest. Perhaps this is a very basic data collection with a very specific signal. It indicates that you are "over fit a little bit" that there is a gaps between your test and your train failure. You may aim to exercise for less times to use regularization or removal to minimize excess fitness, potentially compromise training loss and improve test loss.
Hi Bernard,
From my little experience, I think it would be more appropriate to work with the overall average accuracy obtained when the trained model is run on the test set. One reason for this, is because the performance (accuracy) of the model in the presence of unseen data (test set) would mostly likely give you an idea of how your model would perform in practical applications. I personally do not recommend working with the accuracy obtained with training set because there are high chances that such accuracies won’t reflect the true performance of the model in practical settings.
I hope you would find this helpful.
Good luck.
Related to your query, I suggest you follow : https://papers.nips.cc/paper/1995/file/1896a3bf730516dd643ba67b4c447d36-Paper.pdf
https://watermark.silverchair.com/160412.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAAr0wggK5BgkqhkiG9w0BBwagggKqMIICpgIBADCCAp8GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMyyqQRzWwBxB11tX8AgEQgIICcJppE8TVOQqDUugGRDl2zxkMHiZvdBAlyNOjK2fNctPcmb0AJtGVB7oVGu9SEBGjO9rYGwpP_Q9EwBpwtRjHyBJFtQtseV0u9QN0rAxF1alT5kjrBaXAD16VO-ZtVqiMokHvpiC_TduV8Kk-52c2VLI29ZW7FCRSaBw-0c0zHhhTMOF1UlvYOjAwhNzdRhfqCcDxu-ElCRmWmyqyZRmkit3okjlo5w4Vs9ln2Lt2ozeMlokvDfp9LqsflunIic3ui8-6ugN6OMPPH2KaVBnbkucAI1-d38ExwBMXzTc1YyO0flN7ysyaizEPtqirzxVMkKkbEERIzi4vsNUu-dl2FLRyKfbf03h_L-XH9zxekT__g8KFxXBHo_wm1c6Hmov1rf8Hoyu5lEkRYZfSGeDcfkl3aybOWu7Gqic2e5cj8j4r7UoOW_EnYV6gfyUzVM3LvN9pxDIV2BvKy-yYz6awtoDOND5wW99msdfYs_w5845YboTLO0rGbFvA9mxC3F6oaujEGzmQiDgNxTaa48bgi7pAV5-vyOj0bgsOq_ZLLHlx7dnyYectMfOQO8h6Ojpl4RKw0uWOBOPQ5vgX6xC_27F6L-4l_AgSvNnhrZsJ2UDqfh3uADPSWVuRasyfNWGCnKZh0C0l6KiKWago36OVQdWs-e-cMZVNyknOjFl5t559REGwM-DJiEXYaWLX4njVG-uQWFBKj-7DVR7m1Y1i6R39afjZB2P6OObyNNuBpMnDfO5ZDraCJGWiBkjka466zayV1eg5CYZojtXnirwYiYKTHO2VYyTKYgW_9posmOygNF4Aiez93pOU5xX2EzwOKQ
The accuracy calculated on the training set should only be useful for comparison with the calculated test accuracy. That is, in the training set you calculate the accuracy to have a basis for comparison, in order to know if the accuracy obtained in the test set is satisfactory or not. Generally, the accuracy in the training set is always greater than in the test set.
So, for practical purposes what matters is accuracy in the test set and the best way to measure it is to use cross-validation, as this way you have more guarantees that the results are not just associated with a particular choice of the test set.
Researchers generally consider average accuracy. But ideally, you should run some statistical hypothesis test in order to check the average, because it could be that a high average is due only to some higher results.
- Badges
- Science topic