Hi,
as deep learning is a data-driven approach, the crucial is to have quality data. There exist a lot of datasets for free, but they differ in the quality of labels.
I'm now working on an index, which can tell a researcher quality of the labels, so the researcher may decide if such a dataset is useful nor not. I do have established a pipeline on how to produce such an index in a fully autonomous way. Note, I'm focusing on object detection tasks only, i.e., labels given as bounding-boxes.
The question is: does such the index exist already? I googled a lot and find nothing. It would be nice to compare our approach with existing ones.
You can take a look in this link:
https://labelbox.com/blog/how-to-measure-quality-when-training-machine-learning-models/
I would suggest checking this paper
"Detection and Correction of Mislabeled Training Samples for Hyperspectral Image Classification"
The methodology can be applied in different domains as well.
Yongkai Liu
A agree that the uncertainty is a bonus. But when the labels are precise, you can distort them slightly to obtain the uncertainty. On the other hand, you cannot produce precise data from the imprecise one.
But the real problem is during the testing phase. It may be difficult to distinguish between "the test labels are ok, but the NN performance is poor" vs "the test labels are poor, but the NN performance is ok".
Petr Hurtik
I tried to look for the pixel-wise uncertainty of prostate zonal segmentation before. Lighter regions indicate the most uncertain area -- mostly due to the imprecise annotation -- can this help?
[1] Article Exploring Uncertainty Measures in Bayesian Deep Attentive Ne...
Yongkai Liu
Ah, this is what you mean as uncertainty, now its clear. Unfortunately, I think such uncertainty is measurable only when you have labeled data made by various persons before they are aggregated to the final label.
To determine the quality of data, you can create a model to identify labels and determine the validity of the model. You can improve the quality of your data. To do this, you need to determine the authenticity of the identification of each of the tags and discard those that are poorly identified from the data. To assess the reliability you can use the F-metric van Rijsbergen.
You can take a look in this link:
https://labelbox.com/blog/how-to-measure-quality-when-training-machine-learning-models/
- Badges
- Science topic