1. As I learned from related papers, open-set conditions refers to new objects encountered that were not seen during training. for object classification, open-set conditions can be cat images for a classifier which is trained to classify dog and fish. This problem can also be solved as novelty detection.
2. Open-set conditions are mostly studied in classification field, merely mentioned in object detection field.
3. My question is, should the open-set conditions be always treated as a classification problem, even in the object detection field? I mean, in an input image of a detector, despite new objects the detector never saw, could the relative positions of different objects and various kinds of background cause open-set conditions for the detector? Do we need to consider these situations? If so, how?
Thank you!
In typical object detectors, there is a classification part included that classifies given region proposals to be either of the known classes, or the background class. Usually, these classifiers use softmax and were trained on the known classes and samples from the background of the training images as an additional class. For example, if 10 classes should be detected, the classifier is trained on 11 classes (one being the background class).
In a typical evaluation, the face detector is only used in a binary classification task, which means that each of the classes is evaluated separately. This includes that different operating points (different probability thresholds on recall values) have to be defined per class. For example, you would accept a detected cat when its probability is over threshold 0.5, but a dog only if its probability is above 0.9. If it is below threshold, it does not matter how the percentages are actually distributed, i.e., if the maximum probability is in a different known class or even in the background class. Some people might find this intuitive, I don't.
So, to answer your question: do we need to treat open-set detection as a classification problem? People in the object detection field would clearly say: no since detection is treated as a binary classification, even when the detector is trained on multiple classes. Coming from the biometric face recognition side, metrics such as the Detection and Identification Rate (DIR) curve are usually applied in evaluating open-set detection/recognition. Conference Paper Toward Open-Set Face Recognition
This curve clearly separates samples of unknown classes (and backgrounds) from samples of known classes, and it evaluates a multi-class classifier. I personally believe that this should be the right way to evaluate open-set classifiers. We currently have submitted a corresponding paper that introduces this curve to the object detection community to NIPS.
Thank you for your kind answer, Manuel Günther , could you share your opinion on how to define "open-set conditions" for a detector? For a classifier, an open-set condition can be a face that it never saw during training ,but what's an open-set condition that for a detector?Could two overlapped familiar faces in the distance of an image be an open-set condition? Another question is, is it possible we output objects with the "background" class?
- Badges
- Science topic
- Similar topics
- Computer Science
- Images