I am tackled with a industrial research issue in which a massive-scale data which is mostly a stream data is about to be processed for the purpose of outlier detection. The problem is that there are some labels for the so-wanted outliers in the data, even though they are not reliable and thus we should discard them.
My approach to resolve the problem is mainly revolving around unsupervised techniques, although my employer insists on finding a trainable supervised technique by which there will be a major need to have outlier label for each individual data point. In other words, he has got trust issues with unsupervised techniques.
Now, my concern is whether there is any official and valid approach to generate outlier labels, at least to some meaningful extent, especially for a massive-scale data? I have done some research in this regard and also have experience in outlier/anomaly detection, nevertheless, it would be an honor to learn from other scholars here.
Much appreciated
Sayyed Ahmad Naghavi Nozad
Given the unreliable nature of the outlier labels and the employer's insistence on a supervised approach, a potential solution could involve a semi-supervised learning technique. Semi-supervised learning combines aspects of both supervised and unsupervised learning by using a small amount of labeled data along with a larger amount of unlabeled data for training. This approach could help leverage the limited reliable outlier labels while also utilizing the vast amount of unlabeled data available for outlier detection.
NB: The best analytical solution would depend on the specific characteristics of the dataset, the nature of the outliers, and the practical constraints of the problem domain, it may be beneficial to experiment with various approaches, including semi-supervised learning, cluster analysis, etc., to determine the most effective solution for the given scenario.
Cheers,
Dr. Samer Sarsam
Thank you very much Dr. Samer Sarsam ,
However, the situation is like we do not have any reliable labels for the massive-scale dataset which is somehow a stream data as well! I know it is like a problem with no solutions, even though I thought that probably there could be some novel resolution to this issue.
I have heard that there are some methods in image processing tasks that generate some distorted images to utilize for training a (semi-)supervised model in the end. However, I am looking for a generic strategy that could be successfully applied to non-image data too.
Much appreciated
You are welcome, Sayyed Ahmad Naghavi Nozad .
I see. A potential direction could involve leveraging techniques from active learning or human-in-the-loop approaches. These methods allow for iterative improvement of models by selectively labeling data points that are most informative or uncertain. By strategically annotating a small subset of your data and iteratively refining your model, you may be able to achieve reliable outlier detection without relying solely on predefined labels.
I hope that helps.
Kind regards,
Dr. Samer Sarsam
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs