What all techniques have you applied?

You can used clustering algorithm like

1. K-means/medoid

2. Fuzzy c-means

3. partition-based clustering etc. for the classification issue.

Hi Niloy,

Why you're concerned about implimenting

unsupervised learning approach for such task? You may, instead of that, apply supervised learning technique, where you can easily evaluate the performance of the resulting predictive model.

Let me know if you still have further issues.

HTH.

Samer Sarsam, PhD.

Hi Samer,

I am concerned with unsupervised approach because I will have a unlabelled dataset. Otherwise I could have applied supervised approach.

If infrastructure is not an issue here then you can incorporate Neural Networks which will eventually learn the representation of your data and cluster them accordingly. If the size of the data is of higher dimensional then also Neural Networks comes into your rescue.

Read more here: https://pdfs.semanticscholar.org/049e/a1e5bffdd7a494cdc6f560ef56c5a9aee2fa.pdf

If you plan to extend your work to Hadoop ecosystem then also you can apply Neural Networks with map-reduce.

Thank you Sayak. I am implementing LSTM for the same. Already tried with Kmeans, Gaussian Multivariate , LOF etc.

Share the results if possible.

Sayak ,

I think I'll not be able to share the results due to data privacy policy of my workplace.

Gaussian Multivariate is showing good result.

Better than LSTMs? Which evaluation metrices are you considering for the Unsupervised Learning method?

Niloy Chakraborty There is no single or straight forward answer for the question....If you are looking at technique applicable for unlabelled datasets, unsupervised algorithms are the better choice. But then, which algorithm suits well depends on the choice of data used for anomaly detection. This highly releies on the context for which the Anomaly detectors were built...