I'd like to develop an anomaly detection. I have historical data from sensors in the form of time series. The time series can be divided into data of a normal state and data of an abnormal state i.e. I have good data and bad data from sensors. Before I want to implement the actual anomaly detection, I want to investigate whether the good data and bad data can be separated by an explorative analysis. Thereby I want to find out if I can distinguish between good data and bad data. I hope to find out if anomalies can be detected later. Now I am looking for statistical methods or algorithmic procedures to detect and visualize this.
Currently I tried to reduce the dimension of my normal and abnormal dataset to two dimensions with the help of a principal component analysis. Furthermore, I visualized the main components in a scatter plot to detect visual differences between the normal and abnormal dataset.
Are there better and other methods to do such an investigation?
Thanks!
Basically you are interested in detecting a sensor failure. While complete failure is easy, a sensor will often experience some period of degraded performance. It is also possible that degraded performance could be due to contamination.
It is possible, but only to a point. Temperature is "easy". I would start by defining temperatures that are not allowed. Out of bounds temperatures are flagged for removal. These can be low or high temperatures. The next step would be to mark areas where temperatures change a great deal. A temperature drop of 30 degrees in two minutes would be very unusual. If available you could scrape data from a local weather station and compare your data with their data.
I like Timothy's answer. He thinks like a physical scientist yet I wonder what you are trying to achieve. I suspect you are not trying to detect faulty sensors that need replacing but rather you are trying to remove outliers from the data so you can fit the data to a model or distribution & prove the results are statistically significant. Many data measurements that involve liquid flows (i.e temperature) exhibit heavy-tailed distributions. If this is the case separating the two datasets might be equivalent to voodoo. Clear physical cases like Timothy mentioned are easy to identify but any methodology based upon the measured variance of the data will tend to put the "normal" heavy tails into the erroneous measurement dataset.
The key here is that you are the one that needs to first identify what constitutes bad data. No matter how you look at the data you are ultimately the one that defines bad. Any quantitative approach will require you to identify the value that represents bad data, and another threshold where the bad data is now good within the background of natural variability.
I will emphasize a part of Jack's answer. If you are defining abnormal data as outliers then you are abusing statistical methods. I can always gather data, discard data points that disagree with my hypothesis and have a significant outcome. This approach is unethical, but I assume that you already know this. The only data that can be deleted or changed are in cases where there was some clear mistake. A typographical error is an easy example. Departure from the Normal distribution is not a clear mistake.
- Badges
- Science topic
- Similar topics
- Methodology
- Sensors