I have real life data (time series), there might occur distortion occasionally which is to be detected. The detection itself is simple, i have some difference from a predicted values, of which outliers clearly identify it. Of course there is some noise but three or six sigma thresholding does the job, however when there are no distortions present, the noise levels are so low that I catch false positives, cause they are above the threshold.
So the problem is in robusting the method, in a way. Any ideas?
Przemysław Skurowski I would have seen a few red flags here that the extended use of "outlier" is much wider than in extant literature.
Have you tried to apply Anscombe or Freeman-Tukey transformation, to the data before looking for outliers?
Good luck.
It would be great to have a minimum tolerance that is not calculated from the data but oriented on real-life relevance.
Thanks for replies ppl.
Pradeep Paraman - yes, 'outlier' here is a bit informal. I came to the topic from signal processing , not statistics. I looked for the recommended those transformations, but they are not relevant for the case, anyway thanks for suggestion - I have learnt something new.
Michael Short - unfortunately these are real life recordings from motion capture, so uncertainty is pretty difficult to determine (I even have a paper on the topic ;) ), however we presumed some tolerance at which the distortions are negligibly small (as Jochen Wilhelm suggested).
Fortunately, yesterday had a moment and now I have an idea how to transform signal with second derivatives - it exhibits changes, the artifacts can be small but their dynamics is much higher.
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude