We are working on an NDT method to find flaws around fasteners in an aircraft. By experience the vast majority of fasteners are unflawed. The measurement returns approximately 5 descriptors depending on how we set things up. At the moment we are using a minimum covariance estimator to produce a covariance matrix on the fly. We then use that to calculate a Mahalanobis distance for each point from the unflawed centroid and make decisions about which is a flaw and which isn't based on a cut-off distance. It works pretty well, but has some challenges. The problem is basically one of finding a few outliers in a large (medium?) sample from a multidimensional normal (by assumption) distribution. Does anyone have any suggestion for an approach that might work better?
I have a rather unorthodox suggestion, but one that I think is useful and will work well.
If I were faced with this problem, I would definitely go with an artificial neural networks (AAN) approach. I can go on and describe it (what and how to do), but if you were interested only in an elegant statistical solution i) I don't have it, and ii) I would have wasted my time explaining a solution you won't pursue.
So, if you would like to know about the ANN solution I see to this problem let me know and I'll do my best to explain it to you.
Cheers,
SCM
The problem with distance measures in high-dimensional space (the so-called "curse of dimensionality") is uniform data sparsity. Basically, because points are far more spread out in general, they tend towards equidistance. There are numerous algorithms and approaches that address this, of course, but a general method which retains the distance measure approach is to use subspaces (distance-based subspace outlier detection). For some methods, see e.g.,
http://www.dbs.informatik.uni-muenchen.de/~zimek/publications/PAKDD2009/pakdd09-SOD.pdf
http://neuro.bstu.by/ai/To-dom/My_research/Papers-0/For-research/D-mining/Anomaly-D/KDD-cup-99/p157.pdf
&
http://www.ipd.uni-karlsruhe.de/~muellere/publications/ICDE2012.pdf
For one related alternative that addresses the subspace approach, see:
http://www.dbs.informatik.uni-muenchen.de/~zimek/publications/SSDBM2010/SNN-SSDBM2010-preprint.pdf
Mahalanobis distance (d) is a good tool to determine the multivariate outliers for your case. To find quality cut for a given probability of flaw you can use corresponding chi-squared cumulative distribution function for d2.
- Badges
- Science topic