The package Autoclass http://ti.arc.nasa.gov/tech/rse/synthesis-projects-applications/autoclass/ constructs distribution models for each class, for instance, perhaps a set of multivariate normal distributions corresponding to a few classes. You could have a diffuse class’ distribution overlapping a different very concentrated class’ distribution. And, if the classes are well separated statistically, you could have observations (data points) that do not fit into any of your existing classes; meaning, the likelihood of these observations is extremely low under all your existing class distributions. In that scenario, you would have a Bayes classifier with class distributions “separated by unrecognized regions”. That’s when things get fun. You may have found members of a newly discovered class; or, you may have found bad data.

The technical reports and journal articles available and/or cited at the above website provide a more detail explanation.

.

the "historical" reference on this topic is

C. Chow, On optimum recognition error and reject tradeoff, IEEE Trans. Inf. Theory, 16, 1 (1970)

(attached below)

see also

Bao-Gang Hu, What are the differences between bayesian classifiers and mutual-information classifiers ?, IEEE Trans. Neural Networks, 25, 2 (2014)

http://arxiv.org/abs/1105.0051

.

Kevin - thank you very much for you interesting advices.

Fabrice - thank you very much for articles. This is what I need.

If I understood well, you would like to classify the regions of interest in the domain where the distribution of information is not coherent rather it has distributed with reasonable no-surface regions or HOLES. You can make use of sum of Gaussians to

analyse the regions of interest. Please go through the link below, which directs you to "Recursive Bayesian Estimation Using Sum of Gaussians".

http://www.sciencedirect.com/science/article/pii/0005109871900975

Thank you Krishna.