In all classifiers, the number of data points needed to adequately represent a dataset with a high number of features grows exponentially. This is known as "curse of dimensionality". Under such circumstances it is quite possible that within high dimensional data, classes or clusters appear in separate sub-spaces. What practical measure would you suggest to handle the situation?
I found the 2013 Data Science field guide published by Booz Allen helpful (link #1 below). Page 86/87 addressed concerns about the "curse of dimensionality" and practical approaches including a use case. There is also a section on Feature Engineering & Feature Selection which may be more suited for your purposes. The 2nd link is to a detailed paper on the topic. Hope this is helpful!
(1) http://www.boozallen.com/insights/insight-detail/data-science-field-guide
(2) http://pdf.aminer.org/000/339/495/clustering_massive_high_dimensional_data_with_dynamic_feature_maps.pdf
As a first jab at such a problem I would tend to incline myself to a density measure. As the density becomes lower with n dimensions the description that you will get will most likely be weak in terms of adequacy ( this may not be necessarily true but at least would raise a flag).
I am also interested in additional comments and other alternatives since this is a very interesting question. Thanks for asking it.
Usually, not all the data in the feature space are important for classification so we may opt for:
1- feature selection (filtering): normally information gain using entropy, or searching for the best (near optimal) features, using some heuristic searching algorithm such as genetic algorithms.
2- feature extraction: transforming the feature space to a reduced dimension, this approach depends mainly on the fact of the redundant data. many approaches and method can be found for feature extraction, those include but not limited to:
principal component analysis (PCA), linear discriminate analysis, Fourier transform, wavelet transform, discrete cosine transform and many others.
recently I published a paper about a new distance metric that is invarient to the scale of the data in different dimensions link:
https://www.researchgate.net/publication/264995324_Dimensionality_Invariant_Similarity_Measure
Article Dimensionality Invariant Similarity Measure
Curse of dimensionality
How does Deep Learning overcome this hurdle in machine learning and why?
In high dimensionality applications deep learning does not suffer from the same consequences as other machine learning algorithms such as Linear Regression. This fact is part of the magic that makes this methodology of modeling with Neural Networks so effective. Neural Network’s imperviousness to the curse of dimensionality is a helpful characteristic in today’s world of big data.
https://hackernoon.com/what-killed-the-curse-of-dimensionality-8dbfad265bbe
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining