I am doing a research on feature selection for extra-high dimensional data (big data). Does any one have good advice about finding the redundant features? Please don't mention PCA, decision trees or information gain :(
Dear Noura, the other way is to use the big data new techniques such as Hadoop geoportal database, please check attached links. Regards
http://www.tutorialspoint.com/hadoop/hadoop_big_data_overview.htm
http://www.sas.com/en_my/insights/big-data/hadoop.html
http://bigdatauniversity.com/courses/hadoop-course/
Hi Noura,
You could go old school and compute an intercorrelation matrix showing the pairwise correlation coefficient of two features at a time. And remove them if they have: (1) very low variance such as less than 0.01 or (2) if their pairwise correlation coefficient is greater than, for example, 0.99.
Please see Figure 4 from the full text of the link provided below.
Hope this helps.
Best regards,
Chanin
Article Unraveling the origin of splice switching activity of hemogl...
see also this: https://www.researchgate.net/publication/282576688_Infinite_Feature_Selection
Hi,
This work http://content.iospress.com/articles/intelligent-data-analysis/ida795
could be useful for you.
Hi,
We have made some contribution to this field. You can find some code and more info in the link below:
http://sci2s.ugr.es/BigData#Preprocessing
Here, my github repo:
https://github.com/sramirez/spark-infotheoretic-feature-selection
This framework is basically built on mutual information, but it's shown that it's actually useful in real scenarios.
See ECFS -> Feature Selection via Eigenvector Centrality the code is available for Matlab https://it.mathworks.com/matlabcentral/fileexchange/56937-feature-selection-library?requestedDomain=www.mathworks.com
Spatially fair feature sampling:
https://arxiv.org/pdf/1705.03566.pdf
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data