I have a text classification task. I used sliding windows method to pupolate my data set. The problem is that the size of the data set is huge and the data points are very similar in my data set. I would like to reduce the data set without losing informative data points. I am aware of variable selection techniques such as "kruskal.test", "limma", "rfe", "rf", "lasso", .... But how can I choose a suitable method for my problem without doing computationaly intensive operations.
I think Subtractive Clustering can be other choice for your work. It usually uses to reduce the number of data in such samples you presented. I had experience using this method for some works and I got good results, Just try it.
Your problem is not very easy to solve, and definitely it does not exist an answer which is for sure correct.
The first thing that come to my mind is that you can probably cluster the data points with a standard clustering algorithm using (number of clusters) = (number of data points you would like to keep). Then you can keep only one point per cluster (probably the centroid of the cluster) and this set will be your new dataset.
Does this make sense?
Cheers
Marco
One option is to use Self Organizing Maps (SOM). SOM is a Neural Network that will help you cluster your data.
M A K
I have similar problem: we are using image features for algorithm selection. The data - features - are very numerous and clumped together. I used various clustering/dimension reducing techniques but unfortunately I was not able to find a general method that is suitable for it all. But you should try to have a look for selections using minimum redundancy and maximum relevance for instance. PCA - also multi-linear PCA can help you to reduce the data size by selecting how much of the information you want to loose. BTW no matter what kind of technique you will apply you will in general loose some data - even by clustering.
I agree with Mr Lukac.
One of the most important stages after feature extraction, is feature reduction.
in this stage you should rank your features to keep important features. This means max dependency and min redundancy between input and target . PCA in Matlab has a criteria that shows after PCA transform how many features you need to keep your precision high.
Also max dependency and min redundancy is usable as fitness function for genetic algorithm that is very good feature selector.
If you want to use PCA, help of Matlab is enough to start. if you want to use GA and mutual information, you can use these papers:
Hanchuan Peng; Fuhui Long; Ding, C.; , "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy," Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.27, no.8, pp.1226-1238, Aug. 2005
Ludwig, O.; Nunes, U.; , "Novel Maximum-Margin Training Algorithms for Supervised Neural Networks," Neural Networks, IEEE Transactions on , vol.21, no.6, pp.972-984, June 2010
Ithink you are choose Newral Network Method and Cluster your data, for your problem without doing computationaly intensive operations.
Well, I recommend trying the PCA function in Matlab. In fact there is no unique variable selection technique that's best for all applications and problems.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence