I have performed feature selection on my dataset using Binary PSO. I wanted to know if I run my algo 30 times and get different feature subsets everytime, can I take the moST repeated features in the feature subset in the 30 runs as the final subset by applying a rank on tue maximum no of selected features?
e.g if feature F1 appears 10 times, F2 appears 8 times and F3 appears 7 times and rest of the features randomly appear 1 time im those 30 runs. Can i say features F1,F2,F3 are my most important features and my final subset for feeding into an SVM should be F1,F2,F3?
I think you can train the SVM with the obtained subset S1 = (F1, F2, F3,...) then train your classifier with this subset and get the TP, TN, FP, and FN and then compute your measures. Then repeat this procedure for N times and get your average results.
Yes, you can do some statistical calculation in order to determine which are the most un-relevent subset of features (or removed features). However, this process is not important when you are trying to investigate if the algorithm can be applied for general feature selection. You may do your process only when you are dealing with a specific problem or dataset, such as a medical dataset.
Regards ..
You can randomly divide the data into train set and development set, train the SVM on the train set using different subsets of features, then pick the subset that minimizes some performance measure on the dev set. The performance measure can be accuracy, F1 score, or cross-entropy loss.
Hi Ali Mousazadeh , Sinan Salih
My problem is following :
1. I am performing Binary PSO based feature selection. After running the PSO algorithms 30 times on the train data for 1000 iterations each, I am getting a feature subset 30 times. e.g.:
1st run - S = {f1,f2,f5,f6}
2nd run - S= {f1,f4,f5,f7}
3rd run - S= {f1, f5,f8,f9,f10}
... ...
30th run - S= {f1,f5,f6,f8}
My question is following -
If f1,f5 and say f6 are the most repeated features in those subsets should I consider my final set as S = {f1,f5,f6}
OR
should I run SVR every time and look for minimum MSE/RMSE? I am using SVR not SVM classifier.
I got your question from the first time,
My answer was clear..
" If f1,f5 and say f6 are the most repeated features in those subsets should I consider my final set as S = {f1,f5,f6} "
You can say that f1, f5 and f6 are the most irrelevant features, but you can not say these are the final results. Because you can not be sure that these features are not good alone, maybe when the algorithm removes f6, it has an effect on f4 so f4 should be removed as well. This is the problem of feature selection ( What are the most important minimum number of features which help the classifier/regressor to reach the maximum classification accuracy or minimum MSE/RMSE ? ).
This is the main differnece between the wrapper and filter methods. Wrapper methods should find a subset of features when there is a statistical relation between them.
" should I run SVR every time and look for minimum MSE/RMSE? I am using SVR not SVM classifier. "
SVR works inside each particle as a fitness function, so the final result for each run time has been tested already using SVR.
--------------
Usually we do like this. Take the subset of features from the best results ( best of 30 run time). Then, test these features using different regression models, like ANN, SVR ( another SVR with different kernel for example) .. DT ... etc.
By using this process, you are trying to proof that BPSO has found the most important features which enhance the other models as well, not only SVR. Do not forget to compare these results with the same models based on the original number of features.
Hope this helps,
Regards ..
Hi Sinan Salih - You summarized everything very crisply and clarified my doubt. Thanks a ton and best wishes for your research projects !
Regards,
Bhaskar
Hi Sinan Salih - Just one more doubt :
Will the below approach be correct :
1. Select features based on the most appropriate cost function(rastringin, rosenbrock, ackley etc) which converges the best.
2. Get feature subset, use feature subsets for forecasting using SVR or any other model for the regression problem and get MSE/RMSE
2. Repeat this process for N times and get Median MSE/RMSE
3. Compare this with MSE with the MSE of SVR applied on all features selected.
Absolutely no,
The first point is wrong, what is the relation between feature selection and rastrigin, ackley .... etc ) ?.
Feature selection is an optimization problem just like rastrigin, ackley, sphere, ,,, . Each one has its own variables, constrains (or without), max/min .. so on.
The objective function for the feature selection problem is one of the following :
1- Maximization only : Find the best particle which has the maximum classification accuracy only.
2- Minimization Only : Find the best particle which has the minimum error rate.
3- Multi-Objectives : Find the best particle with Max Accuracy or Min Error rate, with the minimum no. of features. This type can be by using a multi-objective particle swarm algorithm ( i.e., Pareto Sets ), or by using an aggregative model (Depends on weights ) such as the following equation ( there are more equations):
ObjFun = Min ( a * ( 1 - Acc ) + ( (1 - a) * SelectedFeatures / AllFeatures)
where a (alpha) is a real value in range [0,1]. Alpha represents the weights in the equation above. Lets say a = 0.8. It means that the error rate ( which is 1-Acc) has the weight of 80%, while the selected features has only 20%.
In this answer I tried to give you the most important ideas and keywords to start your work, however, my answer is not enough, you have to read a lot of papers.