I'm in the process of creating classification models using a substantial dataset (approximately an MxM matrix). To enhance the performance of these models, I'm planning to conduct some feature selection as a preliminary step. It seems like a common practice to start with variance filtering, which involves eliminating variables X with var(X) that's close to zero. Given that my dataset contains variables with varying orders of magnitude, I'm unsure whether I should normalize the data [x - mean(x)] / var(x) before or after applying this variance-based feature selection.
For context, I'm aiming to build several models using a batch approach, which includes Logistic Regression, LDA, QDA, k-NN, Naive Bayes, Decision Trees, Random Forest, XGBoost, BART, among others.
I would greatly appreciate any insights into the optimal sequence for these preprocessing steps.
What is a variance "close to zero"??? 0.1, 0.01, 0.000001??? Any value is an infinity away from zero. Orders of magnitude are not important; they will reflect in the parameters your model will fit.
Instead of just blindly shooting in the wild, you should first reflect about which kinds of relationships between your variables you might expect.
Examples here:
https://www.lerenisplezant.be/teksten/FittingKVdm-Manual.pdf
Variance-based feature selection is a simple and effective way to start the feature selection process. If a feature has a near-zero variance, it's almost constant across all samples and is likely, not informative.
Regarding the sequence of preprocessing steps:
So how accurate is the normalization/scaling/conversion technique we use?
Different Normalization techniques sometimes do not affect the data at all, but sometimes reverse the results. Therefore, I think the innocence of these techniques should be questioned. Depending on the data type and problem type, the choice of normalization techniques may also differ. Even the number of alternatives and criteria influences fair selection. These are such volatile techniques that there are some who claim that they sometimes cause “ranking reversal” in MCDM rankings. There is no one-size-fits-all normalization. Min-Max is often preferred in artificial intelligence and machine learning applications, which is wrong. I wonder why the alternatives (vector, Max, sum, logarithmic) are not used? Let's not forget that normalization homogenizes different units, but does not always guarantee a fair conversion. Sometimes what transforms turns away from the original and turns into something completely different. Sometimes things go well when the water evaporates, but sometimes the coal turns into ash.
The decision regarding whether to conduct feature selection before or after normalization is contingent upon the particular scenario and the objectives of your data preprocessing workflow.
Hi, please check this recent research in this field:
Article Abaxial leaf surface-mounted multimodal wearable sensor for ...
Check the PCA methods for normalization
- Badges
- Science topic
- Similar topics
- Control Systems Engineering
- Filtering