Finding the best features (sometimes called metrics or parameters) to input into a neural network by trial and error can be a very lengthy process. Classic feature selection methods in machine learning are 'extra tree classifiers', 'univariate feature selection', 'recursive feature elimination', and 'linear discrimination analysis' (a supervised learning version of PCA). Are there other more modern methods that have evolved recently which are more powerful than these?
Inputting too many redundant or worthless features into a neural network reduces the accuracy, as does omitting the most useful features. Restricting the neural network input those most relavant features is key to getting the highest accuracy from a neural network.
Yes, there have been advancements in feature selection methods beyond the classic approaches you mentioned. Here are a few modern methods that have gained popularity:
1. L1-Regularization (Lasso): L1-regularization adds a penalty term to the loss function of a model, forcing it to select a subset of the most relevant features while setting the coefficients of irrelevant features to zero. This encourages sparsity in feature selection and can be effective in reducing overfitting.
2. Tree-based Feature Importance: In addition to Extra Trees classifiers, other tree-based models such as Random Forests and Gradient Boosting Machines provide feature importance scores. These scores indicate the relative importance of each feature in the model's predictive performance, allowing you to rank and select the most relevant features.
3. Mutual Information: Mutual information is a measure of the dependence between two variables. It can be used to estimate the relevance of features by calculating the mutual information between each feature and the target variable. Features with high mutual information are considered more informative and can be selected for model training.
4. Stability Selection: Stability selection is a technique that combines multiple subsamples of the data with randomized feature selection to estimate the stability of feature importance rankings. By repeating the process, it identifies features that are consistently selected across different subsamples, reducing the impact of noise in the feature selection process.
5. Deep Feature Selection: Deep learning-based approaches for feature selection have emerged, leveraging neural networks to learn feature representations and select relevant features automatically. These methods often incorporate regularization techniques and feature-ranking mechanisms within the neural network architecture itself.
Peter, thank you for the link to the paper, it was a pleasure to read. As a beginner in this area i tried a number of machine learning techniques (LDA and variants of) on my own rather limited data before trying neural nets, concluding that they had similar and attrctive accuracies. Following the pack, my assuption is that ultimately neural nets would be equal or superior to logistic type regressions, but you may have to pay the price with complexity and processing time. However, due to the complexity of neural nets i'm wondering if at all the optimum features can be selected, since nothing like training then testing the net can give the result. Certainly, feature selection makes sense with logistic regression, as with these simpler machine learning algorithms it's still possible to appreciate how they work. I'm sure if you use the classic feature selection techniques that this will better the out come of a neural network, but i tend to think the features selected are probably only a subset of a set which potentially contains superior features. These superior features perhaps can only be identified by actually running a huge number of different features through the neural net and just seeing by trial and error which are best, which of course takes a very long time. This brings me back to my original question about if it is possible to identify which features from a large data set would give the highest accuracy from a neural net? So perhaps the answer to the question is to start with classic feature selection techniques and then make some variations on this when training and testing the neural net to see if it can be improved, but again this takes time. Would be interested to know how others see this problem. many thanks N.
In dimension reduction of spectral data I usually use NCA (Neighborhood Component Analysis) algorithm, but I don't know if it is the best one. Usually gives better results than PLS, PCA or AutoEncoder.
Diwakar, thank you for that good summary of the techniques. Can all 5 of these technqiues be used to identify the most relevant features so they can be input to the neural network program, or are the most relevant features identified and selected during the training of a network? In order not to waste processing time it would be useful to know specifically which features are most valuable for the neural net.
Could you recommen any good books of papers that explain these techniques in detail.
Many thanks, N.
After checking the correlations of a single independent variable and the output, the model based on two independent variables (each of them should have a high correlation with the output, but a low mutual correlation) can be created, and MSE can be checked. Then the third type of input can be added, which lowers MSE the best. Then the fourth type of input, and so on, just to the moment that reducing MSE by adding any new independent variable is not possible. It is called the step-up procedure. The step-down procedure is similar, but you start with all types of input data and search for lowering MSE by removing the inputs one by one.
Hybrid methods perform better than single methods for obvious reasons. This paper presents a 3-step feature selection method which is available in most python libraries and easy to apply. The feature selection process also provides explainability.
Article Modeling Bitcoin Prices using Signal Processing Methods, Bay...
Hubert, yes step-up and step-down sounds good, but could take quite some time as i've about 100 features to investigate. Many thanks, N
thank you, these are all great responses and provide a good range of techniques to investigate. N
You can consider using metaheuristics-based optimization or hybrid metaheuristics approach for feature selection for neural network optimization. The hyper-heuristic optimization methods are also efficient depending on your considerations.
Metaheuristics are often useful as a feature selection method. You can refer to this article:
Article Metaheuristic Algorithms in Optimal Power Flow Analysis: A Q...
At least solving descriptive models tasks, this is an optimization problem: How to find the best combination of transfer functions, number of hidden layers and quantity of nodes in each layer. It is needed to minimize the standard error and to mizimize the regression coefficient ot hte approximation function with regards to regarding yhe original data. Generally, during the parameterization of their RNA models, researchers search by trial and error and by successive approximations. The correct solution of this problem requires the combined application of decomposition techniques and metaheuristics.
LASSO method is an essential method for feature selection. However, it may be insufficient if there is multicollinearity in your independent variables. In these cases, the PCA method can be used. You can go to the link below for detailed review.
"The main difference between PCA and Lasso Regression is that Lasso Regression is a variable selection technique that deals with the original variables of the dataset. In contrast, PCA (Principle Component Analysis) deals with the eigenvectors created from the covariance matrix of the variable."
"In short – use PCA when you have variables that all contribute equally to the variance within your data or your data has high amounts of multicollinearity. Use Lasso Regression whenever variables can be eliminated, and your dataset has already been cleansed of multicollinearity."
For detailed explanation:
Ref: https://enjoymachinelearning.com/blog/lasso-regression-vs-pca/
Thank you, these are great answers!
What about area under the curve (AUC) or c-statistic (c = concordance)? Do these offer any benefit to feature selection? That curve being the plot of true positive as a function of false positive.
- Badges
- Science topic