Negin Zarbakhsh، in general the work with many features suffers from the same issues of parameter estimation such as parameter dependency or dependency. There several ways to address such as adding constraints or building your model gradually by using few features and then increase the number of features gradually.

Negin Zarbakhsh، in general the work with many features suffers from the same issues of parameter estimation such as parameter dependency or dependency. There are several ways to address such as adding constraints or building your model gradually by using few features and then increase the number of features gradually.

Gamal Seedahmed thank you so much for your answer. Some of my features are the mitigation policies implemented during the pandemic to control the spread of COVID-19 disease. My issue is that the country that I am doing the analysis for implemented many mitigation policies at the same time to control the spread of disease, so these features are positively correlated to each other (almost 80%). When the coefficients become 0 as a result of the Random forest regression model, I could remove these features from my regression model to increase the speed of my model without affecting the result. At the same time, I am curious to realize which policy was affected my target variable most, and I am not sure how I should deal with multicollinearity in my case.

Alessandro Ferrarini Thank you for your response. Based on my Pearson Correlations, some of my features appear to be highly correlated with each other. I was thinking of combining features, aside from removing features. It is possible to combine these features since they have the same scale. I want to consider X1*weight (1) + x2* Weight (2) + ..... X(n)* Weight (n), while I am not sure which weights are appropriated for my model. How can I choose the best weights in the combination to increase my target variable?

Alessandro Ferrarini

Thank you so much for your answers.

I found the feature selection method used in the first article very interesting! It is well described and very practical. The problem I am trying to predict is a Regression problem, and I am not sure if I could use information gain, Gini index, relief, ANOVA and Chi-square.

I tested Random Forest Regressor and Gradient Boosting Regressor without using any feature selection techniques. My models showed a very good performance with both methods, while when I analyse feature importance to see how features have been handled by the resulting models, I can see from 29 features, coefficient of 8 features become 0.

Some of my features include Government restriction measures during the pandemic so most of these features are highly correlated (Pearson correlation about 80%) as the country I am trying to do the prediction for have experienced strict restrictions during the pandemic.

From 9 Government restriction measures, my model only kept one of those measures (Coefficient >0). With this explanation, do you still believe that combining these types of features may end in a pseudo-predictor?

Hi,

For nonlinear data, Random Forests are recommended.

For linear case, LASSO is a good choice.

Good luck,

Anis.

Moreover, variable selection can be effeciently performed by Random Forests.

You can give a try to the Robust Linear Regression classifiers like Theil Sen Regression, Huber Regression.

Mam Negin Zarbakhsh :

I agree with Sir Anis Ben Ishak , Random Forests are recommended non linear while LASSO is a good option for linear case.

Best regards;

Dear Nergin,

one of the first things that I would do is to first analyse the data prior to modelling: is the collinearity of the features due to some causal process that you can hypothesise on? If so, is there a set of root variables that you can identify? Factor analysis in this case might be a good thing to look into.

Another thing to try before modelling might be clustering and selecting a subset of representatives from the clusters and input those into an algorithm for selection, separating the features into groups of collinear features and taking one of each according to some criteria. Another thing might be to use PCA on the clustered features to combine and reduce the number of features.

The choice of algorithm depends on how to model your data, If you have a linear relationship the usual LS penalised approaches could be fine, but if the Random Forest performs better on a similar number of features and without model overfitting (because that is a common thing with thsi ensemble approach), the relationship might not be linear so it gets a better score using the trees. For interpreting the results, the TreeInterprteer in Python or LIME could perhaps help you give a better understanding of what the random forest is doing.

I hope this is helpful and lots of luck!

Asier

Anis Ben Ishak Thank you so much for your answer.

Rubaiath E Ulfath Thanks for your answer and for introducing these 2 models

Alessandro Ferrarini Thank you so much for your time and your answer. Looking for the linear combination of features, PCA works well in my case.

Dear colleague,

I've been doing Genetic programming lately and I'll try to give you a more general answer. It is a method that is based on evolutionary algorithms, specifically in genetic programming, a genetic algorithm is used. You define the input and output variables, however, when an initial population of individuals is formed it goes through recombination and mutation processes through a number of generations. In doing so, certain fitness criteria are used for the selection of individuals to whom recombination is applied. The goal is to find the best individuals in each subsequent generation according to a defined criterion. Those variables that do not contribute to the accuracy of the model through this evolutionary process fall out.

Those variables that dropped signs do not contribute to the accuracy of the model and the algorithm eliminated them.

With respect,

Miljan Kovacevic

Multicollinearity occurs when some of the independent variables in a model are mutually correlated. It creates a problem when your goal is to interpret the individual effect of your factors, it makes the estimated coefficient unstable. It does not necessarily affect the precision of your orediction and goodness of fit of your statistics.

The best way to reduce or eliminate multicollinearity is to remove (leave at least one of the correlated factors) those independent variables from your model or linearly combine the independent variables by just adding them up. Those who advised that you try LASSO or Ridge regression are right. Note, you should be mostly concerned if your analysis goal is to interpret the individual effect of factors in your model, otherwise, you will be fine with multicollinearity in your model if you want to just predict.

Here is a good reference for you: https://statisticsbyjim.com/regression/multicollinearity-in-regression-analysis/

hello

I thing the simple model of Logistic Regression can suit.

thank you so much

Miljan Kovacevic Thanks for this example