What you mean is variable selection (not feature selection).

There is no best way to select variables using data. If your model has a specific purpose, you should select variables according to that purpose. Depending on the purpose, this may NOT be one of the usual statistics (RMSE, R², AIC, BIC, ...).

Note that your initial model already poses infinitively many restrictions on the result, as you may consider only linear relationships and no interactions (or only linear and quadratic relationships and some interactions; or you use splines and some interactions; and you may still miss a great deal of important variables of which ju simply have no information).

In any case, make sure that your model building (variable selection) is done on data that is NOT used to evaluate the performance of the final model.

Having to do a variable selection already shows that you don't have a meaningful theory in your mind. If you have a theory, then this (and not the data) should guide you how to build the model. Note that variable selection is NOT suited to derive at models that help you construct a theory that justifies the model (that was build using the data... thats evetually a circular argument).

Hello Fazla,

There is no exact answer to your question. You need to test them to find the best suited. Depending on the size of the algorithm, you can go without removing any feature (since 75 is not a big number).

I can recommend Genetic Algorithm. I recently implemented it and performed well. See the examples on the toy datasets (Iris and Boston) from the following repository.

https://github.com/mcsorkun/Genetic-Selection-Cheminformatics

What you mean is variable selection (not feature selection).

There is no best way to select variables using data. If your model has a specific purpose, you should select variables according to that purpose. Depending on the purpose, this may NOT be one of the usual statistics (RMSE, R², AIC, BIC, ...).

Note that your initial model already poses infinitively many restrictions on the result, as you may consider only linear relationships and no interactions (or only linear and quadratic relationships and some interactions; or you use splines and some interactions; and you may still miss a great deal of important variables of which ju simply have no information).

In any case, make sure that your model building (variable selection) is done on data that is NOT used to evaluate the performance of the final model.

Having to do a variable selection already shows that you don't have a meaningful theory in your mind. If you have a theory, then this (and not the data) should guide you how to build the model. Note that variable selection is NOT suited to derive at models that help you construct a theory that justifies the model (that was build using the data... thats evetually a circular argument).

There are two things that might guide you, in addition to Jochen Wilhelm 's useful pointers.

First : what is the purpose of the model? If it is atheoretical prediction, then practical considerations are important. You need to prioritise predictors that are simple to measure and widely available. You also need to consider the time taken to use the system and the consequences of missing information – is the system operable from a minimum feature set? To this end, I'd also recommend looking at fast and frugal trees as an efficient and generalisable prediction model.

Second : what do you know about the predictors? If the aim is to understand the phenomenon, then quite possibly the predictors represent a smaller number of underlying dimensions. Procedures such as principal components analysis and clustering around latent variables may help you to understand the underlying dimensions and reduce the number of predictors considerably, while giving them a richer theoretical meaning.

A model is a tool, and building one must be linked to the purpose of the tool. Until that is clear, there are no right answers. Actually, afterwards there are generally no right answer, but you can at least tell which ones are least wrong.

Of all variable selection Algorithms I consider Adaptive lasso the best because you get the optimal prediction equation for the predictors that you start with(oracle property). Notice as Jochen Wilhelm and Ronán Michael Conroy suggest choosing poor predictors would always give you poor predictions. In any case look at the link: https://www4.stat.ncsu.edu/~boos/var.select/lasso.adaptive.html

For a start in R see:

http://sgpwe.izt.uam.mx/files/users/uami/gma/R_for_dummies.pdf

Best wshes, David Booth

PS I do not agree with @Mohammed Deeb eg stepwise doesn't even give the same answer on successive runs. See section 1 of the attached.

Thank You David Eugene Booth Mohammed Deeb Ronán Michael Conroy Jochen Wilhelm Murat Cihan Sorkun

Are the data normalized? Some methods require data normalization. PCA is a popular and recognized option. t-SNE is a recent technique that allows you to visualize your data in 2D/3D. Might help you get better insights. I would advise you to understand your data and select data exploration methods that you know how to use.

I'd personally go with PCA because you mentioned multiple linear regression. after you do on your existing data, you get a transformation matrix which you use to apply PCA and feature extraction on future data. since your data is linear, you can multiply the PCA transformation matrix with the linear model matrix, and you get a nice compact model. With PCA you have the added advantage of visualization as well, which gives valuable insight.

I know you specified multiple-linear-regression, but when training neural networks, there are certain techniques (tweaks) that mimic feature selection.

You can use Chi-Square Test for Feature Selection.It depends on the model and the dataset.

We probably need to begin by saying feature selection terminology is a data driven approach for buidling a model, compared to the traditional theory driven process where hypotheses drive model buidling.

If you want to use a linear model you can resort to linear regression with elastic net penalty. Elastic net is a weighted combination of ridge and lasso penalty. In R you can find it in the package glmnet.

See also https://www.google.com/url?sa=t&source=web&rct=j&url=https://web.stanford.edu/~hastie/TALKS/enet_talk.pdf&ved=2ahUKEwii-8vz77fqAhVliIsKHUrZAqIQFjAdegQIDRAB&usg=AOvVaw2Qq9yEG-muL9PhoMeRGU3y

is it possible to derive patient dataset from symptoms data set. Is any mathematical approach is available.

There are very useful and valuable answers here and I would also suggest something. Usually, a model should source its form and shape from the general theoretical frameworks which might be a result of your own considerations or something ready from other theories. Then, I like to experiment. As Ronán Michael Conroy mentioned, the purpose of the model, availability, and your knowledge will provide the new horizons to include new variables. Lastly, I always try to use the shortest route to connect point A with point B. So, it means the model should utilize the best optimal variable size. However, occasionally, we observe very comprehensive and unnecessary big models to demonstrate that this approach is fancy but in reality that might be very nominal. Good luck!

https://en.wikipedia.org/wiki/Occam%27s_razor

Thank You everyone for valuable answers.

I would suggest Lasso regression. It uses L1-penalty that automatically shrinks the parameters of less-relevant features to zeros. In another word, it can perform feature selection automatically. Elastic-net is also an alternative, which uses a combination of L1 and L2 penalty.

You can easily implement these models using Sklearn, and you can find more detailed model introduction on the website.

The variable selction, feature selection, or attribute selection are the same, it is just that statistical literature prefers variable selection more. You can also call it model selection since regression with x1 and regressiong eoth x1,x2 are considered different models. If you want near best variabke selection, dont use stepwise based onf R2 or F stat. Use AIC, BIC, or consistent-AIC is even better. Assuming you choice cAIC, The best variable subset is the subset providing the minimum cAIC. Now you have a optimization problem since you cant do full subset search and build models for 2^75 subset alternatives. So, your iptions is to use genetic algorithm to find a sibset minimizing cAIC

Thanks a lot dear oguz akbilgic Xiaoyang Chen Jean Maccario for the answer.

Jochen Wilhelm makes a good point: for instance you might need all the variables you have, honestly 75 does not seem a large number of predictors

Variable selection becomes relevant with thousands of regressors.

There is NEVER a best method, especially in an exploratory analysis like yours

In 2020, I think that models should be estimated using out-of-bag or test data, then replicated in order to assess the variability of the results, i.e. regressor X1 enters the model 70% of the times)

see https://pubmed.ncbi.nlm.nih.gov/19679825/

You should not perform variable selection using the same training data, i.e. fit a model on training data then assess it using test data.

Then using criteria that are external to your training data you can decide how many and which variables to retain (minimum error, maximum R2...)

caret in R does this very elegantly

The relaimpo package could be useful to assess with variables are most relevant in terms of R2 decomposition

-------------

Also variable importance in a Random Forest could be a quick way of solving your problem

The best variable (or feature/vector in machine learning lingo) procedure is the researcher's brain with knowledge of, or theories about, the variables. The other things listed above are of secondary value if you lack a brain, knowledge, or theories.

I lack a brain 🙋‍♂️

i found that random cross-validation was useful in revealing how few variables were needed before I was modelling noise and not signal. That then helped me think about these key variables, 75 does seem a lot to think about, simultaneously. Because of issues of reproducibility, observational epidemiology now often thinks about whether this particular single exposure affects the outcome (conditioning on covariates) rather than what are the complete and multiple ’causes’ of the response. So does smoking produce lung cancer rather than what are the causes of cancer.