I would like to ask can we define feature relevance in terms of regression coefficients? Lots of works report definition of feature relevance in terms of probability and mutual information but I did not find any literature defining the relevance of features in terms of regression coefficients although many works have experimentally proved this.
The short answer is Yes. Typically, you use regression coefficient p-values to determine which features to keep in the regression model.
A more detailed answer: The p-value for each feature tests the null hypothesis that the coefficient is equal to zero (no effect). A low p-value (< 0.05 or whichever alpha level you are using) indicates that you can reject the null hypothesis. In other words, a feature that has a low p-value is likely to be a meaningful addition to your model because changes in its value are related to changes in the response variable. Conversely, a larger (insignificant) p-value suggests that changes in the feature are not associated with changes in the response, so its not a relevant feature.
To get the p-values in R, fit a regression model using glm and call it myfit (say). Then summary(myfit) will list the p-values. Full details are at:
http://data.princeton.edu/R/glms.html
Thanks Sir, But in case of large datasets (say with about 9000 features), elimination of one feature from the dataset may not effect the regression coefficients of the model. In that case how can we define that feature's relevance in terms of regression coefficients. It will be kind on your part if you suggest me in this regard.
This can be done using the step() function in R. This identifies unneeded features that can safely be removed from the model. A full tutorial and work example is provided at:
http://www.ssc.wisc.edu/sscc/pubs/RFR/RFR_RegressionGLM.html
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression