If two or more columns are highly correlated in a data frame, Is it always true that they will convey similar meaning and hence one of them is useless and should be dropped?
How does this change in the case of the Pearson correlation score :
1 0.95
2 0.8
3 0.5
What is the effect of a decrease in correlation on it?
Hi
While Pearson's correlation is linear, there are also other type of associations, e.g. Spearmans correlation, which assesses monotonic relationships (whether linear or not), or entropy.
Some algoritms are more sensitive to correlation than others: you do not say what you would use so I can suggest so much.
A solution would be to compute principal components then use the components, while keeping the amount of information unchanged if the model is sensitive to correlated features.
To my experience with a correlation of 0.95 there will be little to no contribution in the prediction, but you can try different models.
It really depends on what you need.
the package caret in R has a specific take on pre-processing and filtering that might help you out.
Hello Ashay,
When are two variables too redundant to warrant retaining both in a model? The answer depends on many things: theory, desire for economy (e.g., fewer variables in a model is somehow better, in an Occam's razor perspective), the cost of collecting one variable vs. the other, or the concern over excess collinearity among a set of variables (e.g., as in the case of high VIF among IVs in a regression model yielding biased estimators of regression coefficients). As well, measurement folks refer to a phenomenon called the "jangle fallacy," the idea that just because two measures have different names (here, two variables), they must automatically be representing two different attributes.
However, what if the variables interact with respect to the outcome of interest? In such instances, discarding one would prevent you from finding such a useful combinatorial functioning of the variables.
I'd suggest: (a) be clear about your research/modeling objective(s); (b) consider context of the variables, and how the model would be used; (c) evaluate whether the data are congruent with the logic, rationale, theory (or whatever motivated you in the first place); and go from there.
If that's a bit vague, I apologize, but it wasn't clear to me what you were trying to accomplish from your query.
Good luck with your work.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics