I am working with a large dataset that has many categorical variables with many classes in each variable. I want to encode the categorical variables without increasing the dimensionality of the dataset like in the one-hot and dummy encoding. Please suggest me some good methods in python to do so.
Thank you beforehand!
Laith
hi, you can use pandas-method get_dummies https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html
also can encode your category in such way: cat1 - 1, cat2 -2 , cat3 - 3 and so on, or other codes (100, 200, 300 ... or 11, 22, 33 ...) . it is not very right in my mind but you can try such approach, may be good results, write me back if it helped
Statistically you should have a look at the underlying systematic on how the data is generated. If the data has no order (i.e, is nominal scaled in nature), there is no way to avoid this but to encode them first with a dummy method and then use feature selection to determine the most important variables (e.g., backward or forward selection). As an alternative you can also try to employ a dimensionality reduction for categorical variables:
Conference Paper Dimensionality Reduction of Categorical Data: Comparison of ...
You can try Principal Component Analysis (PCA) with python.
https://towardsdatascience.com/dimension-reduction-techniques-with-python-f36ca7009e5c
- Badges
- Science topic