I am trying to perform machine learning (classification) on a public health dataset for diagnosing a disease status (Positive, Negative). In my dataset, all of the features are categorical like educational level (Illiterate, Primary, Secondary, Higher), gender (Male, Female), Wealth index (Poor, Middle, Rich), Marital status (Unmarried, Married, Divorced, Separated, Widowed), etc. Many researchers before performing machine learning, they convert these features into numeric variables. Then, they standardize the features by transforming it into zero mean and unit variance. Since my background in Statistics, it seems quite illogical to me.
I have applied few tree based models (DT, RF, GBM, XGBM) and non-tree based models (SVM, KNN) using categorical variables without standardization, and got performance around 0.75 to 0.85 for accuracy, precision, sensitivity, specificity, and AUC.
Now, I have the following question:
I applied several machine learning classifiers (Decision Trees, Random Forest, GBM, XGBoost, SVM, and KNN) to a public health dataset for disease status classification. All features were categorical (e.g., education level, gender, wealth index). I did not encoded these categories as numerical values (e.g., Illiterate=1, Primary=2, etc.), and also did not standardize these categorical features.
I expect suggestions in this situation from the scientific community who are expert in this field. What should I do in this case??
Standardization or normalization of categorical features is only valid if the categories represent a meaningful, ordered quantity. For nominal variables, it’s better to use encodings that respect their non-numeric nature; forcing numeric scaling otherwise has no statistical justification and may harm model performance.
No, you should not standardize or normalize your numerically encoded categorical features. This practice is methodologically inappropriate and statistically invalid. Core Reasons: Violation of Measurement Principles: Standardization (Z-score normalization) and normalization (Min-Max scaling) are designed for continuous numerical data where magnitude and order are meaningful. Applying them to numerically encoded nominal categories (e.g., 1=USA, 2=Canada, 3=Mexico) imposes a false mathematical relationship and distance between categories. Distortion of Feature Relationships: This process creates artificial numerical distances that can mislead algorithms, particularly distance-based ones. No Benefit for Tree-Based Models: Algorithms like Decision Trees, Random Forests, and Gradient Boosting are invariant to monotonic transformations of features. They only care about split points, so standardizing provides no performance benefit. Correct Approach: For Distance-Based Algorithms (e.g., KNN, SVM, K-Means) and Linear Models: The correct method is to use One-Hot Encoding (OHE). OHE creates new binary columns for each category, representing them in a vector space without imposing an artificial order or magnitude. For Tree-Based Models: You can often use label encoding or ordinal encoding directly, as these models do not rely on feature distance.
If u have two labels for a feature say yes/no and u encode them to 0/1 , is there any need for standardization?? Doesn't make sense to me
Dinesh Reddy Sagam & Lotfali Bolboli Thank you for your invaluable suggestions. I completely agree with you and think this is the best approach. Could you kindly clarify one more question? In my study, I am using eight machine learning classifiers, tree-based (DT, RF, GBM, XGBM) and distance-based (KNN, SVM). All the features in my dataset are categorical. Is it acceptable to directly use the data to apply the tree-based methods and only One Hot Encoding method to distance-based methods??
To me, a better smarter encoding, which u may call embedding is needed when the textual data is longer,with dependencies and meaningfulness.
Sadia Tariq It follows binomial distribution for categorical features before standardization, and N(0, 1) for numerically encoded categorical features after standardization. Standardization helps to reduce the distance among the numeric features. For example, Income ($4000, $3500, $5000, ...), and Age (28, 30, 27, ...). Applying standardization on Income and Age would convert them to same scale. Could you please explain why doesn't make sense to you??
Sadia Tariq Yes, I am agree with you that numerical conversion is accepted. However, I am anxious about the conversion to N(0, 1) from binomial.
Can u tel me distance between 0,1 in this case? For one hot encoding no matter how many zeroes u have, would u need to "normalize" the distance?
Sadia Tariq I am new to machine learning and currently reading some books. My concern was clearly explained by Dinesh Reddy Sagam and Lotfali Bolboli, and which also found consistent as those methods mentioned in the books. However, I am now completely confused by what you wanted to explain to me. Could you kindly explain it again using an example? I would highly appreciate any of your suggestion in this context.
Standardizing numerically encoded categorical features generally has no statistical validity because the numeric codes themselves do not represent true quantitative relationships, e.g., “Illiterate = 1” and “Primary = 2” do not imply that Primary is twice Illiterate. Standardization assumes meaningful numeric distances, which categorical variables lack. For tree-based models (DT, RF, GBM, XGBoost), neither encoding order nor scaling matters since they split on categories, not numeric magnitude. For distance-based models (KNN, SVM with RBF), one-hot encoding is preferred to avoid introducing artificial ordinal relationships. In short: do not standardize categorical variables; instead, use proper encoding (one-hot or target encoding) when needed.
- Badges
- Science topic
- Similar topics
- Biological Science
- Bioecology
- Systems Ecology
- Landscape Ecology
- Connectivity