I am doing decision tree analysis as classifier. My dependent variable is continuous and has positive skewness data, i want to convert in to normal distribution and then to convert continuous data into categorical data for analysis. Please i need guidance.
I convert the skewness data into normal distribution by log method but how can i read the converted data and how can i assign the categories to it
Hello Azam,
A log transform can help to make a data set which is positively skewed appear more symmetric in form. What's not clear in your query is: (a) why it's essential to have your DV approximate a normal distribution (one can develop a decision tree model without such an assumption); and (b) why it's essential to convert a continuous (or pseudo-continuous) score to a categorical score (which necessarily involves a loss of information from the data set).
If you're asking how to convert a continuous score to some number of bins/categories, there are any number of schemes available:
1. If existing categories are recognized, use those (for example, the CDC in the USA identifies categories of BMI which reflect weight such as underweight, healthy, overweight, and obese).
2. Divide the scores so that the resultant categories have equal proportions (e.g., with 4 categories, choose 3 thresholds such that each category has as close to 25% of the cases as possible).
3. Divide the scores at points that would be commonly used in a normal distribution (for example: z-score of 0 to +0.67 captures 25%; z-score from 0 to 1 captures about 34% of cases).
4. Have domain experts identify cut points/thresholds that make sense for the variable under consideration.
Good luck with your work.
Dear David
Thank you so much for your reply David Morse .
Actually I want to do decision tree analysis as classifier in R. In R project/ R-studio, if your dependent variable is continuous then analysis is done on regression tree not classification tree. This is the reason I want to convert my continuous data into categorical data. I am doing decision tree analysis to actually find out the structure attributes that command the unit cost of structure.
Will you kindly help me regarding this?
Hello again Azam,
If all you want is two categories, you could:
1. Split at mean or median value (the latter assures equal or nearly equal numbers of cases in each set, regardless of the shape of the distribution).
2. Choose some threshold that makes sense for the variable in question (e.g., for systolic blood pressure, values over 129 represent, minimally, level 1 of hypertension).
Good luck with your work.
Hello Azam Amir. I do not understand why you want to convert your skewed variable to a more symmetrical (approximately normal) variable, and then why you want to carve that transformed variable into categories. Often when people think they need a DV that is approximately normal, they do not. And carving a continuous variable into categories introduces many potential problems. See this summary by Frank Harrell:
- https://discourse.datamethods.org/t/categorizing-continuous-variables/3402
I find myself wondering if your question is an example of the XY problem:
- https://xyproblem.info/
What is your actual research question? What are you trying to achieve? Thank you for clarifying.
Use lasso and ridge to bring the data close to normality. Once you find the significant variables and if the variables have observations very close to each other change it into buckets using a histogram bins. However, this may lead to loss of information. You can not always change continuous to categories
- Badges
- Science topic