Hello! In my last question, I asked a question about how to analyze a dataset with an ordinal dependent variable and multiple categorical independent variables. Here's the question if you'd like to check it out: https://www.researchgate.net/post/How_can_I_analyze_multiple_binary_independent_variables_against_an_ordinal_dependent_variable
My dataset is questionnaire data that has a field about skill level in a certain sport. This skill level is the target variable in the study. The questionnaire also had a question about which of four sports the respondents answers regard. The aim is to compare if the correlations between skill level and answers are similar or not in each sport. So I would like to find which variables predict skill level the best in each sport, and how important the variables are in the prediction.
It was suggested that I use ordinal logistic regression, and test for proportional odds assumption. If the proportional odds assumption is not met, I should use consecutive binary logistic regressions to construct an ordinal model myself. It was also suggested that I could use a Boosted regression tree. I would like to use these both as a cross validating method, as there seem to be uncertainties in ordinal logistic regression.
I understand that the workflow should be as follows:
The binary logistic regressions should be run as follows: Class 1 vs Class 2-4, Class 1-2 vs Class 3-4, Class 1-3 vs Class 4. I understand this and know how to do this, but I do not know if the boosted regression tree should be done in the same manner. Should I make three different boosted regression trees and calculate the importances separately, or should I only create one tree model that I train with all four target classes at once? It seems boosted regression trees don't perform well with target variables with more than two values.
I would truly appreciate your help. Also, if you know of studies that have used a similar method, I would really appreciate if you could link them.
Best regards,
Timo Ijäs
You might find the attached paper of some interest with regard to your problem.
David Eugene Booth thank you for the answer! You linked the paper also last time you answered me and I see you have finished the paper, congratulations! I've read the paper and I think I understand the process you have used. However, in the study you have a binary dependent variable. In my case, the dependent variable is ordinal: the skill level from 1-5. I understand that I must partition the data so that I can train multiple binary logistic regressions, for example logit(0(skill 1) 1(skill2-5)), logit(0(skill1-2) 1(skill 3-5)). You suggest that I use boosting, should I fit multiple binary boosted regression trees in a similar manner as with the logits, or should I only fit one with all the skill levels 1-5 as target variable? The aim of my study is not to create a predictive model, it is descriptive or exploratory. Aim is to find out about the relationships between independent variables and skill level. Thank you for your help.
I apologize for not being familiar with ordinal.vsriables in this situation. However if this is NOT a predictive analysis then these methods don't apply. I would recommend looking at the ecology reference (by Several ecologists and Trevor Hastie at Stanford. He is probably the best person to deal with your question. Best, David Booth
Hello Timo Ijas , I have a feeling that you are making your task way more complicated than needed.
If you want to see if some predictors can explain an ordinal dependent variable, then Ordered Probit, or Ordered Logit, is appropriate. These will give you coefficients, with standard errors, that can be interpreted in a similar way to OLS regression.
So, if the question is, "do people use the same criteria when deciding on their skill levels in different sports?" then the task is to compare the coefficients. Specifically, is the difference between Beta_1 in Sport_A and Beta_1 in Sport_B different from Zero? A t-test. You should make the question more general by starting with ANOVA for all sports. When you already have coefficients and standard errors, you can calculate ANOVA and t-statistics in a spreadsheet.
An even more straightforward approach would be to simply calculate ordinal correlations (e.g. Spearman's Rho) between your dependent variable and each of your predictor variables. Again, with correlation estimate and standard error of the estimate, you can decide whether those correlations are different from each other.
Hume F. Winzar Thank you for your answer! Ordinal logistic regression is exactly what I have been using, and then studying the coefficients and their differences. The problem arose when the results of ordinal logistic regression weren't significant or proportional odds assumption was not met. Additionally, some multicollinearity exists in the data so I wanted a cross-validation method, and thought of using the importance measure from a gradient boosting model.
I'm under time constraints, this study is for my master's thesis. I ended up plotting visualizations of the significant coefficients from the regressions, and just doing visual analysis on the differences between each game. If I have time, I'll consider doing anova and t-tests for the results. However I have seven questions, so I must repeat the analysis process for each one. Too bad I wasn't really aware of all these methods before starting the analysis.
I'm pleased I could help, Timo Ijas ,
I did think about multicollinearity after I posted my first response. ANOVA and t-tests will not remove that problem. Multicollinearity can increase or decrease the values of coefficients and, in extreme cases, even change the sign of a coefficient. With different data sets (sport_A, sport_B) you almost certainly will have different coefficients purely because of the multicollinearity problem.
A straightforward approach would be to systematically add one variable at a time to your Ordinal Logit model, removing a variable if it causes other variables' coefficients to change much. Of course, that means you won't be able to include all of your desired predictor variables.
Another approach is to apply a Data Reduction technique on your data - usually Principal Components Analysis (PCA). PCA takes a set of correlated variables and reduces them to a smaller number of uncorrelated components, thus removing the multicollinearity problem. This comes at the cost of not having an easy grasp of which variables contribute to the solution. Rotating the components, as in Factor Analysis, can help a little, but probably not much.
Another respondent, David Eugene Booth , knows much more than I do about the technicalities of these things. Perhaps he can help.
Hume F. Winzar, that is a great suggestion. I will definitely keep your suggestions in mind for future exploration of the dataset. For now, I had to go with the boosting approach I had implemented, otherwise I wouldn't have made it in time to graduate. I have read that boosted models work well even with multicollinearity. I decided to look at importance measures from a gradient boosting model, as well as do separate binary logistic regressions. I compared their results to the ordinal logit results to see if its "summary" is in fact good enough. I thought of doing recursive feature elimination, but apparently it is not suggested to be used with logistic regression, so I scrapped it.
I acknowledge that multicollinearity will affect the coefficients, so in the end I decided not to focus too much on how large the coefficients are, but rather on their signs.
Actually, this quantitative analysis and dataset is only one part of my master's thesis. The study also has a qualitative side that was used to quantify the results of the quantitative analysis. My survey had two open-field questions that I analyzed using inductive content analysis, alongside some expert interviews that I conducted and analyzed.
- Badges
- Science topic