The outcome of interest is binary. There are 30 covariates, and nearly two-thirds are categorical. My primary goal is to build a predictive model. The logistic regression analysis without any interaction terms shows approximately half of the covariates are statistically significant. This model seems not reasonable to me as the misclassification error rate is about 20%. I will explore other predictive models. Before doing this, I want to know what are the best practices to incorporate interaction terms in the model.
Thank you in advance.
Thank you, Jahidur Rahman Khan bhai. Is this the best strategy? I am not sure how it works because there are several possible interactions. For example, say, I fit a model with all two-way and three-way interactions. Then what will be the strategy for selecting the best combination?
Thanks again.
If it comes to quantification of interactions, I think regression models work well.
Logistic regression is a problematic methodology--highly susceptible to model misspecification. However, there is a very good alternative. Here is the link to a RG thread in which my responses address these matters:
- https://www.researchgate.net/post/What_statistical_test_to_use_for_multiple_binary_questionnaire_responses_to_test_for_and_quantify_a_relationship
If you'd like to see one paper addressing the best methodology for identifying interaction effects in applications like that which you described, I recommend the following free-to-view article which describes a procedure which renders model misspecification impossible:
- https://odajournal.com/2013/11/11/univariate-and-multivariate-analysis-of-categorical-attributes-with-many-response-categories/
Thank you, Dr. Paul Yarnold. One thing still obscure to me that how to choose the appropriate two-way and three-way interactions in the clinical setting. The classification tree might be a good option, but this technique is highly judgmental. However, I am not worried about the model misspecification as only the prediction is my goal. Sensitivity is an issue, and I need to be careful here. Thanks for sharing the article.
Unfortunately, Mr. Hossain, your responses are the opposite of reality!
CTA models ARE NOT judgmental--they are produced by an algorithm that explicitly maximizes classification (predictive) accuracy, and judgement is NOT a component of the algorithm.
Worrying about "appropriate interactions" IS worrying about model misspecification--by definition! Algorithms which identify all statistically viable models which exist for a sample make model misspecification impossible.
Logistic regression does NOT and CANNOT explicitly maximize "the prediction" (i.e., predictive accuracy)--regardless of whether or not the model is correctly specified! In contrast, CTA specifically explicitly maximizes predictive accuracy--which is a function of sensitivity and specificity.
Thanks again, Dr. Yarnold. Maybe I was wrong, and I need to do some exploratory data analysis. However, I found from the literature that over-fitting is not a concern when fitting prediction models since we can get a better prediction using these models (https://www.ncbi.nlm.nih.gov/pubmed/9382394 https://www.ncbi.nlm.nih.gov/pubmed/18311848). I totally agree with you regarding the prediction accuracy. I think there is a bias-variance trade-off. I need to explore this. Thanks again.
I wish you good luck.
Should you decide to identify a model which obtains the maximum (weighted) accuracy possible for a sample, please consider exploring the avenue I suggested. Here is a very brief research note which I hope will clarify what I speak of:
https://odajournal.com/2014/04/06/a-statistical-guide-for-the-ethically-perplexed-chapter-4-panter-sterba-handbook-of-ethics-in-quantitative-methodology-routledge-2011-clarifying-disorientation-regarding/
Md. Belal Hossain I would first look at the treatment of interaction in both full and fractional factorial designs in this book: https://www.google.com/search?q=montgomery+douglas+c.+design+and+analysis+of+experiments&oq=Mont&aqs=chrome.1.69i57j69i59j0l6.6650j0j8&sourceid=chrome&ie=UTF-8 Since your primary goal is prediction I would build your final model by adaptive lasso regression. See: https://www4.stat.ncsu.edu/~boos/var.select/lasso.adaptive.html . Since you have categorical predictors, I would also suggest looking at the attached, esp the protocol. Best wishes, David Booth
Well, the fact that 50% of regressors have significant (!) Coefficients does not imply that your model is not misspecified or that will have good prediction. @Paul does have a point here: but more importantly if - as you say - you want to build a predictive model, why won't you try different predictive models that use explicitly or implicitly interactions, use cross-validation and compare results ?