Assume that I have a large number of predictors and a limited number of observations; How should I build the statistical model? should it be flexible (Spline) or inflexible (Linear Regression)?
Since you have a large number of predictors, I have to assume that you are not conducting experimental design and you are not talking about descriptive statistics. that said then, It is recommended in non experimental research to use at least 30 observations per a predictor for the stability of Beta Weights and it is better to use a statistical formula in determining the sample size
There are other ways of regression when sample size is small than the number of predictors. Please have a look at Lasso.
About flexibility, I think Lasso is more flexible than the methods you mention. If you need even more flexibility, there is Elastic Net ;) (this is not a joke).
See my answers to the question posted below. I posted papers and links on lasso that you may find helpful. The book below may also be helpful, see Chapter 19. Best wishes
https://www.researchgate.net/post/When_do_we_proceed_to_multivariate_regression
http://www.amazon.com/gp/product/0321888030/ref=s9_simh_gw_g14_i20_r?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=desktop-1&pf_rd_r=1KEBX0JVZF3KD0YZWRR0&pf_rd_t=36701&pf_rd_p=2079475242&pf_rd_i=desktop
Thank you guys for your answers,
After multiple readings, investigations and testing, I found out that the large number of predictors and small number of observations will cause the model to have some predictors x without any observed y, and if we used any inflexible methods such as linear regression, it will ignore the effective values of the limited observations. So, using the flexible techniques such as LASSO or Spline will be more fitting as it will capture the model structure without ignoring any important observations.
N.B; it is also preferred to increase the number of observations higher than 30 to achieve the minimum statistical significance according to the central limit theorem.
Ahmad
Do you have small number of predictors or observations?
You said previously you have limited number of observations, now saying the opposite. A typo?
@Mehmet Sinan, thank you for your comment, it was my mistake and I fixed it :)
If you have categorical variables you want group lasso not standard lasso. I have some references in my paper. See attached paper.
- Badges
- Science topic