Multiple Regression
Right now I'm using a training partition to do [partial] correlation analysis.
Then once done I split the data up into folds and do backwards step cross validation and using mape to prune factors.
My question is specific to correlation analysis.
If I do correlation analysis on the training partition (to do factor reduction), and then use that same partition to do cross validation on. Am I not implementing a type of overfitting?
Should I use a separate partition (for correlation analysis) than the one I do cross validation on?
I can't seem to find any examples in my texts (applied regression modelling as well as Data Mining for Business Analytics). All the examples I see online do cross validation on the training partition, but they do correlation analysis on the entire dataset (including test partition).
I'm sorry but I don't see a research question anywhere and your approach using stepwise is well-known to be unstable. I recommend this book:
https://www.bing.com/search?q=ewout+steyerberg+clinical+prediction+models&qs=RI&pq=steyerberg+clinica&sk=AS1&sc=2-18&cvid=B9249005A21947E0B490DE9F770115D1&FORM=QBRE&sp=2
Best, David Booth
Dear Joshua Laferriere ,
I'm not familiar with the specific partitioning your are intending to use. We recommend using the backwards LR method to stabilise a model then a number of validity checks including several 80-20 data splits. This is based on the advice of Andy Field and Tabachnick and Fidell.
Peter Samuels and @Joshua Laferriere, see: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C36&q=Austin+Tu+Automatic+Variable+selection+Logistic+Reression+Infarct&btnG= , followlink to researchGate, For a classic paper that shows P-value or stepwise methods of variable selection don't work. Further for methods that do work, see the attached, Best, D. Booth
I got the partitioning methods from these two books
Wiley Publishing
ISBN: 9781118879368
Data Mining for Business Analytics (Chapter 2, "Creation and Use of Data Partitions")
Wiley Publishing
Applied Regression Modeling
ISBN: 978-1-118-09728-1
p 118 describes stepwise
Somewhere between those two books I got the idea to do backwards step (Applied Regression Modelling) but I decided to try Cross Validation method (similar to the link I paste below).
Peter Samuels
The partitioning I decided to go with was a training/test split with cross validation done on the training. Test used as holdout to confirm results using RMSE and MAPE.
David Eugene Booth
I'm not sure what the requirements are for a "research question" (unless your just pinging me for formalities), this is my first question. This isn't actually being applied to any research. I'm not looking for new or better methods. You are not the first person to tell me backwards stepwise is out of date. Which is why I was trying to make it robust using cross validation (my own project/research).
Thank you for the citations. I will be looking into those.
However, what I'm trying to hammer out is the proper procedure for--according to these texts which were taught at my university--is on what partition to apply correlation analysis to. I know stepwise isn't that stable (and is why I'm doing cross validated stepwise).
I think found the answer though. To avoid overfitting, in my algorithm (again my own research which converges on similar Adj R^2 to leaps best subset see http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/155-best-subsets-regression-essentials-in-r/) I use a cross validation method of partial correlations of Y to each predictor term (on the training partition), deriving average correlation over the folds, and then prune insignificant terms in a backwards fashion. I do something similar for VIF
https://github.com/thistleknot/matrixMultiplicationRegression/blob/cb47ed1aa5d79adbe4f386635526c291ca96cbfb/RMSE%20BackStep%20CV.R
I'm also considering elastic net and xgboost as methods. I wrote an elastic net method that uses cross validation as well, but I'm less confident about how it works than I am of my regression + pca method
https://github.com/thistleknot/matrixMultiplicationRegression/blob/master/Elastic%20Net.R
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training