Hi there,

I'm an epidemiologist, but I am largely a methodologist, so I publish in journals in all fields. I'd have to say that it REALLY depends on the journal. Some don't even care about model details at all. Some go crazy over them.

Just FYI, here is my approach to fitting a multiple regression and then defending it later: https://dethwench.com/which-independent-variables-belong-in-a-regression-equation-we-dont-all-agree-but-heres-what-i-do/

I do logistic more than linear, but I have found that in the journals where I have published a linear regression model, they are pretty cool with the "stepwise selection" approach described in the above link. I haven't gotten questions like you pose so far, but like I said, most of my models are logistic so I don't have a lot of experience with linear.

Good luck with your project!

-Monika

Hello Hooman,

The choice of analytic method should derive from your specific research question(s). I wasn't able to infer what that (they) might be from your query, so I can't offer much guidance for your specific goals.

What I can say is, hierarchical regression (which is what I believe you mean by "blockwise" entry of IVs) does allow you to answer a question that a simultaneous (all relevant IVs entered in one model) regression cannot: What is the additional explanatory power of a variable (or variable set) above and beyond what previously entered variables can account for (for your sample)?

The last thing I would suggest is (and you didn't mention it), do avoid the "step" methods altogether. They have many technical issues associated with them, including: (a) no assurance that you will identify the best ensemble of predictors for the data set; (b) they are very likely to overfit the data set (so generalizability is questionable); (c) minor changes like modification of the entry/deletion criteria will affect the resultant model; and (d) computation of p-values for entry/deletion is often wrong in many statistical packages.

Good luck with your work.

David Morse gives an excellent reply. For more detail, references, and what you might do in place of "step" methods you might see the attached, esp the references. Best wishes, David Booth PS hierarchical regression as used by David is largely an SPSS term. A more common use is described here: https://www.bing.com/search?q=Andrew+Gelman+Hierarchical+models+&qs=n&form=QBRE&sp=-1&pq=andrew+gelman+hierarchical+models+&sc=0-34&sk=&cvid=73CA8CB11DDC4BD9A7A8A7F058484BC6

Best wishes, David Booth

David Morse Thank you for your advice but I was wondering if there is any limitation regrading the number of independent variables if I want to simultaneously include all variables in one regression model.

If you use OLS then yes there is. If p>n then OLS won't even run. There would be a practical limit before that depending on the data. Lasso runs even if p>>n. That's one of its most attractive features in a world of Big Data. Another attractive feature of adaptive lasso is that if you use it to select variables for prediction you get the Optimal set of variables. that is discussed in the above paper and its references.

As David pointed out, there is no difference between the two approaches and the final model will in both cases be the same, so there is no intrinsic need to do a blockwise regression, unless you want to know the increment in explained variance of one particular variable, as compared to the variables already in that model. But with 7 or 11 variables (as you mentioned in another question here on RG), it is hardly to imagine for me that they have such a specific order of entering that it is of practical interest to assess the increment of each variable separately.

For example with your 7 variables, do you really want to know the increment of x4 compared to x1-x3? What if you enter x5 before x4? The the increment of x4 will different, since x5 is already in the model. Is this of interest for you? If "yes" do separate blocks for each variable.

So, to put it simple: if you have no strong arguement, why the ordering is of particular interest, make blocks of practical interest (e.g. all demographics in a first block, all well documented variables from former research in a second block, and all new variables of interest in a third block, to see what this "new" variables explain over and above the well established ones).

Besides that: even if you enter all variables together in one block, you can estimate their unique contributions to the model by assessing the (squared) semi-partial correlation. It tells you the unique explained variance of this particular variable.

P.S.: David gave you the model based upper bound of variables you are able to enter, but I think in most contexts this is not relevant. Other problems may be multicollinearity, suppression and loss of df. Sometimes it makes sense to incorporate variables, which are not correlated with your DV, but may increase R^2 (suppression), on the other hand does it not make any sense to incorporate all variables you have. I think a theoretical based approach is best.

Rainer Duesing After a reconsideration I ended up with 5 sociodemographic independent variables and 3-4 main independent variables. I did a multiple regression and tested the VIF column in SPSS and the biggest values belonged to two main variables which were 3 and I suppose I'm fine with the collinearity.

Rainer Duesing I don't understand your response. While blockwise and stepwise on the full model will sometimes give you the same answer, stepwise itself doesn't always give the same answers on repeated runs of the same data. See the Austin and Tu reference in our above paper. A second copy is linked here. Given that Austin and Tu published in 2004, link here:https://scholar.google.com/scholar?hl=en&as_sdt=0%2C36&q=Austin+Tu+Journal+of+Clinical+Epidemiology+Automatic+Selection&btnG=

available on Researchgate-click the link and follow, Further this result was also known as early as 1995 by Breiman, see: https://scholar.google.co.uk/scholar?hl=en&as_sdt=0%2C36&q=nonnegative+garotte+technometrics&btnG= I can't see any any purpose in still suggesting stepwise or any of it's variants. p-value selection just doesn't work as David Morse and I comment on above. I will attach our two papers and add the next link which provide much better choices as is discussed in all of these. BTW

this link also contains an excellent R program for adaptive lasso for multiple regression model variable selection which is what is being discussed here. BTW lasso doesn't allow multicollinearity and runs if p>n, which is of great importance in many modern fields of research, e.g. medicine. See this link and its references

https://www4.stat.ncsu.edu/~boos/var.select/lasso.adaptive.html

Best wishes, David Booth

Monika Wahi see my references above for some journals that aren't "cool with stepwise". BTW logistic models belong to the group of generalized linear models and thus, are linear in the regression coefficients.

Best wishes, David Booth

David Eugene Booth I think you misunderstood my answer. I never advocated to use a stepwise regression (i.e. a statistical and a-theoretical approach to find the set of predictors that maximizes R^2)! I talked about blockwise regression, where the investigator decides which variables will be entered in which order. So, if you decide a priori that you will analyze specific predictors (e.g. 7), then the final model with all 7 variables will be the same as the model where you enter all variables at once, regardlless of the order of entering in the blokwise regression. R^2 will be identical, the inference of all predictors will be the same and the semi-partial correlations of all 7 variables will be the same in both models. This will be of course not true, if you are using a stepwise regression to find the subset of the 7 predictors that explains the model best, e.g. by excluding non significant ones. But again, I never advocated for that approach.

Rainer Duesing Mea Culpa if I made an error but I claim as my source IBM and SPSS who use forward selection in SPSS REGRESSION. The link is : https://www.ibm.com/support/pages/hierarchical-regression-spss

If that is not the forward variant of stepwise regression then what is it? As I say,

if I misinterpreted you I apologize profusely but I don't believe I did. Please correct me if I am wrong. As always, Best wishes to you for making me think and also that we all escape viruses both computer and corona, David Booth PS here is an interesting link as well: https://www.bing.com/search?q=How+to+do+forward+regression+in+SPSS+REGRESSION&qs=n&form=QBRE&sp=-1&pq=how+to+do+forward+regression+in+spss+regression&sc=1-47&sk=&cvid=82E58EDA873B4712B68FCDA267B310A1

PPS BTW Max R^2 is not a particularly good predictive criterion. See for example:https://www.bing.com/search?q=ewout+steyerberg+clinical+prediction+models&qs=AS&pq=ewout+steyerberg&sk=AS1&sc=3-16&cvid=73C0048FE1B04EDBB982B2467CC26EDC&FORM=QBLH&sp=2

and

https://people.duke.edu/%7Ernau/rsquared.htm

finally:

the model comparison part of

https://people.duke.edu/%7Ernau/rsquared.htm

Interesting thread. My two cents. For complex models entering variables in steps so that the starting values for models are set can lead to models fitting, that if you throw everything in at once can create problems for. The solutions hits a local minima. You can run the regressions in blocks without caring what happens at each block other than convergence. So if the word "hierarchical" is used in its standard sense (mixed, multilevel) this might come up. The second "cent" is if you do look at solutions at each stage, do you interpret the p values? This is another nice aspect of the lasso, where meaningful p values exist. See https://www.crcpress.com/Statistical-Learning-with-Sparsity-The-Lasso-and-Generalizations/Hastie-Tibshirani-Wainwright/p/book/9781498712163 (and this is frree from Hastie's web site: https://web.stanford.edu/~hastie/StatLearnSparsity/ ).

Daniel Wright Thanks for your comment. I agree with you entirely. My "beef" is that if I look at what people do, they use the IBM/SPSS link I cited above which uses the forward variant of stepwise which was shown by 1985 and again in 2004 to be unstable. Perhaps the right approach but the WRONG algorithm. Best, David Booth PS as an example see:

https://www.researchgate.net/post/Hierarchical_Multiple_Regression-interpreting_what_changes_in_significance_mean

The take home message for me in all this is don't use what @David Morse calls the "step" methods.

Thank you all for your help.