Article Power and sample size calculations for comparison of two reg...

Article Sample size calculations for model validation in linear regr...

Reout -

I think the two papers above are for comparing two regressions and for looking at one, respectively, but I think your question is about two predictors within the same multiple regression and telling when you have enough data to say one is more important than the other with some degree of certainty. Is that correct, Reout? Power is sought as in the case of a t-test for comparing two samples. In such a case, I would rather look at a confidence interval around the difference in means, to see what confidence barely does not include zero, so you can say that with what sample size would you have an acceptable confidence that the difference in means is greater than zero.

But the problem here is that when you look at two predictors in the same regression, even standardized, or here perhaps already in the same units, they can impact upon each other: there may be an interaction term; an intercept and/or other predictors may come into play; there could be collinearity or other such considerations. All of this means you cannot look at these two "independent" variables, well, independently.

For one thing, sample size considerations rely on known or approximated or well-guessed standard deviations for which we then see what sample size will reach acceptable standard errors. In an "error term" the estimated residuals can be factored into a nonrandom and a random factor each, and a sigma approximated from the random factors of the estimated residuals. But here we are looking at predictors and their coefficients. If they change in such a way that the standard deviations are not constant with increases in sample size, how would you determine that? The t-test and confidence interval I described for two samples assume the two standard deviations are constants. But if you look at the standard errors for these two predictors, I doubt that you will find that you can determine standard deviations from them that won't change with sample size increases. I think you could try that and see, but I think that will show changing standard deviations. (I'm guessing.)

Perhaps you could graph the changes in standard errors for these two predictor coefficients against increased sample size, and extrapolate to a reasonable estimated standard error for each. That will only tell you that at a certain sample size you expect that one coefficient will have this standard error, and the other coefficient will that standard error.

There may be a better way to look at this of which I am not aware, and/or, the standard deviations may be constant enough, but I know that it is a common mistake to think that one independent variable can be looked at independently of another, such as comparing p-values to see which is more "important," and I think that is often a mistake. An often promoted mistake, but still a mistake.

If I want to know what sample size gives me a certain estimated variance of the prediction error, there is such a variance 'formula' for each multiple linear regression model, but the predictors are then taken altogether. There may be some work somewhere on looking at pairs of predictors and considering all the complicating factors for comparing them, but I think the graphical approach I mentioned, or some more sophisticated simulation, might be a better solution to address your question.

Just some thoughts.

Cheers - Jim

Wow Jim! This is so helpful :) I must say that I do not fully understand power analysis- even though I usually provides one. Maybe what I can do is to address it in the context of multiple regression and see what sample size is needed to yield a significant P for R2 change when I add the second predictor. Does it make sense?