Dear colleagues,
I want to raise this question to the community. I know that in the ANOVA test where we compare means, for example, in different groups we will have problems with multiple comparisons since we only know the F-test results but group-to-group difference is unknown. Therefore, we would choose multiple comparison correction methods, such as Tukey's, Scheffe's, or Bonferroni, to adjust the p-value and explore each pair difference and significance.
However, while I am conducting multiple linear regression analysis, by implying stepwise (backward selection) method, that means I have an DV (QoL scores in eight different domains and two summary components; scores is continuous; the Rand-36 or SF-36), and a group of potential and associated IVs (factors; categorical continuous type) .
For the models that I get from the auto-selection process (stepwise and backward), I would like to ask will there be problems about multiple comparison? why? and what would be the recommended solutions to the kind of multiple comparisons problems in this multiple linear regression model building? Thank you!
There will be problems because you used an auto-selection process.
You can fit a given model to obeserved data, but you cannot use observed data to specify the model that should be fitted to. Model specification should be driven by theory. Surely, observed data may give you ideas about an improved model specification ("exploratory data analysis"), and such an improved model can in future be fitted to new observed data.
Automated model selection is a kind of extreme exploratory analysis, you actually lose completely the possibility to interpret any statistical significance of the data. This is in principle nothing bad. Just don't interpret significance and everything is "fine". However, the automated process is clearly not recommended because it ineviably will just find the best fit to all peculiarities of our observed set of data, disregarding any theory, what is never a very clever thing to do.
Michael Wei-Chih Liu , You are asking about if you need to correct p-values for the multiple terms in multiple regression ? If so, I would say there isn't uniform consensus on that. I've seen people (e.g. textbooks and instructors) do so, but I think it's not that common, and I wouldn't do so. But, also, don't place too much importance on the p-values in the first place. I have some ranting about p-value correction here, which may be helpful in thinking about it: https://www.researchgate.net/post/Is_there_a_problem_of_multiple_testing_in_my_research_question_here
Sal Mangiafico ,
it depends what the model is for and what the coefficients mean.
If the model contains numerous predictors all of which "may or may not" play a role and the purpose is to identify those that explain some recognizable part of the variance of the response variable, then it may be relevant to control the FWER.
If the purpose to to model the response based on a set of predictors that are considered relevant on theoretical grounds, statistical significance may be relevant only for one (or some) of the coefficients in the model (or even not at all!), and there is no need to adjust p-values, because there is no "screening" involved.
Let me give a simple definition of ANOVA and Multiple Regression.
- ANOVA
The analysis of variance (ANOVA) is used to determine whether there are any statistically significant differences between the means of two or more independent (unrelated) groups (although you tend to only see it used when there are a minimum of three, rather than two groups). For example, you could use a one-way ANOVA to understand whether exam performance differed based on test anxiety levels amongst students, dividing students into three independent groups (e.g., low, medium and high-stressed students). Also, it is important to realize that the one-way ANOVA is an omnibus test statistic and cannot tell you which specific groups were statistically significantly different from each other; it only tells you that at least two groups were different. Since you may have three, four, five or more groups in your study design, determining which of these groups differ from each other is important. You can do this using a post hoc test (N.B., we discuss post hoc tests later in this guide).
- Multiple Regression
Multiple regression is an extension of simple linear regression. It is used when we want to predict the value of a variable based on the value of two or more other variables. The variable we want to predict is called the dependent variable (or sometimes, the outcome, target or criterion variable). The variables we are using to predict the value of the dependent variable are called the independent variables (or sometimes, the predictor, explanatory or regressor variables).
For example, you could use multiple regression to understand whether exam performance can be predicted based on revision time, test anxiety, lecture attendance and gender. Alternately, you could use multiple regression to understand whether daily cigarette consumption can be predicted based on smoking duration, age when started smoking, smoker type, income and gender.
Multiple regression also allows you to determine the overall fit (variance explained) of the model and the relative contribution of each of the predictors to the total variance explained. For example, you might want to know how much of the variation in exam performance can be explained by revision time, test anxiety, lecture attendance and gender "as a whole", but also the "relative contribution" of each independent variable in explaining the variance.
https://statistics.laerd.com/spss-tutorials/one-way-anova-using-spss-statistics.php
It is clear that the two (ANOVA and Multiple Regression) are different. One is used for comparison and the other for fitting, estimating and predicting. Their uses are also different from each other.
Abolfazl Ghoodjani , why have you plagiarized from https://statistics.laerd.com/spss-tutorials/one-way-anova-using-spss-statistics.php without attribution.
EDIT: Thanks to Abolfazi for correcting this!
Why have you not addressed the multiple comparison problem in the question?
Back to the question, as mentioned above this stepwise approaches have been heavily criticized. And how to adjust ps depends also on the cost of Type 1 and Type 2 errors.
Daniel Wright Thank you for your admonition.
I had the above text about ANOVA and regression in my writings and I did not know where it belongs.
I never intended to do anything called plagiarism.
Sure.
I corrected my text and wrote the source.
Thanks again for your admonition.
Hello Michael Wei-Chih Liu. Your question reads a little bit like a homework assignment from a graduate level stats class. In case it is, I'll just point to some resources you may find helpful. First, take a look at this Stata FAQ on problems with completely algorithmic "stepwise" variable selection methods:
- https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/
Regarding the so-called multiplicity problem, I find the following two articles to be very thoughtful. Neither of them addresses the specific question you raise, IIRC, but they do provide good general background. HTH.
Article Schulz K, Grimes DMultiplicity in randomised trials I: endpo...
Article Multiplicity in randomised trials II: Subgroup and interim analyses
- Badges
- Science topic
- Similar topics
- Linguistics
- Composition Studies
- Writing