Jochen

In your example grp and age is highly correlated and therefore removing age while retaining grp in the model increases significance of grp. Moreover sex is not very related to outcome, or one cay the effect of sex disappears when grp is included in the model.

Also by prediction it is generally meant the ability of model correctly guessing new outcome values. I guess you meant "estimation" when using the word "prediction" because I can not see anything related with prediction in your answer.

I would not look for the significance of the covariate itself. I would look for the impact of including the cofactor on the prediction (accuracy, precision) of the response variable. It may be that the p-value for a covariate (or a set of several covariates as a group!) is rather large but it yet has a relevant impact on the prediction of the response. It may also be that covariates with very low p-values do not have any relevant impact on the prediction of the response (I would exclude such predictors despite their "statistical significance" to get a simpler model that does its job still sufficiently well).

PS: be careful with analysing "scores" - they are often not interval scaled, and therefore models estimating differences between scores can give a very misleading impression.

Dear Shuran,

your reasoning is correct. it only makes sense to include control variables (the ariates in ANOVA) if they have some influence in your model

Lisete, yes, you are right that they should have "some influence" - but the relevant influence is not measured by the statistical significance of the estimated coefficient for this variable or of the estimated reduction of the residual variance but rather by the impact on the prediction of the response. It can be disadvantagous (not to say "stupid") to keep variables with low p-values and also to remove variables with high p-values.

I will give an example:

grp = (0, 0, 0, 0, 1, 1, 1, 1)

age = (12,13,14,13, 78,45,63,72)

sex = (0, 1, 0, 1, 0, 1, 0, 1)

y = (106.07, 107.67, 106.77, 107.77, 141.46, 125.73, 132.68, 138.95)

The variable grp is the interesting treatment variable, age and sex are possible covariates. The age is deliberately confounded with the treatment.

The full model shows nice small p-values for age (

Dear Jochen,

Thanks fou your reasoning and your explanation. Very helpful!

Jochen

In your example grp and age is highly correlated and therefore removing age while retaining grp in the model increases significance of grp. Moreover sex is not very related to outcome, or one cay the effect of sex disappears when grp is included in the model.

Also by prediction it is generally meant the ability of model correctly guessing new outcome values. I guess you meant "estimation" when using the word "prediction" because I can not see anything related with prediction in your answer.

Dear Dr. Wilhelm, 

Thank you for your answers and the example! I get your point that cannot decide if to include a variable as covariate in the model only by looking at the statistical significance of this variable, but to look for the impact of including this variable on the prediction of the response variable. But I'm not sure how to decide the impact on the prediction of the response (I have some difficulty to follow your example). I can hardly predict which brain regions will show in a whole brain analysis (an exploratory method) in SPM. I have tested and found that results vary quite a bit when including different covariates. So I need to decide which are "correct" covariates to include based on a principle to get reliable results. 

May any SPM experts help to answer?

Mehmet,

yes, I know about the correlation. That was intended to get a simple-to-understand example. I also pointed this out in my post above.

Regarding prediction/estimation: you are right, I looked at the coefficients, which are estimates of effect sizes. I did not look for the prediction of the response (outcome) variable. I was not using the right words - sorry!

Checking the accuracy of predictions is surely a different thing. It may be more relevant to look for prediction than to look for estimation (of coefficients) - that depends on the purpose of the model. However, if prediction is the relevant task, then one would need to apply some (cross-)validation procedures to ge a measure for the performance, and this was too much a demand for a small (possibly not very realistic, but hopefully yet instructive) example.

Anyway, thank you for pointing this out!

following

Jochen Wilhelm . I carried on multiple regression analysis using all covariates as predictors to find out if any of these predict the performance on dependent variable. Results (Beta) indicated no significant results. Can I interpret my results saying these variables (analyzed as predictors in multiple regression) have so significant impact and thus proceed without using them as covariates or even conducting ANOCOVA.

P.S. same predictors are being analyzed after treatment. significant findings were noted among two of these predictors.

Be careful choosing how to include covariates, and be careful interpreting the results. Looking to see, for example, if the covariate has a big association with (or even effect on) the outcome variable does not necessarily mean you should include it. A common example (the birth weight paradox, I have a few paragraphs here and give more cites to research more focused on it, Article Estimating School Effectiveness with Student Growth Percenti...

Absence of evidence is not evidence of absence.

Jochen, In your example group and age are nearly completely confounded (they explain very nearly the same variation) so of keeping both of them is redundant and not necessary which is different from being harmful.

Clearly the treatment is only given to the elderly so the controls are chosen well. The "harm" comes from the choice of controls, should come from the same population as those treated (the elderly).

Then the effect of sex should be added to see if the coefficient of group changes. If it does and the p-value is p≤0.10 then it should remain as a covariate because it affects the variable of interest and refines the model.

If sex does not change the model parameter for group, then you are permitted to leave it out as the treatment effect would be the same for men and woman.

It is hard for me to imagine that a real study would compare treatment of elderly patients with young people - poor design. But it is conceivable and often the case that sex has a small but important contribution to the effect of a medicine on the patient.

For Shuran, if the control and the treated come from the same population with respect to age and gender, and there is no effect on the outcome and the covariate is ≥0.10 then leave it out because it has no effect on treatment.

Jochen's example suggests that age is a very important effect on the response variable but because to the poor study design one cannot decide if the effect of the treatment is simply due to the age. Choose a proper control group that in most respects resembles the treatment group then you can properly test whether the covariate ( the nuisance variables) have an effect on the outcome independently of the treatment.

Your two examples are slightly different questions on the same problem. Both problems are regression (a very general term). So in both cases your instincts are right. If a nuisance parameters (age and sex) have an effect on the treatment coefficient then include them to refine the effect of the treatment. If not then leave them out and increase the power of your test by reducing the number of parameters.