This is a comüplex topic and you will actually have to read several books about it to understand it. So I don't think that a post in RG will get you completely out of trouble here.

First of all, you should see Anova as a tool to "statistically compare nested linear models". If one model contains a factor (explanatory variable) than the other model does not contain and the Anova between these models gives a very small p-value, then the respective factor might be required in the model to be able to describe the data. If the p-value is large, then the small model performes as well in describing your data als the larger model, so you might consider to ignore the respective factor (if is is not neccesary to consider it on scientific grounds, for sure).

However, this tool is really of limited value. It does not do a good job for "variable selection", especially in larger models. There are many problems with this approach, logically and technically. For instance, it may be that the Anovas would telly you that you could omit "Material" if "Color" is in the model, but that you could also omit "Color" as long as "Material" is in the model. Such problems don't have simple answers.

The useful part of the models is that they give you the conditional expectations of the model coefficients. To my opinion, a good plot of these results is the only good guidance for interpretation. Tests and p-values are not helpful here. The same applies to related measures like R² and information criteria (AIC, BIC).

Interactions like A*B mean that the model is allowed to fit a different coefficient of an effect of factor A for each level of factor B. For instance, the fact that the Material doe not impact the response when the color is red, but it does when the color is green, then the model will require the Material*Color interaction to account for this. Without interaction the model must assume that the effect of the Material is the same, independent of the color. If this relation is further depending on the Shape, then the Model requires also the three-way interaction Material*Color*Shape.

When it is obvious that one of the variables does not improve the model, then the interpretation of the results can be simplified by exluding this variable from the model. This means nothing else that the model sees fewer groups with more members (as you said). There is no need to check the assumptions gain, because if the assumptions were ok for the full model and the excluded variable has no impact, than the assumptions must still be ok for the smaller model.

I can recommend you the book:

Practical Regression and Anova in R (free! downlad at: ftp://cran.r-project.org/pub/R/doc/contrib/Faraway-PRA.pdf)

and further this one:

http://www.stat.columbia.edu/~gelman/arm/

Jochen: You tear down quite a few statistical tests! I do agree with you here that RG is not the best source of information when someone needs a lot of information, as Zak seems to be needing here.

Raid, I don't want to tear down tests, I just want to tear down their often inappropriate and unhelpful use in science.

I’ll throw in my two cents.

It might be helpful to think of the analysis as a type of general linear model.  The dependent variable is a linear combination of independent variables plus unaccounted-for errors.  So you can include in this model the main effects (material, color) or their interactions (material x color).  Which factors are included really is “according to the needs of the study”, or more specifically, what hypotheses you want to test.

You could analyze by factor or by group.  Again it depends on what you are trying to find out.  Usually people want to know if there is an effect of the main effects, e.g. Does material matter?  And they want to know if there is an interaction of effects, e.g.  If aluminum is better than plastic, is it equally better for both the red and the green color, or is it more better for one color than the other?

If really all you wanted to know was which group was better, you could just test all eight groups and not worry about the factors. *

In any case, you _can_ do post-hoc testing to determine which group is better than which other groups.  This is sometimes called “multiple comparisons”.

I don’t understand what you are saying about type I and type II errors.  You are probably confused.

The best way to handle the assumptions is to treat the analysis as a general linear model.  This way you can look at the residuals from the analysis for normal distribution, homoscedasticity, independence, linearity.  Of course, the attributes of the residuals will be different if you analyze according to factor or by group.  But that’s not a problem because the models are different.

---------

*  Just to make things interesting:  If the data are analyzed this way, you can still construct hypotheses to test the main effects, e.g. color, by comparing all the red data to all the green data.

I know what you mean, Jochen. It is difficult battle to fight!

That's no reason to avoid fighting it, I think. As scientists we should do what serves science. It is nice when it also serves our careers, but we can't expect this to be always the case.

Dear Colleague,

I Appreciate Raid Amin answer

Yes of course this is true.

Thank you

Thank you for all those comprehensive answers! 

I read many things about last 2 days and now,  it's less confused in my head. 

to  answer some of your questions, my study is a little bit exploratory, so i want to get the maximum amount of informations.

However, the reason why I asked for the two ways to do (by factor/ by treatment) is that only 2/3 factors have a main sig effect on my supposing mediator Var (theorical implications) while i get a 3 ways interaction effect on my outcome one (managerial interests)

I should like to ask you again about treatments to keep for the next step. (analysis by treatment only)

Post hoc tests (i know that thoses tests are also harshly criticized by scinetists) showed that means of my 8 conditions form 3 different groups. ( 1, 4, 5 ≠ 2,7,8, ≠ 3,6)  

I'm using Process macro to test moderated mediation effects, so, I don’t know if I would keep all treatments or keep the 3 different ones (following post hoc results) or only choose the 2 those are extremely different.  (1 versus 6)

 Kd

I’m not very familiar with mediator analysis, so my general advice is to look for information specific to this analysis or to the macro you are using.  For example, the following link has some advice.

https://www.researchgate.net/post/How_do_you_interpret_the_moderation_test_using_Hayes_PROCESS_tool_in_SPSS

As I see it, the main point of your analysis is to determine the relationship between your Factors and your Outcome.   You have a significant interaction for the three Factors.

Usually in multiple regression, if an interaction is significant, we keep the lower-order terms.  So if A x B is significant, we keep both A and B.  If A x B x C is significant, we keep A, B, C, AxB, BxC, and AxC.

So I would keep all your Factors in the model.

And yes, I would do the post-hoc testing on all eight Treatments.  (Especially because the three-way interaction is significant).

 As to the mediator variable.  As you say, the relationship with the mediator variable has theoretical implications.  With what you presented, it sounds like you can say that the mediator variable could be considered a mediator for some Factors but not for all Factors.

Exactly, the mediation is not about all factors. 

For my outcome variable, I don’t get any 2 ways interactions. 

So, if I keep for example 3 treatment (Low, medium and strong) according to my outcome variable means (shown in post hoc test), would this work for mediation effects? normally yes!

I read a paper last night where some research about the effect of website atmospherics on memorization and purchase, 3 factors of web atmospherics showed only 3rd order interaction on memorization and intention to purchase. So the author combines the 3 factors and  creates a single variable (atmospheric richness of the website) with 3 levels (rich, medium, poor) and uses this one for the rest of analysis including mediation and moderation effects.  (Paper published in one of the higher ranked journals in Information Systems)

Another way for handling a significant three way interaction (ABC) is to break down the experiments by one of the three factors, say A. This way, we get as many 2-way ANOVAs as there are levels for A.  You lose information when you form one variable from A,B,C.  The quality of a published article often depends on the quality of the reviewers and the quality of the editorial board.

If you want to know interaction effect by two way ANOVA, you had better use ANOVA  by group as nominal scale such as 2 by 2 or 2 by 3.