If a t-test (whether alone or as part of an ANOVA) makes sense at all depends on the nature of your response.

I think you will not become happy with your wish to simply compare the mean responses at different level combinations. To my opinion you should better look for some sensible model that describes how the factors influence the response, possibly including interactions (what often is one of the most important aspects, scientifically). I don't know what best to do with your factor B, if this factor could, for instance, also be considered as some quantitative variable.

A quite flexible framework to solve such problems like yours are general or generalized linear models.

Hi, Dr Wilhelm, and thank You for Your answer. :) Y is a numerical variable, so are A1, A2 and B. This is the screening design where I only need to determine if factors A1 and A2 have significant linear effect on Y or not.

I once analyzed data in a similar design (without blocking) by trying to fit a linear model to it. However, p-values of the corresponding terms were changing depending on inclusion/exclusion of other terms to/from the model, so I was not sure how to answer the question if a factor had significant effect since some models included it in/as a statistically significant term, whereas others did not. Those different models all had high R2 and adjusted R2 so I could not decide which one to choose. This is why I wanted to try another approach this time.

The issue of interaction between A1 and A2 can be solved with ANOVA, I guess. At least, this is what I read in textbooks, but if You have any suggestion, I would be more than happy to see what you think about this problem.

Anova is only a tool to compare two nested models (i.e. the smaller model equals the larger model but some terms are dropped). Under the null hypothesis that the dropped terms are not associeated with the response, then the change in the residual variance is F-distributed (given the residuals itself are normal distributed), and this is what is judged by an Anova. Comparing a model including the interaction with a model not including the interaction by Anova gives you the significance of the interaction term in the model.

I see, thank You for explaining this. So, if I start with a full model : Y =c0 + c1*A1 + c2*A2 + c3 *(A1A2) , dropping terms one by one and performing ANOVA on the full and thus obtained smaller models will give me results (F, p value) associated with the effect of the dropped term?

I have to ask one more question. I have n=25 subjects and b=5 levels of B. Is it possible to calculate power of ANOVA for detecting effects with f^2 = 0.02, 0.15 and 0.35? I can adjust the values of b and n, so if power can be expressed as a function of n and b, I could calculate them in order to have enough power (say 0.8).

Yes, that's correct.

If I understood your design correctly, then you have 2 x 2 x 5 = 20 measurements (Y-values) for each individual, so here a mixed model with "subject" as a random factor will be likely be appropriate.

I don't know if there is an easy analytical way to determine the power of mixed models, so it might be that you will have to find the power by simulation. But I wonder if you should be concerned about power here at all: are you sure that you know the practical implifactions of the given f² values? And did't you already fix the number of individuals participating in this study (i.e. 25) anyway? Are you really going for Neymanian hypothesis tests or do you rather "just" like to calculate the significance of your data under some null hypothesis (i.e. a Fisherian significance test)?

Thank You for pointing out to mixed models. I will definitely use them. You understood my design correctly, there will be 20 measurements for each individual (since b is 5, but it can be increased up to 30).

I'll try to find an analytical approach to calculating the power of mixed models, but if I do not find it, I'll resort to simulations.

Initially, I have planned a sample of n=25 individuals, but can easily change their number as I'm currently in the planning phase of the experiment. For example, if a power analysis showed that n=15 would be sufficient to detect small or medium effects with the probability of 80%, I would simply exclude 10 subjects and measure the response on the selected 15. :) Increasing the sample size over 25 would be a bit problematic, but doable.

The design that would suit my needs perfectly would be powerful enough to detect effects that can be considered small. If that requires too large n (and/or b), I would be satisfied with the design powerful enough to capture effects of medium size. Now, I saw that f^2 could be used to measure the size of an effect, but You are right, I do not really understand its natural meaning in statistics-free context of my problem. I will try explain what could be considered a small effect in this experiment. Since 0 represents the low values of A1 and A2, and 1 represents their high values, if increasing A1 from 0 to 1  (equivalently, decreasing from 1 to 0) yields a change in average response that is equal to 10% of standard deviation of Y (sd(Y)), and I can statistically confirm this change with a p value lower than 0.05 (or 0.01), I would say that A1 has small effect on Y. Similarly, I can define the thresholds for medium and large effects as 0.5 sd(Y) and 1 sd(Y). 

I'm not familiar with Neymanian hypotheses, unfortunately. I've never heard for them until now. :(

There are two philosophies of testing. One was developed by Ronlas Fisher. He wanted to provide a tool to asses the "statistical significance" of the observed data under a given model and a given ristriction in this model (very simple, the model allows to fit two mean values, and the restriction is that there is only a single mean value. The Anova of the comparison of such two models eventually reduces to the well-known t-test [in this case, t² has the same distribution as F]). This procedure results in a p-value, that has to be interpreted in the whole context of the experiment. If p is very small, the data is said to provide "evidence against the restriction (the null hypothesis)", and the conclusion is that the unrestricted model is to be preferred to explain the data.

Jerzy Neyman and Egon Pearson were of the opinion that the interpretation of p-values after the experiment is too subjective and that "failing to reject the null hypothesis" is actually a non-permissive "action". They thought that an experiment should provide the information required to make a decision between two alternatives (which are meant to be "behavioural alternatives": either act as if hypothesis A was true or act as if hypothesis B was true). The actions are associeated with possible wins and losses, and by a clear definition of a loss function it is possible to set up decision rules that will minimize the "expected loss" (or maximize the expected win). This is where the power and the size of a test come into play. They need to be oriented at the loss function which depends on actions you will take. This startegy is surely resonable in an economic context (where the "loss" may be defined in monetary units) - but I have never seen or understood how this will fit into research.

Both approaches (Fisher and Neyman) are logically incompatible, but unfortunately they are yet mixed into a "null-hypothesis-significance-testing" (NHST) approach that is tought in many stats courses and books for "non-statisticians" (biologists, for instance). This induced a lot of confusion about the logic of testing and the concepts of power and size and significance and all that. But NHST is an inherently logically flawed procedure. At the end, it will never make really sense - the only "sensible" fact about NHST is that everyone thniks he understands it and that reviewers will accept papers that adhere to it. But that's mor religion and tradition, not science.

So, power analysis actually belongs to Neyman's approach, and statistical significance to Fisher's and they should not be mixed? I read several papers where it was advised to select the sample size based on the power of detecting the effect of a predefined size. Detecting the effect was a matter of finding p value less than 0.05 or 0.01. 

If I understood You correct, if I'm going to make a decision based on p values, then I should not be dealing with power. Am I right?

Yes, you are right.

The strategy to make a "decision based on p-values" is to make the best possible experiment with any feasable effort, and then judge the significance of the obtained data to see if your initial ideas and investments were worth to continue research that assumes the unrestricted model. If you find the significance is too weak, it means that either your idea wasn't that good and/or that you simply don't have the means to demonstrate that the restricted model preformes too bad to be used in further research.

Eventually, I also don't see any great benefit in calculation significances at all (at least in many cases). To judge what the p value means you must anyway consider all other aspects (sample size, kind of data, difficulties of the experiment, a priori plausibility of the model, plausibility and meaning of the restriction, the observed and expected size of the effect, and much more). If you consider all this, knowing the p-value does not add any further useful information.

To my opinion, significance tests (if they are really done as such) and NHST (what is usually seen in papers) are largely misused because people think they get "something for nothing", and they do not need to think hard (and it is hard!) about their data, and that they think that "p

I'm not a statistician myself, but I have to use statistical tools if I want my papers to be publishable. :) As You said, it's a convention.  In the field that I'm currently researching, metaheuristic optimization, statistical tests and frameworks such as Design of Experiments are getting more and more important.  Statistical significance is understood as a means to objectively confirm or reject certain hypotheses. However, whenever I delve deeper into statistics I get stuck because:

I agree that p-values are misused. For example, are p=0.03 and p=0.063 so different that the former deserves to be investigated further, whereas the latter should be completely disregarded? Also, I do not find p values to be intuitive.

You mentioned that we need to consider observed and expected size of the effect, sample size, etc.  If we return to the original problem, what should I do? :) For example, how do I calculate the effect size of A1? I found that someone  had already asked a similar question: http://stats.stackexchange.com/questions/221174/effect-size-for-comparison-of-two-nested-models  , but there was no answer there. :) I found this tutorial for mixed models: http://www.bodowinter.com/tutorial/bw_LME_tutorial.pdf but the author interpets the output of anova(model1, model2) based on the p value. Is there a way to to derive the effect size from the results of anova(model1, model2)? :)

P.S. I'm sorry if I'm asking too much questions and taking You too much time. :)

The problem with effect sizes is that ther is no simple answer, especially when then model becomes a little more complicated. Should one consider absolute (like a mean differcence) or relative effects (like Cohen's d)? What if several aspects are to be considered, how to weight them? (e.g. A maes a "large" effect, but B doesn't - would this be relevant? What if both, A and B only have a "small" effect? etc)  

Statistics like eta-squared and similar are in facct "statistical effect size measures", because ther are methods to handle such values (e.g. to determine a sample size for a given power etc.), but it usually remains unclear what it tells us about the real-world phenomenon we want to analyze. A simple mean difference is easier to understand. If you are particularily interested in the mean difference between A1==0 and A1==1, then I would suggest that you make up your mind how precisely you like to estimate this difference. The sample size required to achieve the desired precision under the given residual variance can be determined by simulation (for mixed or otherwise complicated models).

That's a nice tutorial you linked.

Ok. I see. Then, is it possible to estimate (absolute or relative to sd) mean difference between A1==0 and A1==1 by comparing the full model with the one without A1?

The reason why I asked if a paired t test can be applied in the first place was that it seemed like a straightforward way to compare mean values of Y for different values of A1 and if I specified the minimal difference that I would like to be able to observe I would have a wide choice of ready-to-use software packages to calculate the required sample size.

The tutorial is really nice, it helped me a lot. :)

I don't know any details of your research, your data, and your model, so it's hard to give you any good advice here. The fact that you have several explanatory factors (A1, A2 and B) makes me think that the difference in Y at A1==1 vs. A1==0 might depend on the values the other factors take (i.e.: there exist interactions between these factors), so that there is no "general effect of A1" but rather an "effect of A1, given A2==0 and B==0", and an "effect of A1, given A2==1 and B==0" and so on. Removing A1 from the model will impact all these interactions. The Anova will tell you if excluding A1 from your model will make your data unexpected (the p-value is a measure for "how unexpected" your data is under the restricted model). Due to the complexity of the model, there is no simple answer to the effect size related to A1 (becuase there are many possible aspects that can all come together).

I think for planning your sample size according to A1 you should focus on one difference in Y between A1==0 and A1==1 (not looking at the values of the other factors) and ask yourself, how precisely you want to estimate such a difference. I would still try to get this by simulation of the full model.

I see. Main effects of factors ignore any influence of other factors, which is what I wanted to test with comparing mean responses for two values of A1. Of course, I would also test interactions to gain more insight. I will try two methods for determining b and n: simulations and confidence intervals for means of Y at different leveles of factors. I found the results of a previous study and will try to use the estimates of standard deviations to find such values of b and n so that the confidence intervals are not wider than a predefined width (precision). Then, I will compare the results of the two methods. Thank You for Your comments and suggestions. They really helped me a lot. :)

I am not really preview to the details of your research work. It would be quite cumbersome giving you a precise advice on what you want to do. The mere fact that you have  a lot of  factors (A1, A2 and B) makes me believe that the difference in Y at A1==1 vs. A1==0 would depend on what the other factors take on in terms of the values that emanate from the other factors. It would be somehow good if you resort to using ANOVA to help you find where the differences are or better still run a MANOVA to help you know the unitary effect of the various factors.

In another jurisdiction, you may want to run a multiple regressin analysis iff you want to know how each of these factors are contributing to the determination of your key variable.

Also, it would be ideal for you to plan your sample size based on your indicated variables as in A1 etc and focus on one difference at a time as indicated earlier by Jochen and quiz yourself on how precisely you want to carry out such task.

I would advice you consult Andy Field's boo on "Discovering Statistics Using SPSS" third edition for assistance on how to go about some of the things since you know exactly what you want.

Thanks

Thank You for Your answer. I will consult the suggested book. :)