Dear Jochen,

   I am probably missing something, but I am not quite on the same page.

You are expecting that the sample size will decrease when your estimate of variance is more precise. The expectation that it will be more precise seems reasonable whether you simply pool the estimates or use more sophisticated techniques to combine them. I do not understand why you expect that the estimate of variance will then decrease with better precision; in principle your estimate of variance from the smaller n situation should be more positively skewed then from the larger n situation. So on average I have no expectation that variance will go down if I use ML estimation. The expectation that it will decrease if I use normal estimation over n-1 is pretty small as it is driven by the tiny difference in biases between the two estimates. In the situation you describe I would not expect it to change the sample size at all. 

Power calculations such as what you demonstrate assume that you are able to specify the variance. Normal practice is to take our mean point estimate as the best guess of the population estimate; the precision of this estimate is not considered in situations such as you describe. It might be if a Bayesian model or something more exotic were being considered. 

Using the additional information to alter the std error of the observed mean differences requires assumptions that I rather doubt I would ever make, which is why I don't use Bayesian models...

Anyway, sorry if I am just expressing my ignorance.

Good question, Jochen.  My initial, off-the-cuff thought is to use an iterative trial & error approach with simulation.  I.e., take a guess at the required sample size (based on ordinary power analysis), and see what power it yields in a simulation (using maybe 1000 samples).  Then adjust your guess, and try again.  And so on until you arrive at the sample size yielding the desired level of power.  I am not an experienced useR (i.e., R user for the uninitiated), but I expect it can do those simulations fairly easily.

HTH.

Dear Jochen,

   I am probably missing something, but I am not quite on the same page.

You are expecting that the sample size will decrease when your estimate of variance is more precise. The expectation that it will be more precise seems reasonable whether you simply pool the estimates or use more sophisticated techniques to combine them. I do not understand why you expect that the estimate of variance will then decrease with better precision; in principle your estimate of variance from the smaller n situation should be more positively skewed then from the larger n situation. So on average I have no expectation that variance will go down if I use ML estimation. The expectation that it will decrease if I use normal estimation over n-1 is pretty small as it is driven by the tiny difference in biases between the two estimates. In the situation you describe I would not expect it to change the sample size at all. 

Power calculations such as what you demonstrate assume that you are able to specify the variance. Normal practice is to take our mean point estimate as the best guess of the population estimate; the precision of this estimate is not considered in situations such as you describe. It might be if a Bayesian model or something more exotic were being considered. 

Using the additional information to alter the std error of the observed mean differences requires assumptions that I rather doubt I would ever make, which is why I don't use Bayesian models...

Anyway, sorry if I am just expressing my ignorance.

@Bruce Weaver:

Yes, iterative simulation is surely a way to go... but I hoped that there is a nice (possibly approximate) analytical sulution out there...

@Bruce E Oddson:

The power calulation needs an estimate (or an educated guess) of the *expected* variance, that's correct. But the determination of sample size is basedon the fact that the actual sample test statistics, t = delta/SEdelta, are based on the sample data, and this needs to take into account the uncertainty of SEdelta. This is usually done via the shape of the t-distiribution, by using a t-distribution with an appropriate number of degrees of freedom.

When I use more sample data to estimate SEdelta, there are more d.f. for the t-distribution. Not the "population variance" goes down, nor will the SEdelta become smaller. More data should only reduce the uncertainty in SEdelta, what should be reflected in the increased number of d.f.

I am not sure if I got you right and if this explanation was clearer now...

Dear Jochen,

   Your clarification does not entirely put me at ease. The df of t is based on the number of differences involved. It is not based on the precision of the variance. The SE(t) is a function of the number of differences you observe and consider. Are you going to change your estimate of the mean change on the basis of other groups? No. Are you going to change your estimate of the SE(t) on this basis? Not unless you are convinced that the variance was incorrectly estimated - but this would require assumptions about how the variances are linked over the groups. 

   What you are calling SEdelta is a function of the variance - which can arguable be better estimated according to your setup - and the number of observations from which the mean difference was derived. Why does the df involved increase because you have improved the estimate of the variance? The df underlying your choice of t distribution is a statement of how many means were used to derive the difference. It is a statement about the precision of the expected difference of two random variables.

   A long time ago I proposed a method (unloved by reviewers) by which certain classes of missing data could contribute the expected amount of uncertainty per participant to the unexplained variance but not increase the precision of the estimate of those differences which were actually observed. The suggestion that you could do the opposite makes me wonder. If I took a population estimate of the variance, on unlimited df, would you think that it alter the expected distribution of the two sample n number of observations?

Dear Bruce (E Oddson ;)),

We seem to confuse ourselves... I programmed some simulation in R to check the effect of extra information on the power of a 2-sample t-test. The script is attached. Even if you don't have or know R you will be able to read and to understand the code. It "manually" calculates the pooled variance, the residual standard error, the standard error of the diffrence, and the t-value.

The code contains two functions for the simulation. A simpler one for the 2-sample case (not allowing any extra information) to better understand the principle, and another one that allows to add information from an arbitrary number of extra groups of arbitrary size.

The functions simulate normal distributed data (obtained via the function "rnorm") and calculate step by step the p-value of the comparison of two group means. This is replicated 10000 times. The power is per definition the fraction of p-values that are smaller than alpha, and this value is returned.

The simulations show that using extra information does in fact increase the power but does not inflate the type-I error, and that it does not matter (much?) if the extra information comes from a single large group or many small groups.

The key is that the pooled variance is calculated as sumSq / df, where df is the total sample size minus the total number of groups (-> so including the extra groups). Hence, including extra groups decreases the estimate of the pooled variance, and thus the residuals standard error and the standard error of the mean difference. Further, the p-value is obtained from a t-distibution with this (larger) df.

Notably (I tested this but it is not in the script) when the p-value is obtained from a t-distribution with df = 2n-2, using extra information leads to a decreased type-I error (and the power drops back to the value as if no extra information was available). Therefore I think that using a t-distribution with df = (total sample size - total numebr of groups) to get the p-value is correct. However, I lack the theoretical explanation.

I'm concerned, in principle, about using the other groups as part of the variance estimation.  If these are other exploratory groups, what relevance does their data have to the comparison of the two principal groups?  In general, something makes me uncomfortable about using those groups as part of the variance calculation.

I suppose that I'm really just reiterating Bruce Oddson's concern above, but it's important.  If the other k experimental groups are different than the 2 principal comparison groups, they should not be used in estimation of the variance.  Full stop.

But why not? This is actually a standard oneway-ANOVA design, only that one is interested in just one planned comparison (for which the type-I error should be kept).

I'll give an example:

You have some animal model where you treat animals (with whatever). The response depends on the time. You know from previous studies that, if these animals respond at all (if there is a treatment effect) you will surely see it after 4 weeks. So this is the test: compare untreated animals with treated animals 4 weeks after the treatment. BUT you are also interested in a vague idea if the effect is measurable alredy at some earlier time point, say after 1, 2, 4, 7, 14, or 28 days. Maybe there is even a peak at some earlier time, and knowing this can facilitate future experiments. So instead of analyzing animals only after 4 weeks you also gather some data at earlier time points. But if this data is measured, why shouldn't it be used to improve the effect of the 4-week treatment?

A standard design would be: use the same number of animals for all groups (time points) and do Dunnetts post-hoc tests (multiple-to-one comparison: all time points againts the untreated controls). This procedure makes use of the pooled variance estimate! The only difference is that the family-wise error rate (FWER) over all the many tests is controlled, what leads to a large sample size required to get a reasonable power. But if there is only one hypothesis to test, there is no need to control the FWER. That's all. I don't see the point why in this case the variance should not anymore be estimated from all the available data.

"My" design would, to achieve a desired power, require much fewer animals for all early time-points and additionally some require fewer animals in the two groups that are to be tested.

"But if this data is measured, why shouldn't it be used to improve the effect of the 4-week treatment?"

Because the distribution of the outcome variable at 1, 2, 4, 7, and 14 days is NOT necessarily the same as the distribution of the outcome variable at 28 days.  It's that simple.  

If you want to make use of the full time course data (i.e. perform comparisons of the treated animals vs. the untreated animals at each of the individual time points), that is fine.  But that's just a standard repeated-measures ANOVA, not the hand-wavy thing you're trying to do.

"Because the distribution of the outcome variable at 1, 2, 4, 7, and 14 days is NOT necessarily the same as the distribution of the outcome variable at 28 days."

I don't see why then the distributions of controls and after 28 days should be assumed similar?! *IF* the assumption of a similar distribution (and more precisely: of an approximate normal distribution of the residuals) and that of homoscedasticity is sensible for the two groups "control" and "28 days", *AND IF* there is no systematic difference for the measurements taken inbetween (except the time-point), then the assumptions should be sensible for the whole of the data.

"But that's just a standard repeated-measures ANOVA, not the hand-wavy thing you're trying to do."

No, it is not repeated-measures if the animals need to be sacrifieced for a measurement. I thought this way clear, because I assumed to have a group sizes (more animals in the controls and at 28 days, fewer animals inbetween), sorry.

But we are running off-topic. My question was not about the validity or sensibility of the assumptions. I said: "consider all assumptions are met". Take that as granted. My question was how to find the sample size for a desired power *given* all these assumtions are met.

I'm sure if I looked through my books on Bayesian Analysis, I would be able to find some formula for updating the test data based upon the total data. I'd have to look though. Interesting idea. 

Are you running a toxicology test by chance? 

Hi Andrew,

no, not only toxicoloy. Here are many labs running basic-research experiments related to lung diseases, many of which are animal models. They study the lung histology, compliance, heamodynamics, gene regulation, and much more. Many experiments are planned to include many time points or concentrations, and the typical way to analyze the data is in fact using some factorial model and planning multiple-to-one or even all-pairwise comparisons. I think they could save the lives of quite a number of animals when they would go for "t-tests using pooled variance estimates" for the most intersting comparison (there usually is one). The problem is that the PIs have to write applications where they have to present a sample-size calculation...

Looking forward to your findings!