I am asked often this question and I would like to know how to do it properly.
The scenario is as following: Simple values are measured in two groups of animals, e.g. treatment and control groups. We are interested if there is a significant difference between the means of the measured value. Since the values are close to normal distribution, it is text-book case, we can use a simple t-test.
The problem: we have calculated that we need 12-12 animals to have the appropriate power to measure the expected differences, but for technical reasons (outside of our control), we can do the measurement only in 3-3 animals on a given day. Unfortunately, the wet-lab partner noticed, that the measured values are affected substantially by the day the measurement was taken because of environmental factors (again, outside of our control).
* Question 1: how to do the statistics to eliminate the environmental influence and measure only the effect of the treatment?
- My solution 1: let's just ignore the variance from the environmental factor, and do the simple t-test as if all the measurements were taken on the same day.
The problem with this: in some experiments (and on certain days) the effect from the environmental factor can be greater than from the treatment. Measurements are prone to fail, so there are days from which there are different number of samples from one or the other group, therefore the effect of environmental factor is not "balanced" throughout the measurements.
- My solution 2: We can calculate the mean of the value for each day in both groups and perform paired t-test on the treatment - control pairs where pairs are the means of the measurements taken on the same day. I am really not sure if this approach is sound from the point of statistics.
* Question 2: I want to help the wet-lab partner to design this experiment properly. How to design an experiment in the future to eliminate the (changing size) effect of the environmental factor? Constraints: In theory, in a normal t-test we would need 12-12 samples, but we can measure a maximum of 6 animals in one batch.
Thanks for any comment on this issue.
I would appreciate also any pointer what topics to study further for similar issues.
Hi Csaba,
I would suggest a variant to make N of measurements (3 experiment + 3 controls) with the N as big as possible, calculate difference (meanExp - meanContr) or meanExp/meanContr for each day, plot the distribution of differences, remove outliers and find median/mean, depending on the form of distribution. This would probably catch the sensible difference between exp and contr, but I don't know which statistics could stay behind this wild idea.
One thing I am sure is that you cannot merely pool all experimental and all control values from different days together. You can calculate a difference only for experiment and control measured both at the same plate+day.
Thanks Nina, that is approximately what I thought. I hope someone can make a comment on the statistics way to do this.
You can't use paired t-test. You can use paired t-test only if one subject in one group is paired with one subject in the other group. For example, if they are siblings, or if they are the same subject with measurements under different conditions.
The design you want is a randomized complete block design. Each day is treated as a block. You then have an equal number of subjects in treatment A and in treatment B on the same day. This design takes into account the differences in different blocks. This design assumes that the effect of block is the same for both treatments. That is, that on day 1, both treatments are higher by x, and on day 2 both treatments are lower by y.
This is the correct way to do Solution 2. And avoids needing to solve Question 2. And is similar to Nina Gubia's idea. Statisticians already did the difficult work for you 100 years ago.
This design is very easy to analyze with any decent software. The analysis might be called one-way analysis of variance with blocks.
In all cases, plot and examine your data first. For example, if it is not true that "the effect of block is the same for both treatments", it is not difficult to adjust the analysis to take this into account.
And, of course, make sure your data meets the assumptions of the analysis. In this case, treat the analysis as a general linear model, and make sure the residuals are normal, homoscedastic, and independent.
Thanks, Salvatore, I have suspected that statisticians have already done the hard part of the work, that is why I have asked for pointers. I will definitely study further randomized complete block design.
I agree with Salvatore's answer. Treat day as a blocking variable. The one additional point to make is that sample size will need to be larger to deal with the additional variable.
If the effect of day is larger than the effect of treatment, then you might want to do a preliminary study to really understand the effect of day. Also one presumes that the difference between any two consecutive days will change over season. This can cause major problems in your experimental design if you are forced to spread out sampling over several months. It also means that your sampling this year will likely produce slightly different answers than sampling next year. All of these effects will result in more blocking variables that you will have to adjust sample size to properly account for them.