Statistical analysis in cell culture experiments has always been a debate in our lab. In cell culture we most often do each experiment 3 times so we end up with n=3. Sofar we have been using one way anova to test for differences between the groups. But is that acctually correct ? We always use the nonparametric test (Kruskal Wallis + Dunn's post test) since you can't determine normnaility with only three samples (or can you ?) Maybe someone has a better suggestion on how to analyze cell culture data.
Cell culture data is anyway a difficult subject. What is the highest "level of repetition", what is a real "biological replicate"? For primary cell isolates, this is an isolation, but for an imortal cell line, this is not quite clear. Preparing different plates is nothing but a pure technical replicate. But what is the preparation of a new culture from a frozen stock? Or woud it be required to use -for each "n"- a new lot of the ordered cells from the vendor? And what if there are different vendors?
So it might not be very clear what "n" actually refers to. I know that many cell culture experiments are analyzed based on, say 3 culture dishes per group, and that the interpretation is based on the assumption that these were n=3 repetitions. However, to my opinion, these ae "only" technical replicates, and the results do not contain any information about their replicability in other studies, where the lab has to start with an entierly new (and thus more or less different) isolation and culture preparation. To my opinion the least requirement for a relibale interpretation of the results is to prepare n fresh cultures (best on n different days) from the stock that is available (it would just be too much to ask for having n different stocks!).
So if your n=3 refers to three culture plates that were obtained by splitting a culture into three plates, then you only have technical replicates. In this case, a generalizability of your results is questionable to an extend that will make any hypothesis test meaningless anyway.
Now to your actual question:
if you want to analyse mean (i.e. expected) differences between groups (more precisely, if you want to test the null hapothesis that the expected difference between groups is zero) there is only one procedure available, and this is the t-test. Dunn's test is a test on equal frequency distributions (that is particularily sensitive to a location shift). These tests are about different hypotheses, and actually, the hypothesis to be tested should be given a priori, based on your aims.
The distributional requirements for the tests are assumptions. This means for instance, under the assumption of normal-distributed residuals, the t-test is an exact test, and the error rates are controlled exactely. If the assumption is not true - and this is sure for any kind of real-wolrd data!- the test is not anymore exact but "only" approximative. The key question is: is the achieved approximation in the current case good enough for the purpose? Well, this question is hard to answer and it requires an educated guess.This can hardly be demanded from statistics (or from a statistician), since this depends too strongly on your aims and the specialities of your project and your field of research.
Therefore formal tests on these assumptions are illogical and nonsensical - they can not answer the relevant question. A rejection of H0 may be wrong because the "significant" difference is not relevant, and a non-rejection may be wrong because the power to get a relevant difference may have been to low.
And even if all this is ignored and such tests are performed, there are further difficulties. First, these tests must hten be performed on different data. Using the very same data for both tests will inflate the error-rates for the "target" test (rendering the whole procedure meaningless). Second, testing distributional assumptions typically requires considerably larger sample sizes than the subsequent "target" tests. But it would be stupid (in a standard basic research setting) to spend n=50 to test an assumption that is subsequently used to test n=3...
Assumptions are justified by theoretical considerations. They should be reasonable. They can not be derived from data. However, they should not be in an obvious contradiction to our experience, and this can be checked by inspecting similar data, e.g. from old experiments. Often there is a whole series of similar experiments. These data can be pooled and analsed as a whole to see if the data contradicts the assumption about the distribution. For this one has to have a model that allows appropriate differences between the different experiments, typically some mixed-effects model. The distribution of the residuals can then be analysed (not by a formal test, but in an explorative way).
With only three replicates you anyways have the problem that in a statistical sense you only have three samples to describe your entire population. You have to keep in mind that if you do them in parallel on one plate this is at least risky. If you do three replicates on three different occasions, i.e. in a blocked design, you are better of, finally you could do three plates (blocks) and measure multiple wells on each plate, which gives you a nested design that can be adequatelly analysed with e.g. a mixed effect model. It is true that using only three replicates it is hard to say if the data is normal. In ANOVA slight non-normality is not such an issue, however you need to ask yourself if you have equal variances. In my opinion a even more important point is you have to ask yourself if with three measurement you really adequately characterized your response. In pure mathematical sense sampling only three points from a distribution gives a huge random component (you can simulate that in R and you will be surprised how your mean responses jump around in each sample). To summarize... you have to ask yourself what is the effect-strength e.g. the change in the mean and how is your data distributed are the points all close within treatment levels and well seperated between treatments...you are kind of happy. If there is a huge effect but also huge variances... I would be more careful. The bottom line for me is that NO statistical test can save you from shortcomings in a experimental design... chosing a non-parametrical test especially after a exepriment has been done is not necessarily the best option. And for sure don´t do that just to achieve significance. Design the experiment properly, look at your data, plot it, use honestly your basic human sense... then apply stats to answer pre-defined hypothesis. Good luck!
Cell culture data is anyway a difficult subject. What is the highest "level of repetition", what is a real "biological replicate"? For primary cell isolates, this is an isolation, but for an imortal cell line, this is not quite clear. Preparing different plates is nothing but a pure technical replicate. But what is the preparation of a new culture from a frozen stock? Or woud it be required to use -for each "n"- a new lot of the ordered cells from the vendor? And what if there are different vendors?
So it might not be very clear what "n" actually refers to. I know that many cell culture experiments are analyzed based on, say 3 culture dishes per group, and that the interpretation is based on the assumption that these were n=3 repetitions. However, to my opinion, these ae "only" technical replicates, and the results do not contain any information about their replicability in other studies, where the lab has to start with an entierly new (and thus more or less different) isolation and culture preparation. To my opinion the least requirement for a relibale interpretation of the results is to prepare n fresh cultures (best on n different days) from the stock that is available (it would just be too much to ask for having n different stocks!).
So if your n=3 refers to three culture plates that were obtained by splitting a culture into three plates, then you only have technical replicates. In this case, a generalizability of your results is questionable to an extend that will make any hypothesis test meaningless anyway.
Now to your actual question:
if you want to analyse mean (i.e. expected) differences between groups (more precisely, if you want to test the null hapothesis that the expected difference between groups is zero) there is only one procedure available, and this is the t-test. Dunn's test is a test on equal frequency distributions (that is particularily sensitive to a location shift). These tests are about different hypotheses, and actually, the hypothesis to be tested should be given a priori, based on your aims.
The distributional requirements for the tests are assumptions. This means for instance, under the assumption of normal-distributed residuals, the t-test is an exact test, and the error rates are controlled exactely. If the assumption is not true - and this is sure for any kind of real-wolrd data!- the test is not anymore exact but "only" approximative. The key question is: is the achieved approximation in the current case good enough for the purpose? Well, this question is hard to answer and it requires an educated guess.This can hardly be demanded from statistics (or from a statistician), since this depends too strongly on your aims and the specialities of your project and your field of research.
Therefore formal tests on these assumptions are illogical and nonsensical - they can not answer the relevant question. A rejection of H0 may be wrong because the "significant" difference is not relevant, and a non-rejection may be wrong because the power to get a relevant difference may have been to low.
And even if all this is ignored and such tests are performed, there are further difficulties. First, these tests must hten be performed on different data. Using the very same data for both tests will inflate the error-rates for the "target" test (rendering the whole procedure meaningless). Second, testing distributional assumptions typically requires considerably larger sample sizes than the subsequent "target" tests. But it would be stupid (in a standard basic research setting) to spend n=50 to test an assumption that is subsequently used to test n=3...
Assumptions are justified by theoretical considerations. They should be reasonable. They can not be derived from data. However, they should not be in an obvious contradiction to our experience, and this can be checked by inspecting similar data, e.g. from old experiments. Often there is a whole series of similar experiments. These data can be pooled and analsed as a whole to see if the data contradicts the assumption about the distribution. For this one has to have a model that allows appropriate differences between the different experiments, typically some mixed-effects model. The distribution of the residuals can then be analysed (not by a formal test, but in an explorative way).
Thank you very much for your response ! We always use three different plates with 3 replicates in three different passages of cells and on three different time points. But in my opinion the best would be to just say e.g. there is a 20 % decrease/increase observed instead of saying it is a significant effect. But somehow erverybody else really wants the p-value.
I think that is curious to conduct an ANOVA on data set and conduct nonparametric post hoc test after.
As with ANOVA you work on means and variances, while on non parametric test you work more on medians and quantiles, so you don't measure and therefore test the same things.
Concerning the data normality of three repetitions, you can test the normality for sure and you'll see that there are most of the time. So it is no relevant to test data normality on three repetition.
See few normality tests here :
> shapiro.test(c(1,2,3))
Shapiro-Wilk normality test
data: c(1, 2, 3)
W = 1, p-value = 1
> shapiro.test(c(1,10,100))
Shapiro-Wilk normality test
data: c(1, 10, 100)
W = 0.8176, p-value = 0.1572
> shapiro.test(c(1,10,1000))
Shapiro-Wilk normality test
data: c(1, 10, 1000)
W = 0.7568, p-value = 0.01497
However it is most important to look at your data themselves and on the variance of each group.
Indeed to have the "right" to conduct an ANOVA or any parametric test you need to validate three hypothesis :
- group independence : here it means that the each repetition should came from different culture/population. But independence is a sort of opinion, because your measure will be measured with the same medium, in the same laboratory etc. so are they really independent. It is more a point of view question. But repetition coming from a same initial culture are for sure not independent.
- data normality : we already discuss this point.
- variance homogeneity : which is, i think, the most crucial point on data analysis with only few repetition. It is there that you see if you need to do a data transformation to analyse "properly" your data : usually we use the log10 or log2 transformation on cell culture analysis with loss > 1 log etc.
If we take two of the normal populations tested upper, as example, we can see that we need to use a log10 transformation (for example) to test them with a parametric test.
> var.test(c(1,2,3),c(1,10,100))
F test to compare two variances
data: c(1, 2, 3) and c(1, 10, 100)
F = 3e-04, num df = 2, denom df = 2, p-value = 0.0006671
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
8.555564e-06 1.301301e-02
sample estimates:
ratio of variances
0.000333667
> var.test(log10(c(1,2,3)),log10(c(1,10,100)))
F test to compare two variances
data: log10(c(1, 2, 3)) and log10(c(1, 10, 100))
F = 0.0582, num df = 2, denom df = 2, p-value = 0.11
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
0.001492615 2.270267230
sample estimates:
ratio of variances
0.05821198
Please consider my example only as example. I hope your data don't look like them.
I hope I answer some of your questions...
Regards
Thank you all so much for your answers it is highly appreciated !
To sum up and check if I understood all of the answers correctly here an hypothetical experiment (with a cell line so no possibility to get "new" ones each time):
Experiment Design:
- 3 plates (each with three replicates wich are randomly distributed on the plate)
- 3 different passages
- expermient performed on 3 different days
null hypothesis: The means do not differ significantly
Plot the data to visualize the distribution of my 3 means and then check for normality with the Shapiro-Wilk test/ take an educated guess
Most of our data is as % of control and I would then use the one-sample t-test to compare the means with my hypothetical mean of 100 %. If p
Maren, your experimental design seems clear now, also the null hypothesis you want to test. This hypothesis is about the means, so you must use the t-test. There is no other test for this kind of null hypothesis.
If you want check the residuals, this is the way you should go (in contrasts to what you wrote above):
Fit a model with the factors "passage" and "day" and the response "% of control". For 3 plates and 3 passages and 3 days you should have 3 x 3 x 3 = 27 values and thus you will get 27 residuals. Make a normal quantile-quantile plot of the residuals. If the points are all roughly on a straight line, the data in in accordance to the normal assumption. If you see a strange pattern, like a curved shape, then there is an at least recognizeable violation of the assumption. In this case you still have to decide if the violation is relevant (this is the difficult part!). In this case you need to refine your model (i.e., transform the data and/or the response). You may try different transformations and check when the residuals lie on a nearly straight line in the normal-quantile quantile plot. If you found a convenient transformation, you would know how to transform data of the following experiments to test the H0:"means (of the transformed variables) are equal". It is, however, quite common practice to run the test on the very same data and just pretend that it was anyway clear how the data has to be transformed. This practice is in a strict sense not correct because this procedure (looking for the data, deciding on a transformation based on this data, then do the test on the transformed data) will not control the error-rates at the desired levels. Since the control of error-rates is the only purpose of doing these tests at all, this should actually be a no-go (but either people do not know or they do not care).
Applying a test (like Shapiro-Wilk) to test the distribution of the residuals is nonsense, especially when the power is not specified. These test are made for a different purpose: to particularily select those instances that differ from the assumed distribution, for whatever reason. They are helping to distinguish relevant and irrelevant deviations. From a purely logical point of view these tests on normal distribution are non-sensical because the normal distribution is a theoretical model based on some infinitisemal calculus. The distribution explicitely allows values of infinite size and infinnitly small differences between adjacent values (-> continuous variable), and this is simply impossible for real values which are neccesarily finite and also discrete (typically because of a limited precision of the measurement, but at least because of the quantum structure or the fundamental physical variables). So we know, we are absolutely sure, on a priori grounds, that no real data will be normal distributed. It's just impossible, and so it is only a matter of sample size (power) to get a significant result of a test on normal distribution. And we know than any "non-significant" result is a false-negative. So there is no need to perform this test in the first place. Whatever we do, the important question remains: is the observed difference relevant to out problem?
http://en.wikipedia.org/wiki/Normal_probability_plot
http://en.wikipedia.org/wiki/Q%E2%80%93Q_plot
Maren: Not being a cell biologist, your experiment is not at all clear to me. I've read many replies regarding t-tests, F-tests, etc. My question to you is: what is it that you are measuring? And at what resolution? For example, if you have the coarsest resolution- which is simply two possible values, and if you have a very small data set, then all bets above are off the table. But you do have the freedom to revert to rudimentary distributions (e.g. Bernoulli, binomial, multinomial) and create hypothesis test on your own. Sorry if I'm being vague here. It would be nice to have more detail as to what you are actually attempting to measure, at what scale, and whether there is any measurement 'noise' involved. Ciao!
Thank you for the clarification Dr. Wilhelm ! I will do as you said and prepare a Quantile-Quantile Plot of the residuals and then see how my data looks like. Thank you so much for your time and effort to answer my question. I hope this will also be useful to others since I know a lot of people that struggle with "small n" statistic.
Dr. Sherman, my question was more of a general one. We meassure various parameters like cell viability or fluorescent intensity and all are expressed as percentage of control. I choose to keep this question very general as I was hoping this would be a good help for many others.
Maren, just a comment on your response to Peter:
I see your point, but Peter is also correct. Your question is practically relevant for biologists, and it is good when others profit from this case. For a biologist the assays and the kind of data is quite clear, so there is nothing much to explain.
But (IMHO) biologists are not the best audience to answer the question (and, yes, I confess being biologist, too). Most biologists -to my personal experience- have only relatively limited knowledge in statistics. They rarely are trained in statistical thinking and rather apply methods they have seen successfully published in papers of collegues. There is a lot of papers that are rather bad w.r.t. statistics, and the typical behaviour "we do what others do" simply leads to a multiplication of the same old errors and abuses (although I have to admit that this strategy is in fact successful w.r.t. the publishing output - unfortunately). Additionally, biologists tend to be rather test-focussed, hypothesis testing seems to be the ultimate goal in whatever they do. This is a bad and scientifically flawed tradition (despite the fact that almost no author, reviewer and editor is able to correctly explain what the resulting p-value of test really means!). And, lastly, biologists tend to use 100-years old techniques, ignoring that statistics is a dynamic and evolving science itself where huge progressions were achievied during the last decades.(*)
It would be more beneficial to get this question adressed to people who might not be well trained in biology but who have a deeper understanding of statistics, who are capable of statistical thinking. The are only few who know statistics as good as biology. SFor these people it is important to give as many details about the aim, the experiment, and the data as possible, since all this is very relevant information to find a good way for the analysis.
(*) It is often not the fault of the biologists. The whole system of statistical education in universities (for non-mathematicians/statisticians) is deeply down-at-the-heels and based on many serious misconceptions promoted by text-books written by biologists for biologists. There is no "environment" for biologists to grasp the important principles unless they are by chance very talented in statistical thinking anyway. Even worse: the current environment is focussed on a high publication output where just these "applied misconceptions" are a key for success. So we are in the very unfortunate situation that a scientifically good advice is often an advice that effectively hinders publications and carreers. That is a bad position, but I hope that it will slowly turn for the better.
http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf
Dear Jochen,
Dear Maren,
For use Shapiro Wilk test, here, with only 3 repetitions has non sense, as is all normality test with only 3 repetitions for me. That was my point when I used the Shapiro Wilk test, I'm sorry if I wasn't clear in my previous answer.
(Concerning the choice of the normality test, I usually use Shapiro test because I know what it does and the risk I take, I'm kinda use to it. For sure it exist some other tests... but I think that is not the question here. Anyway please find a paper which discuss well about this bellow.)
The H0 hypothesis formulated by Jochen is clearly a key point : ' H0:"means (of the transformed variables) are equal" '
But you can be more interested by the effect than by the mean (of the transformed variables) themselves. Especially in MANOVA. I think that depends on what you are looking for, in that case clever data transformation may help you to increase the power of the test.
I must agree with everything Jochen said about data transformation.
And Yes, I still don't know why everyone is talking about alpha (usually 5% -- type I error) but no body (except few and Jochen :) ) is talking about beta (type II error -- the hidden part of the tests).
http://instatmy.org.my/downloads/e-jurnal%202/3.pdf
Hi Everyone. The question is about the best way to statistically test for a difference in means with 3 independent experiments. There is no good answer for this because 3 experiments are not enough to provide statistical evidence that there is a treatment effect in this cell type under 3 different conditions unless there is a lot of data available to test the necessary assumptions, mainly the normality assumption.
From the above text I understand that you have 3 samples (the 3 independent experiments) with 3 treatments that you want to compare. There may also be some technical replicates within each treatment. The best way to show that the treatments have an effect on the cells is to show the data. Plot each data point - you can average technical replicates (which should have a very small variance). Showing the data allows other researchers to see 1. the variation between experiments and 2. the difference between treatments.
In my opinion, summarizing over 3 data points and making huge assumptions regarding the distribution of these measures and then providing a p-value is not helpful at best and is obfuscating or even misleading at worst. Why not show the data?
I understand the plight of experimental researchers when reviewers ask for statistical analysis but a p
Hello everyone,
My comment comes very late but I have a problem with all the suggestions made above. Parametric and non-parametric testing are meant to compare groups, each made up of strictly independent observations. Running independent experiments on different days, and even using different passages does not mean the observations are independent. The fact remains that we are using the same cell line and that our n=1! So talking about statistical significance in the first place is not valid. Using 3 different cell lines is an option; then n is really 3.
Nezar, this is only true if you want to statistically generalize across cell lines.
The best thing to do is assessing your outcmes using continous data is the best solution in case of smaller sample size.
- Badges
- Science method