It is a misconception that you would "testify the hypothesis of normal distribution". Tests use to "check for normal distribution" are actually tests to check for deviations from the normal distribution (the null hypothesis is that the data are realizations from iid normal random variables). Failing to reject this hypothesis is not a "proof" that the hypothesis is correct (it's only[!!] indicating that the sample size is not large enough to clearly see that the real-world data will not perfectly align with some idealized theoretical model!), and being able to reject it is no "proof" that the deviation is of any relevance (it only demonstrates that the sample size is large enough to clears see some deviation, but that might not at all be relevant).

The assumptions are based on understanding of the data-generating process. Often, this understanding is rather limited, and one needs to make some educated guess. Of course it is then appropriate (and recommended) to cross check if the data is roughly behaving "as expected" under these asumptions - provided the sample size is large enough to get some reliable idea about the distribution. This is not achieved by tests like Shapiro-Wilk, Anderson-Darling, Kolmogorov-Smirnov or others. It's better to check residual diagnostic plots, one of which is a normal-quantile-quantile plot (normal-QQ plot). Provided the sample size is large enough, this plot can show systematic deviations and give an impression of the kind and severity of the deviations. Together with other diagnostic plots one may make a judgement if the (set of) assumptions is reasonable / acceptable or if there ought to be a better set of more resonable assumptions.

With n < 30 (or so), you should not consider the CLT as argument that the assumption (normal distribution) is irrelevant. Even for n > 30 relying on the CLT can be probelmatic when the distribution is strongly skewed (as often in biology! E.g. for gene and protein expression, concentrations of biomolecules, titers of pathogens, etc.).

A particularily stupid example of applying a test to check for normal distribution is if the data are clearly binomial (yes/no, present/absent) data is coded as 0/1, or proportions (bound between 0 and 1, or as percentages bound between 0% and 100%). Similarily stupid is the example of testing data from counts and from strictly positive variables (concentrations, intensities). In all these cases the assumption of a normal distribution can possibly be okay, but we have very obviously more appropriate distributional assumptions (binomial, beta, Poisson, neg-binomial, gamma aor log-normal) and we have obviously more appropriate tools to deal with these kinds of distributions.

Thank you Jochen, for this thorough and insightful answer!

Also for small replications setting (eg. 3 per conditions), to examine significant difference, should I use non-parametric test instead of parametric test, such as teh kruskal-wallis test?

Thanks again!

Switching from t-test to U-test because of a small sample makes no sense at all, because of two reasons:

1) they are about different hypotheses. If I'd ask you about if I am late to the party but you don't have a watch, I would not ask you instead about if it's raining.

2) given the hypothesis tested by the the U-test is of interest, the test has a miserable power for small samples. With n=3 per group the smallest p-value you can get is 0.1.

Thanks Jochen! But should we in application just simply apply ANOVA test between conditions with 3 replicates? What are the hypothesis lying behind, normal distribution and more? How should we keep an eye on it that they are tenable?

Thanks again!

You should not "just simply" apply ANOVA. You should known what hypothesis you like to test and which procedure will allow you doing this. The procedure will rely on some assumptions, and you should know which those assumptions are and if these are reasonable in your case.

We are often interested in making a claim that something is increasing or decreasing, that the the values are higher or lower in one group. This seemingly simple question can be tricky when the distribution of the variable is not symmetric and the difference in the distributions is not just a simple shift. This can lead to scenarios where, for instance, mean(A) < mean(B), but at the same time median(A) > median(B) (I am talking about population parameters here). If you have a (sufficiently large) sample and use a test of which you don't know what hypothesis it tests, and that is "significant", what will your conclusion be: A>B or B>A? Both are correct! It depends which is the scientifically meaningful question.

An often useful measure of the "location" of a distribution is the expectation (aka the "population mean"). The t-test (ANOVA) is a tailor-made tool to test differences in means, but it is based on the assumption that the conditional distribution is normal with fixed (but unknown) variance (that is, only the mean is allowed to vary). In cases where this assumption is clearly not reasonable, this procedure should not be applied.

In the biomedical field we often deal with concentrations of biomolecules that varies in a large dynamic range, with 0 as a lower bound. There are many publications explaining the theory and providing real examples that such variables should be considered log-normal distributed. This is nice because we can then analyze the log-concentrations using ANOVA (or linear models, to be more general).

Sometimes the distributions cannot be normal, but are practically close to normal, e.g. when the concentration is tightly controlled and cannot vary much relative to its mean value. Examples are hormone levels or blood cell concentrations. Here it won't matter if the log-concentrations are used. It's often just more convenient to work without transformation.

In other cases we may have counts or proportions, which have to be treated differently, anyway.

It's rare that you don't know anything about the variable you want to analyze. You may find some analogy or investigate the data-generating process. You can figure out if there should be a (theoretical) dependency between mean and variance or if the variance should be constant and independent of the mean. If the variable is calculated from other variables, the distribution of the other variables might be known so that you could derive (at least approximately) the distribution of your target variable, at least you can make an educated guess.

Now after you have thought about all that and made up your mid about which properties the distribution of your variable should have (to the best of your knowledge), then you can decide if it is okay to use ANOVA, with or without transformation, or some other procedure. And if you came to the conclusion that ANOVA should actually be okay, than use it and cross-check the residual diagnostic plots if your sample data tries to scream at you that your thought might be wrong.

This looks like an interesting conversation between a biologist with poor knowledge of quantitative analysis and a statistician a bit away from the wet lab. I am afraid that Zonghan Gan confuses the observational units ( i.e.: the number of times he measures something) and the experimental units, which are the minimal units that can b eindemendently assigned to a treatment or condition. Maaybe if Zonghan Gan gives details about what he is doing we can give straight away indicatins about how to treat these numbers.

Fulvio Magara Thanks Fulvio for the kind consideration and reply! Yet sorry to declare, my confusion is only about replicates, which means the units of same setting for one condition, not measuring times. Thanks

Jochen Wilhelm Thanks Jochen! I see, it's firstly to observe whether the distribution is parametric/normal first. If not then transform into a kind of normal one (eg. in log), right? Non-parametric is a last choice and I tried it for my data, yes bad with 3 replicates samples. However, in clinical/aminal study, the replicates are often required to be >8. Is it for the reason confirming whether normal distribution, or there are other consideration behind?

I don't know why the required sample size is >8. Maybe that's from experience. Using a sample size of just 3 is extremely risky. If one sample does not "behave properly", the whole experiment will remain inconclusive.

For checking the distribution, the total number of samples is relevant. If you have 3 per group but 20 groups in an ANOVA design, you have 60 data points, and the analysis of residual diagnostic plots will be able to provide useful information. Same if you have just 2 groups but with 30 samples each.

It's moderately difficult to estimate a mean, it's much more difficult to estimate the variance, and it's again more difficult to estimate a skew, and this goes on to higher moments of the distribution. It is very difficult to estimate the form (i.e. the family, or general shape) of a distribution. The more difficult the estimation gets, the more data is required for a useful precision. The point here is: to check the assumptions in a meaningful way you would need way more data than to compare the means.

Practically, violations of the distributional assumptions are rarely the major problem. The biggest problem is typically the violation of the linearity assumption and the fact that conclusions are drawn that are not addressed by the hypothesis tests that were performed.

http://www.youtube.com/watch?v=Hz1fyhVOjr4

Fulvio Magara , you made my day. This movie is sooooo close to reality, if you haven't experienced it yourself, you won't believe it :)

Another question is: "I have these data and made that test. Can you make that significant?" (if you know a videao about such a discussion, I owe you two cups of coffee!)

Fulvio Magara truth has been spoken.....

Jochen Wilhelm I work as a consultant for behavioral testing and acually I considered making more of this kind of videos based on all the weird requests I received by students and even from group leaders. The previous video was brought to my attention by Prof. Richard Preziosi who gave great lessons on experimental design. I do not have a video with "can you make these data significant", but I do have a better one about the "non-significant result". I just wrote a paper on this topic that will appear in the next weeks on Laboratory Animals Limited journal http://www.youtube.com/watch?v=kMYxd6QeAss&feature=related