There are quite a number of possibilities, Rezvan:

1. Null hypothesis is true, there is no difference between means (True negative). The large difference in mean value (I guess, you are talking about 12) may be due to sampling variance, which may be only decreased by increasing the sample size. What is your sample size here?

2. Null hypothesis is false, there IS difference between means, but you cannot observe it due to the high level of noise compared to the level of the signal (False negative). You may use a operating characteristic curve to determine the sample size for a sufficient "power" of your test. You may check any statistical text about "design and analysis of experiments" for a detailed treatment. Practically, increase the number of samples and check again. However, this time you will be increasing the type-I error, so beware!

3. Are you using alpha=0.05? What is the P-value of the test? 

4. There are alternatives to ANOVA, basically nonparametric methods like Kruskal-Wallis test and robust ANOVA, which may a have higher power compared to conventional ANOVA, if your data are sampled from nonnormal distribution. Though you say that your data are normally distributed, I may add that it may be difficult to guarantee normal distribution with a small sample size. Hence, using those methods may assist you in your work.

Good luck!

There are quite a number of possibilities, Rezvan:

1. Null hypothesis is true, there is no difference between means (True negative). The large difference in mean value (I guess, you are talking about 12) may be due to sampling variance, which may be only decreased by increasing the sample size. What is your sample size here?

2. Null hypothesis is false, there IS difference between means, but you cannot observe it due to the high level of noise compared to the level of the signal (False negative). You may use a operating characteristic curve to determine the sample size for a sufficient "power" of your test. You may check any statistical text about "design and analysis of experiments" for a detailed treatment. Practically, increase the number of samples and check again. However, this time you will be increasing the type-I error, so beware!

3. Are you using alpha=0.05? What is the P-value of the test? 

4. There are alternatives to ANOVA, basically nonparametric methods like Kruskal-Wallis test and robust ANOVA, which may a have higher power compared to conventional ANOVA, if your data are sampled from nonnormal distribution. Though you say that your data are normally distributed, I may add that it may be difficult to guarantee normal distribution with a small sample size. Hence, using those methods may assist you in your work.

Good luck!

Please post the data. Without seeing the data, any discussion about this is cumbersome.

Dear Dr. Burak Alakent

Thank you very much for your helpful tips. By your guidance I understood the problem was sampling variance (due to small sample size!). I increased the number of samples, and then significant differences were obtained between treatments!

Rezvan -

Hello. - This is the misleading nature of looking at a single p-value. They are functions of sample size. With a large enough sample size, you can 'disprove' virtually any null hypothesis, as they are generally not going to be 'exactly' correct, and with a small enough sample size, you generally cannot 'reject' even the most wildly incorrect hypothesis. It is all, instead, rather a matter of degree. It is not practical to test whether or not something is 'true,' but rather "How far off is it?"

So, although it is traditional to report p-values and "significance" (a misleading word), you may be much better served by reporting the various means with their standard errors, and perhaps confidence intervals.

Best wishes - Jim 

PS - Notes at the attached might be helpful, in general. 

Article Practical Interpretation of Hypothesis Tests - letter to the...

Dear Dr Knaub

Thank you very much for your precise answer and useful information. 

I agree with James regarding the reporting of the means and SEMs.  All too often I see results reported as 'significant' or 'no significant difference', when the sample sizes these conclusions are drawn from are too small for 'significant' or 'not significant' to have any meaningful value.  While it is possible to obtain a p-value when the included sample sizes are as low as 1 (yes, I have seen people say 'no difference', then march ahead to make some pretty broad generalizations), it should be noted that in order to obtain reliable, reproducible results, sample sizes often need to be much larger than what they are.  This is particularly true for molecular biology and -omics work, as conducting these sorts of experiments is costly (i.e. a single RNA-Seq library is going to cost you >$1,500, so no way is it feasible to have a sample size large enough to state with certainty that what is observed is most likely causal, and not merely a reflection of individual variation).  Hence, whenever your sample sizes are less than thresholds indicated by a power analysis, it is wise to tread lightly...and acknowledge the fact that your data 'suggests' but does not 'show' there to be a difference (or no difference, if what you want to see is an inability to reject the null).

MJ, I often hear that small sample sizes do not give "meaningful results". I wonder: given the assumptions of the test hold, whay should a "significant" result be not meaningful? The type-I error rate is controlled, and there is nothing more a test can do. And: with small sample sizes, a result will more likey be "significant" only when the effect is large (and hence likely to be "relevant").

Further, (in life-sciences) the type-II error rate is usually not controlled, so a "non-significant" result is anyway not interpretable (authors who do interpret such findings are wrong, but this is not the problem of the test or of an "unreliable" test result).

On the other side, "significant" results from tests with large samples are likely obtained for small, possibly "irrelevant" effects. Interpreting such findings can be quite unreliable...

So, what is "unreliable" on tests with small samples?

I think the problem is that the assumptions may not hold and that people think that this is not "testable" on small samples. I agree that it is not testable on small samples (I'd say that it is not reasonably testable at all!). But performing a test that is to be interpreted as controlling the type-I error rate requires to know a priori that the assumptions hold. Looking at on set of data to decide on the test (or ar tarnsformation or anything else) and then - using the same data again - to perform and interpret the test just makes such control impossible. Therefore I do not accept this argument ("Tests on small samples are not reliable because I can not make sure that the assumptions hold").

Increasing your sample size increases the power of your test, so that a result with the same effect size can easily become significant when the sample size increases.

More to the point, the pattern of means you reported suggests that the reason your effect size is small is because 4 out of the 5 treatment categories are not different from one another. Now that you do have a significant difference, you can legitimately run a multiple comparisons tests such as the Tukey test to determine whether the source of the difference is indeed concentrated in a limited subset of the categories.

The reason I argue caution in stating that there is a "significant" result when sample sizes are small is that both systematic and unsystematic variation is inherent in any system.  If one is to simulate distributions for a "population" of known value having a normal distribution, using, for instance, R, where there are no confounding variables and no human error (i.e. unsystematic variation does not exist), it is easy to note that it is not until you start getting 20-30 "samples" that what is observed begins to accurately approximate normal (may be worth the time to refresh yourselves on "the Law of Large Numbers", LLN).  For this reason, small sample sizes are said to follow a "t", rather than a "z", distribution.  Given that one of the fundamental assumptions of most parametric tests (i.e. the ANOVA for which this thread is concerning) is that the data are normally distributed, and that for small data-sets, it  can NOT be assumed our data are normally distributed, it is vital that appropriate tests and/or approaches be implemented (if nothing else, apply some form of correction, use bootstrapping, and/or perform nonparametric tests).

Furthermore, it is necessary for us, as researchers, to keep in mind that the 0.05 cut-off for "significant" or "not significant" , suggested by Fisher in the first part of last century, was never intended to be an absolute (see Experimental Design and Data Analysis for Biologists by Quinn and Keough, 2002).  This, it is vital that as researchers, we critically think about our system, and report in our manuscripts a logical and interpretable meaning of what is to be deemed "biologically significant", rather than falling back on an incorrect application of "statistical significance".

One final note... I feel it is important that in the biological sciences, we move towards reporting "effect sizes", which are standardized measures of the effect of a treatment, in either the form of Cohen's d, Pearson's r, or an odds ratio (such an approach is increasingly becoming standard in other disciplines, such as Psychology; in fact, the APA recommends that one of these measures be included in the results of all published works).

Thank you for the response, MJ.

So you say that non-normal distributed data is the problem when small samples are used for hypothesis tests. What can happen here with biological data? Most bio data is right-skewed. When one wrongly assumed that this data would be ok to be used with tests assuming normal distribution, what would happen? The test is NOT considering the case that there is an effect. The test is considering the case that there is NO effect (H0). So what would happen in this case? Well, the test will sometimes give a significant result, for sure. But just because of the skewedness of the data the frequency of such "false-positives" will be lower than the alpha used for the test (usually 5%). This is the only thing that is to be controlled in almost all projects (exception: well-planned studies based on a lot of previous experience with controlled power). The test will be more conservative than expected, the power will be unneccesarily low, but this is no problem when "non-significant" are not interpreted anyway.

"Systematic variation" as you called it is not influenced or reduced by larger sample sizes.

In many cases biological data is not extremely skewed, there are many examples where the data has a quite symmetric distribution. You can simulate with R that the averages of as few as 4 or 5 values already have a distribution very close to the normal distribution.

I want to note that not the data is t-distributed, and neither the means are. It is the t-statistic that is t-distributed, and the reason is that it contains the standard error in the denominator what has to be estimated, and that this estimate has a relatively low precision (for the normal model, the standard error [or the variance] is independent of the mean; this is different in other models like Poisson or Gamma)..

Finally I find it interesting that you note Fisher. It was just him who continuousely stated that a formal control of error-rates is stupid non-sense. He was aware that not error-rates are important but the qualified judgement of an expert who will use the "p-value" only as a rough guide, as one little mosaic stone while considering much more information that is not used in the test and not considered by the p-value. This said, it is relatively irrelevant for Fisher how good the normal-approximation of the likelihood finally is. There is no need (and no practical possibility and also no sensible reason) to keep a long-run error-rate. It is the individual result that must be judged, and a p-value is clearly not the right tool for this task.

And to you final note: It is good to really consider effect sizes and interpret them in a scientific context. But I am afraid that these relative or normalized effect sized you mentioned are only a by-pass for what is done with p-values anyway. People will start looking for standradized advices like "0.3 < d < 0.5 is a 'medium' effect" and "0.1 < d < 0.3 is a 'small' effect" and then everyone starts thinking that such 'large' effect is automatically relevant, like a small p-value seems to indicate 'relevant' effects. This will only restart the whole mindless misery.

I prefer eta-sq as an effect size, since explained variance is a pretty straightforward concept.

Why not try adding four more higher concentrations, same increment, above that which gave the value 12? Results occasionally can change suddenly at one concentration; which itself may be a valuable finding.