Statistical significance is a "surprise index". The p-value tells you how much you should expect effects of the size you actually observed (or larger effects) under a specified null hypothesis (and a proability model, and given the data you have).

In your case you seem to have a null hypothesis (H0) about the frequency of "FF", the value may be f0. In your data you observe the frequency fdata. Given your probability model you can calculate how likely you will get such values of f that are at least as far away from f0 as your fdata.* This is the p-value. So, if H0 was true, then the p tells you how "surprised" you should be by your data, given H0 was true. The lower the p-value, the more surprised you should be.

If you do a series of such calculations like in a screening, you get such a "surprise index" (that is, a p-value) for each comparison. There might be one (ore more) quite low p-values, where you would say "hey, that's so suprising if H0 was true that I will reject this hypothesis and believe that the alternative hypothesis is true". But if you'd look at the whole series of p-values you have, it is less surprising to get some small values even if H0 is always true. This is like drewing lots from a lottery wheel. If you take a single lot, you would be very surprised to win. The more lots you take, the less surprising will it be to win. So in order to correctly express the "degree of surprise" that at least one of the p-values is very small, the individual values need to be corrected. If you have 4 p-values and if you would reject H0 in a single case when p

Statistical significance is a "surprise index". The p-value tells you how much you should expect effects of the size you actually observed (or larger effects) under a specified null hypothesis (and a proability model, and given the data you have).

In your case you seem to have a null hypothesis (H0) about the frequency of "FF", the value may be f0. In your data you observe the frequency fdata. Given your probability model you can calculate how likely you will get such values of f that are at least as far away from f0 as your fdata.* This is the p-value. So, if H0 was true, then the p tells you how "surprised" you should be by your data, given H0 was true. The lower the p-value, the more surprised you should be.

If you do a series of such calculations like in a screening, you get such a "surprise index" (that is, a p-value) for each comparison. There might be one (ore more) quite low p-values, where you would say "hey, that's so suprising if H0 was true that I will reject this hypothesis and believe that the alternative hypothesis is true". But if you'd look at the whole series of p-values you have, it is less surprising to get some small values even if H0 is always true. This is like drewing lots from a lottery wheel. If you take a single lot, you would be very surprised to win. The more lots you take, the less surprising will it be to win. So in order to correctly express the "degree of surprise" that at least one of the p-values is very small, the individual values need to be corrected. If you have 4 p-values and if you would reject H0 in a single case when p

Some authors do not recommend using Bonferroni correction, because some simulations show its bad fit to the real significance (expectation). If you have small number of coefficients, you can just use a level of p = .05. P after Bonferroni correction can also produce Type II error.

I'm curious! Why were you correcting when you are only doing four statistical tests (as evidenced by your corrected p value)? I doubt most people would consider the risk of type 1 error is so large with four tests that they would need to correct.

Aside from that, the Bonferroni correction has its critics as Selman Repisti mentions above- it is very very conservative. I favour the Holm-Bonferroni correction which has less likelihood of Type II error.

However, as discussed wonderfully above by Jochen Wilhelm, if your calculated p-value (p=0.012) is below your chosen cut-off (p=0.0125), then it is 'significant'. Clinical meaning is a completly separate issue, and is a clinical judgement, not a statistical one.

I concur with the above answers. Clinical significance is all about the amount of change that would make a difference (e.g., in prognosis, treatment effects, level of risk, etc...) sufficient to justify the costs of some intervention (e.g., expenditure of time and resources, intrusiveness).

hi,

i like to ask if p value is very very less as an example 5.22701E-13 ..what does it mean

It means that one would absolutely not expect to get the observed data or more extreme data under the given model and the tested hypothesis.

Hi Jochen, 

Thank u for ur reply.Is there any other statistical test I could do if ttest value for p is coming too small

Any particular test is about an own particular hypothesis. I don't know of that there would by any possibility to use a different test without changing the tested hypothesis.

But why are you uncomfortable with a low p-value? Where is the problem?

Hi Jochen,

Thanks.Is not that too small to tell significant....will it be  ok if I claim that the signicant difference is significant 

??

Claim that the significant difference is significant?

The very small p-value you obtained may serve as a reason to believe that the data is incompatible with tested hypothesis*, and this is usually stated as "the observed difference was statistically significant" with some indication of the actual p-value. The p-value should be given as is, but distinguishing very tiny p-values numerically does not make much sense**, so that it might be sufficient in your case to state that "(p < 0.0001)".

---

* this also means that one believes that the model is correct, what is surely not the case (because all models are wrong). Having large data sets will inevitably give low p-values, actually demonstrating this fact that the model is wrong. This is also known as "overpowering"; the analsis gives tiny p-values for irrelevant small effects.

** p-values are tail-probabilities of a statistical model. The model is never exact, and the relative error or discrepancy between model and reality increases in the tails. Therefore, the smaller p-values are, the less "correct" are they. If study A reports p = 0.02 and study B reprots p = 0.15 (for the same design, same sample size), one might believe that the data observed in A provides more evidence against the tested hypothesis than the data observed in B. But for p values of 1E-8 and 1E-12 I would not dare to make any comparison between the two studies.

when you comparing treatments with Control (one at a time) the appropriate method is Dunnett's test

References

Dunnett, C.W. (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the Americal Statistical Association, 50, 1096-1121.

Dunnett, C.W. (1989). Algorithm AS251 Multivariate normal probability intervals with product correlation structure. Applied Statistics, 38, 564-579.

Hsu, J.C. (1996). Multiple Comparisons Theory and Methods. Chapman & Hall, London.