Andrew's answer is correct, as usual, but thi time I'd like to widen it.

The power assures a minimum rate of correct rejections of H0 under H1 (effect is at least of the specified size, let's call it E>R [R for relevant efect size]). This des not mean that a rejection is done only when E>R in a particular case. In fact, if you imagine that E=R (where you expect the rate of H0-rejections equals the power), you will get (given the normal assumption) 50% of the effect estimates being greater than R and 50% being smaller than R. There is no bound to these differences (although Andrew gave an estimate for a reasonable range that will include most of such deviations). It can happen (for sure these are rare but possible events) that the estimate is quite far off the "true" value, and according to the rules such results (when their p

Could it be that the SD is smaller than expected when doing the power calculation? Or did they include more patients in the study than indicated as the minimum number of patients in the power calculation done before study start? Finally choice of statistical test can of course influence the power to detect the difference, so it could be a different power calculation method from the statistical test done.

How man samples do they have? If they have say 49 samples, and a Std Deviation of 1.7, the standard error is 1.7/Sqrt(49) = 0.18. If the have a t-test, they can state that a difference of +/-(2.0ish)*0.18 is significantly different. 

Andrew's answer is correct, as usual, but thi time I'd like to widen it.

The power assures a minimum rate of correct rejections of H0 under H1 (effect is at least of the specified size, let's call it E>R [R for relevant efect size]). This des not mean that a rejection is done only when E>R in a particular case. In fact, if you imagine that E=R (where you expect the rate of H0-rejections equals the power), you will get (given the normal assumption) 50% of the effect estimates being greater than R and 50% being smaller than R. There is no bound to these differences (although Andrew gave an estimate for a reasonable range that will include most of such deviations). It can happen (for sure these are rare but possible events) that the estimate is quite far off the "true" value, and according to the rules such results (when their p

Your question makes an important point. The big problem with this is that statistical significance is the criterion for publication in most journals. As Jochen said, this makes no sense for a single experiment. Replication is necessary. Application is necessary. You cannot reliably apply an effect that is smaller than its range of error. Statistical significance may indicate an area of further inquiry, but it does not indicate a controllable process or an establish an effect.

Jochen, thank you for stating that so well. We are far too dependent on NHST and consistently misinterpret p as indicating meaningfulness. Statistical significance is a function of sample size, error variance, direction of the alternative, alpha, in addition to the effect size. Any of those other factors could account for a statistically significant result in the face of a very small effect. You can't look at p alone.

Thanks for the nice feedbak, Amy.

It is correct that the "parameters" influencing the significance (p-value) are numerous and too often neglected. This alone is often a major cause of confusion. However, I wanted to stress the point that even the meaning of the result (p-value or rejection/acceptance of H0) is not clear. So what is usually done is that this procedure or "function"

y = f(x1,x2,...)

is applied to solve some problem. Theis "function" is the hypothesis test. You stated that people tend to oversee the relevance of the parameters x1,x2,... (some migt not be even aware of all the parameters). That is correct and I agree. I said that even the meaning of y remains completely unclear or is severly misinterpreted.

In conclusion, large parts of the scientific community seem to apply this y = f(x1,x2,...) ignoring the x's and not knowing what the y means. Where is the rationale there?!

One more minor remark: You wrote "NHST". This is an artificial thing, reproduced in many (not so good, mostly "introductory") stats books as a conglomerate of the Neymanian hypothesis test and the Fisherian significance test. Both procedures are logically incompatible (and have different aims!), and Fisher and Neyman had a tough controversy about it and never found a unifying solution. The NHST is a logical freak, something like an amyelencephalus, born out of a complete non-understanding of the underlying philosophies.

Right Jochen! NHST = amyelencephalus.

Thank you all for an amazing discussion. I come from the point of view of a meta-analyst in the healthcare world trying to interpret the results of many trials. So perhaps I can feel some comfort that when several trials are combined, we are in fact addressing the problem of the single experiment when we are trying to look at replicates over various trials that have been published. I thank you all - my question is quite likely due to ignorance and a simplistic question of considering one trial in isolation. If that is the case, then perhaps I can feel more comfortable that the results, although perhaps nonintuitive in isolation, may make more sense when combined, and i was fretting over nothing.

Thanks Sergio! I have recently become more interested in the Bayesian approach. Especially in term of evidence based medicine and the need to integrate all forms of evidence, not just clinical trials. Can you guide me to the best way to learn more about the Bayesian approach in detail?

Thanks in advance!

Sharon

Calculation of sample size is really an educated guess.  When we calculate sample size we specify an "expected difference".  This does not preclude a smaller measure of effect than our preliminary estimate can be " statistically significant"

Jochen's point that hypothesis testing does not help us learn about a particular case is key in communicating p-values and hypothesis testing results to our non-statistical colleagues!  Especially in the case Sharon asks about.   Power calculations are useful to give you a good study design, to set you off on the road to good research.  Don't confuse your power calculation with assessing your result.