Statistics is the mathematics of uncertainty. It tries to give you the most likely answer given available information. However, this is no guarantee of anything despite how we treat the numbers. Every number that you calculate, from coefficients to p-values, has a distribution. For something like a p-value you tend not to see the distribution, but think of it this way: You did your experiment once and calculated a p-value. If your friend Venkat did the same experiment, it is unlikely that he would get exactly the same values, and therefore his calculated p-value would be different. If you did the same experiment tomorrow, you would get a slightly different set of random patients, and you would get a slightly different p-value. Do this 1,000 times and you will see that the p-value has a distribution.

The point is that the sample size calculation gives an approximation. It is only as good as the relevance of the data used to estimate the standard deviation in predicting the standard deviation of the sample to be collected.

The p-value is what it is. However, with a small sample size it is possible that my results cannot be replicated. That is, the collected data do not accurately represent the population and therefore the conclusions (while correct for the sample) are irrelevant to the population at large. This can also happen with a large sample, is it simply less likely. So, are the conclusions important enough to warrant publication, and what is the cost of having made an error? Likewise, readers should not assume that "it is published therefore true." All publications are the best guess of the scientist based on the data they were able to gather. It got past reviewers and editors, but that is not a guarantee of validity.

If you are very lucky, you should be able to get the right answer to any research problem with a sample size of 2. I don't recommend this, as I am unwilling to assume that anyone has that degree of luck no matter how many times they have won the lottery jackpot.

So what does your data tell you about your situation? If you had known this beforehand, would you do anything differently? Might this outcome (if true) help other people in a position similar to yours? What will happen if your results are not applicable to a broader population? Can you help readers appreciate this?

Are there other ways to analyze your data? Different ways to answer the same question (e.g. a parametric method versus a non-parametric method)? Do all methods give you the same qualitative result? How are your conclusions influenced by your method of analysis? Given that you don't know which analysis result is correct (and assuming that there are different outcomes), how do you adjust your answers to the questions in the preceding paragraph?

I would like to add some comments about sample size computations (but I must before say that if I agree with Timothy's comments, I don't think his answer addresses your concern). Sample size calculation performed **before** running a clinical experiment is not a statistical necessity but an ethical one : you must give some insurance that your trial will give an answer, positive or negative, with the number of patients involved. Your trial will be informative and patients involved with an acceptable benefit/risk ratio (benefit for the society, risk for the patient of course). Sample size computations performed **after** a trial is useless, or nearly so. Then the useful information is given by power computation. In fact i)true difference between placebo/new treatment (if any), ii) the alpha risk and/or related p-value, iii) the size(s) of the sample(s) and iv) the power are related, knowing three of them you are able to assess the fourth.

The dichotomy "positive or negative" is not quite right. It is too easy to interpret this such that a failure to reject the null hypothesis is equivalent to showing that the null hypothesis is true. It might be better to say "positive or indeterminate." You could have a three way score "positive, indeterminate, or negative." In this case you might consider negative as either proving the null hypothesis or as a statement to the effect that a sample size sufficient to reject the null hypothesis is so large that we will never be able to achieve with current methodologies. Proving the null hypothesis is a more difficult task, and rightly so. It is too easy to get false negatives using small sample sizes and poor experimental designs.

There is at least one example of proving the null-hypothesis: my manuscript on the effects of voltage on Asian citrus psyllid behavior. It is not that voltage has no effect, rather that it has no effect within the currently accepted range of recording given the typical set-up of this type of equipment. That is a useful conclusion because some labs use 20mV while others use 100mV or more. These voltages are all below the detection level of the insect. It helped that the experts at the time all agreed that there would certainly be a significant behavioral effect at 600mV.

Article Effect of Substrate Voltage on EPG Recordings of Ingestion a...

Even if sample size requirements may not be fulfilled, appropriate statistics can lead to proper further hypothesis development, but not conclusions. Of course, if the goal is to get a significant p-values, then there is no use. As mentioned earlier, a power calculation can be still done and tells you about how confident you can be about the differences observed. But, not all studies need to be clinically relevant from the very start. You can do a high throughput plasma or serum profiling to start developing hypotheses. But, these are molecular biology studies (with different but valid statistics) and not statistical & epidemiological ones. Goal and resources define the means, I would say.

Thank you all for your views and suggestion. I think Sample size calculation and justification is must if results are being presented with p-value and significance.

Every research has one objective with at least one outcome. Sample size can be calculated for hospital based-registry even the limited number of cases available.

Practice of sample size calculation is not common in hospital set-up however it must be.

Definitely for very rare cases of research, power can be back calculated.

I would like to recommend the practice of sample size calculation for every research as every research ends with p-value whether using Chi-square test or any other.

Coming from a clinical quality improvement background, my projects tend to be pilot or exploratory studies. I typically don't run power calculations beforehand. Still, I suggest when presenting results that a limitation might be the sample size.

I agree with Timothy A Ebert. Calculating the minimum sample size required to do a study is more an ethical issue than a statistical one. If you have a very small sample, you will most likely conclude that the null hypothesis of no difference is true. In this case you must also state the power of the study to justify your p value. The minimum accepted power for clinical studies is 80%. For small sample size the power will be quite small which will indicate the usefulness (or uselessness) of doing such a study. As Timothy, correctly pointed out, the sample size, the power, the p-value and the expected difference are all related. If you expect to see a large difference with the new intervention you can conduct it on a very small sample. But if you want to prove that there is no difference between two interventions then you will need a very large sample size to prove your point with adequate power.

Dear RG colleague,

Determining the sample sizes involve resource and statistical issues. Usually, researchers regard 100 participants as the minimum sample size when the population is large. However, In most studies the sample size is determined effectively by two factors: (1) the nature of data analysis proposed and (2) estimated response rate.

For example, if you plan to use a linear regression a sample size of 50+ 8K is required, where K is the number of predictors. Some researchers believes it is desirable to have at least 10 respondents for each item being tested in a factor analysis, Further, up to 300 responses is not unusual for Likert scale development according to other researchers.

Another method of calculating the required sample size is using the Power and Sample size program (www.power-analysis.com).

Regards,