I am afraid I have to point out that it would have been better to have done that in advance, and to have collected the appropriate sample size.

So-called "post-hoc power analysis" makes little sense in most cases because you have to assume that your estimated effect size is equal to the population effect size, which is very unlikely. [Unless, of course, you have a huge sample, in which case you do not need to prove sufficient power.]

See:

Article The Abuse of Power: The Pervasive Fallacy of Power Calculati...

Article Brief Report: Post Hoc Power, Observed Power, A Priori Power...

I am afraid I have to point out that it would have been better to have done that in advance, and to have collected the appropriate sample size.

So-called "post-hoc power analysis" makes little sense in most cases because you have to assume that your estimated effect size is equal to the population effect size, which is very unlikely. [Unless, of course, you have a huge sample, in which case you do not need to prove sufficient power.]

See:

Article The Abuse of Power: The Pervasive Fallacy of Power Calculati...

Article Brief Report: Post Hoc Power, Observed Power, A Priori Power...

Heiko Breitsohl Thanks!

I have submitted a paper to a journal reporting are results obtained from 600 sample size for a target population of 2.76 million. I used formula of Krjecie & Morgan (1970) for computing sample size. . One of the reviewers has asked me to include a power analysis on my data to work out how big a sample size I would need to adequately power a study to to prove the adequacy of sample size. Basically he wants me to "prove" my study was not under powered.

Well, I see two options, then:

• If you actually conducted an a-priori power analysis before collecting data, report that analysis.
• If you did not (which I assume is true), you may have to educate your reviewer that what they are asking for will not be very helpful to anyone.

Unless you know the population effect size (which would render your study useless), there is no way of "proving" adequate power.

Here is another way of looking at this: Your study was powered just enough to detect an effect size that produced a p-value just below your alpha level. That is all you are able to say after-the-fact.

For this very reason, it is so important to do power analysis beforehand: Once you have run your hypothesis tests, you can never go back. You have to know the necessary sample size in advance, so you can make sure you have enough power. [Collecting more data does not help either.]

I have been in the same situation. Unfortunately, many people (including reviewers) have misconceptions about power analysis, and it becomes our job to tell them that.

I would hope that, if you explain the issue politely, citing the appropriate sources, you will be fine. [The very unattractive alternative would be to go along with the reviewer and perpetuate a misconception.]

Hello Bivinia,

If you used a formula to compute the sample size **before** you did the study, then report the results of that power analysis. There's nothing wrong with that. The problem lies in conducting the power analysis after the data are collected, as pointed out above. Make sure that it is clear that the power analysis was conducted on prior data, not the data you collected. You can't use power analysis from the data you collected to "justify" your sample sample size.

You can also carry out a "before the fact" (a priori) power analysis with the data you collected, as a guide to future work on the same measurements. It won't justify your choice of sample size, but it is useful as a guide to future work, if presented in that way.

Hope that helps,

David S

a-priori: This effort may be useful to the scientist conducting the research. As a reviewer, if you include it I will ask how you justified your estimates of effect size and variability that you used in the calculation. If you used published data, then I would like to know how you decided that the results of some other study (likely different country and different time) was a good estimate of equivalent measures in the new study. If the estimates are accurate, then a simulation might have been an easier approach. If the estimates are not accurate then the sample size calculation was pointless to begin with. Unfortunately, this additional material might significantly distract from the story that you want to tell.

post-hoc: see Heiko's answers.

Sample size estimates are only as good as the values that you enter into the calculator. They are probably based on other studies that have sample size issues themselves. Even if a miracle happens and the values that you enter are correct, the sample size estimate is a lower bound. It may be grossly insufficient if your data has many missing values. It may also be insufficient if your analysis depends on derived factors.

Prevalence of a disease is an example of a derived factor. I want to know the prevalence of some condition in the population of two cities. In each city I will test all the people at each of ten grocery stores. I can then run the analysis in two ways:

1) Calculate a prevalence for each city and 95% confidence interval. I'll ignore the fact that the data were gathered in different locations.

2) Calculate the prevalence at each grocery store, and then use the ten values in a t-test for city differences.

I may get different answers depending on the approach, and I will need a larger sample size for approach #2.

Hi Bivinia,

If you did power analysis before conducting the experiment, congratulations. Here is a quote

Fisher: To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

If you did a prior power analysis, I'd report it. Here are some more quotes.

Voltaire: “The best is the enemy of the good.”

Confucius: "Better a diamond with a flaw than a pebble without."

Shakespeare: “Striving to better, oft we mar what's well.”

If you did not do power analysis before the study, then all that can be done is Fisher's post-mortem.

Good luck with your reviewer,

David S

David C Schneider thanks for your suggestion.

Timothy A Ebert & Heiko Breitsohl thanks for the reply.

can i calculate the effect size based on my pilot data?? and if i am using the effect size calculated from pilot data for the post hoc power analysis what can be possible issues? I am not using the eefct size of the observed result, but i am using using effect size calculated from R2 value where

effect size f2 = R^2/(1-R^2)

I have gor effect size as 0.298, alpha as 0.05, , with sample size 600, with a power > 0.99.

Also, how to report a the results?