Hello Alex,

A priori sample size evaluation is all about trying to determine how many cases are needed in order to assure some target level of statistical power to detect some target effect size (or larger), when evaluated at a target Type I error risk (alpha level). You're free to set power and alpha however you see fit (and can justify, given the consequences of being wrong in either a false positive--Type I--or false negative--Type II--way).

The key unknown is, what magnitude of effect is of interest to you? You say that you'd like to know if a treatment is effective with prob = .50 (or better). But, compared to what expectation of improvement in the absence of treatment: prob = .00, prob = .33, prob = .40, or something else? If the expectation of improvement in the absence of treatment is zero, you won't need many cases at all to suggest that the amount is far higher than that.

If, on the other hand, the no-treatment expectation of improvement is, say, .40, you'd need far more cases. [In G*Power, Exact tests / Proportion-difference from a constant (binomial test, one sample case). 90% power to detect proportion difference of |.4 - .5| = .1 at alpha = .05 requires N >= 214, using one-tail test.]

The other way to think about the problem is, how precisely would you like to estimate the proportion of cases that do show improvement from the treatment? If you'd like to estimate the true proportion to within +/- .05, with 95% confidence, the worst case scenario (for simple random sampling from the population of interest) would be N ~ 384 (assuming population proportion is .50, which yields maximum variance for a dichotomous variable). As the population proportion moves higher or lower than .50, or as your desired precision is larger than +/- .05, the requisite sample size will decrease. For example, for precision of +/- .10, N ~ 96.

Good luck with your work.

First of I really appreciate the through and educating answer. I'm not sure though I explained my intent correctly - the question at hand is a procedure performed. The clinically justified procedure would be one that the benefit expected would happen in 50% of procedures performed (ie - 1 out of every 2 procedures brings benefit). Any lower percentage would thus deem such procedure as unnecessary (ie- in only 1 out of 10 procedures you get the expected benefit). How would you calculate such a sample size?

Alex,

Method 1:

You could estimate the proportion for your sample, construct a 95% confidence interval, and as long as the lower bound of the CI did not fall below .50, you would say the treatment met your criterion. The larger the sample size, the closer an observed proportion of successful outcome cases can be to .50 order to meet that condition. The general equation is p +/- 1.96*SE(p) , where:

p = estimated proportion of successful cases from sample, = successes / cases (N)

1.96 is constant (for approximate 95% CI)

SE(p) = sqrt[p*(1 - p) / N]

N is sample size used

With N = 25 cases, the observed success proportion would have to be close to .70 for the CI not to fall below .50.

With 50 cases, the observed proportion would have to be about .65 or higher.

For 100 cases, observed proportion of about .60 or higher would be sufficient.

For 400 cases, observed proportion of about .55 or higher would be sufficient.

1000 cases would allow the observed proportion to be as low as .531

Note the diminishing returns with respect to improved precision as N gets large.

Method 2:

Test the hypothesis that: Ho: p >= .50 vs. the alternative that p < .50. A binomial test will suffice, but you'll still have to determine, by how much must the observed proportion fall below .50 before you are willing to characterize that as a noteworthy difference. The smaller this difference (the effect size), the bigger the sample size will need to be in order to have good statistical power to detect such a difference (should it actually exist). As outlined in the first post, G*Power Exact tests / Proportion difference from a constant, would be used to help identify the requisite N.

In this instance, using too small a sample size will actually bias your result towards retaining the null hypothesis (whether it's true or not), due to having too little statistical power.

HTH

Many thanks!