Here is an R function pwr.boot as an R file.

# Replaced below with an updated R file

The PowerBoot function described above was used in my new eLife paper, which shows that rising CO2 shifts plant quality worldwide toward lower mineral concentrations and dilutes nutrients with extra starch & sugars:

http://elifesciences.org/content/3/e02245

Francois, thank you for the response. I meant that the true mean is 5. So the expected effect size is 5. The question is updated to clarify this and thanks for pointing this out.

Type II error means retaining a false H0. So the probability of Type II error then depends on the effect size. Are you suggesting calculating it without using the effect size?

I was thinking more along the lines of comparing the length of CI to the effect size. For example, if the CI obtained via bootstrapping has length = 30 and is centered at 10, but we expect the effect size to be 5, then the CI contains 0 and we retain H0 (see the attached Figure). What is the probability of making Type II error in this case?

Hi Irakli,

In order to do that, you must simulate your H1 first. I recently had the same issue and I will describe what I did. You have a pool of values that represent what you measured, that is your H0, meaning that it represents no change. If for example you expect an (absolute) effect size of 5, then you just shift your H0 to the right or left (by adding or subtracting 5 to every value). Then you have a H0 with your values and a H1 with your simulated situation that represents a shift of magnitude 5. Then you must generate some variability (maybe through bootstrap, I used Monte Carlo simulations), as you take something like 10000 samples of size n from your H0 pool and 10000 samples of size n from your H1 pool and compare every pair with the statistical test of interest (wilcoxon or something). Then you have compiled the result from 10000 statistical tests, where some rejected H0 and some did not (according to a pre-established alpha). The proportion of tests that were not able to reject H0 is your beta, or the probability (out of 10000) of failing to detect an effect size of 5. Then with 1-beta you have your power. The more tests you run, the greater the precision of your estimate of power, and the greater the time you have to spend looking at a computer screen...

There is not much literature on this, but a quick search on google scholar will lead you to it. Search for Thomas and Juanes (1996). Hope this helped. Use R.

Hi Miguel,

Thank you for your reply. You are suggesting to shift the original sample by the effect size in question to generate an alternative H1 sample, and then use bootstrapping to generate 10,000 samples from the H1 pool .

The statistical power of the test - whether parametric or not - is the probability of rejecting false H0. We already have H0 and CI, which we generated via bootstrapping from the H0 pool. Would it be simpler just to check what proportion of those 10,000 samples fall outside of the CI? This proportion then would be the statistical power of the test. The proportion that falls into the CI would be beta.

I generated a simple figure to illustrate what I mean. Please, see the attached figure. What do you think?

Hi Irakli,

I think that is equally correct (though I'm not a statistician). I was suggesting using a non-parametric statistical test to decide whether to reject or not H0, you are using confidence intervals to decide when H0 is rejected. If that is the method you are interested in using to make a decision regarding H0, then that is the one you must use to calculate power. The principle is the same. I was suggesting running 10,000 tests and then the proportion of tests rejecting H0 for an arbitrary alpha would be the power of the test, which is the same thing you are doing here.

Miguel, your suggestion about shifting the sample was very helpful as all this bootstrapping is new to me. I also followed your advise on doing it in R. Turns out the R language is easy to learn - the code to estimate the statistical power of a bootstrapped test takes just a few lines. Thank you for your help!

Glad it worked! No problem!

Miguel, looks like this question had over 500 views. So I'd assume other researchers have interest in calculating power for nonparametric test (bootstrapping). Based on our discussion, I wrote a simple R function, pwr.boot(x). The code for it is short

Here is an R function pwr.boot as an R file.

# Replaced below with an updated R file

The PowerBoot function described above was used in my new eLife paper, which shows that rising CO2 shifts plant quality worldwide toward lower mineral concentrations and dilutes nutrients with extra starch & sugars:

http://elifesciences.org/content/3/e02245

Thanks for sharing. That will be certainly useful for many people.

##Minor updates to pwr.boot R function

https://github.com/loladze/co2

A related question is how to calculate the p-value of a non-parametric test? Say, H0 is that the mean is 0. And we want to use bootstrapping. I am attaching a schematic image of the procedure (similar to the power test). Any thoughts?