First, you need to make a distinction between "nonparametric", "distribution-free" and "exact" methods. A nonparametric method estimates and tests hypotheses about medians, percentiles or other quantities that are not parameters of distribution. A "distribution-free" test does not assume that your data or any statistics computed from your data come from any known distribution. An exact test is a test whose inferences (i.e. p-values or confidence intervals) are equally accurate at all possible sample sizes. A bootstrapping test is always an exact test and it is always distribution free, but it may or may not be nonparametric. For example, you could construct a boostrapping test to compare the means of two samples, which would still be parametric in that the mean is a parameter of the normal distribution and other distributions (T distribution, Poisson, etc.).

So, increasing the sample size will not affect the legitmacy of your p-values or your confidence intervals, because you are using an exact test. However, a larger sample size will give you more information about the true value(s) of the unknown population parameters or percentiles ... presuming that your samples are all unbiased.

The bigger the sample size, the more accurate the results

If you are playing with nonparametric statistics, you do not need to think about much about your sample size. Anyhow, you can get reasonably good results if you use the sample size over 30. I think that it helps you.

Thank you, Luis and Rasiah! I expected also that increasing the sample size is generally better. I wonder below what sample size would the results of nonparametric bootstrapping tend to be come too unreliable to be useful?

First, you need to make a distinction between "nonparametric", "distribution-free" and "exact" methods. A nonparametric method estimates and tests hypotheses about medians, percentiles or other quantities that are not parameters of distribution. A "distribution-free" test does not assume that your data or any statistics computed from your data come from any known distribution. An exact test is a test whose inferences (i.e. p-values or confidence intervals) are equally accurate at all possible sample sizes. A bootstrapping test is always an exact test and it is always distribution free, but it may or may not be nonparametric. For example, you could construct a boostrapping test to compare the means of two samples, which would still be parametric in that the mean is a parameter of the normal distribution and other distributions (T distribution, Poisson, etc.).

So, increasing the sample size will not affect the legitmacy of your p-values or your confidence intervals, because you are using an exact test. However, a larger sample size will give you more information about the true value(s) of the unknown population parameters or percentiles ... presuming that your samples are all unbiased.

Thank you, Jeff, this is a very useful description! To clarify, the procedure I am using is bootstrapping of a data set with N observations by randomly selecting N of them (thus some can be repeated, and some not used) multiple times. Each time a model is fitted to the data set, and parameter values for the model are compared. I am wondering does sample size N affect the usefulness of this procedure?

As I said before, because bootstrapping is an "exact" statistical test, the p-values or confidence intervals from your procedure will be equally valid for all sample sizes, N. Whether or not the test is useful is a completely different issue.

Are you using the correct bootstrapping test? Can your bootstrapping model answer the questions that you want to address with your experiment? Is your sample representative and unbiased? These are issues you need to address to make sure your test is useful.

Assuming that you have set up the correct test and your sample is both representative and unbiased, then you are basically asking about the relationship between power and sample size. Larger sample sizes will provide more information about your population and more statistical power to find significant p-values. When your sample size is large, then even small effects can be statistically significant. If your sample size is very small, then only the largest effects will be statistically significant. If you need to detect small, subtle differences between groups, then an experiment with a small sample size might not be useful.

Thank you, Jeff, for a detailed and useful reply!