I know it would be difficult to answer concretely but I would like to here from everyone's opinions and experience (I know this depends on the setting, but again a relative answer is also appreciated).
The thing is the usual number of replicates per scenario (or case) is 1000. By intuition or just from qqnorm in R we can see that 1000 is better than 200. But in my case computational complexity refrains me from 1000 or even 500.
So again would you think 200 is acceptable (I dare not say good)?
The question of how many simulations are required depends on the question at hand, because it also depends on the variability of the process you are interested. If you are interested on estimating a mean you might need only 30, if you want to estimate a maximum you might need 10000! So 200 might, or not, be enough! If you want to estimate more than just a single variable things get much more complex very fast. A potentially useful reference is Koehler, E.; Brown, E. & Haneuse, S. J.-P. A. On the assessment of monte carlo error in simulation-based statistical analyses The American Statistician, 2009, 63, 155-162 which presents an approach that solves exactly that question. The is an R package out there to implement the analysis described on the paper, but I believe it's no longer supported.
The answer is given by standard sampling theory. As long as the simulations are stochastically independent this works fine. Say you are interested in the value of some variable X from your simulations. Your N simulations allow you to construct a confidence interval for X. Increase N when you think that the interval is too wide ( for a given level of significance).
It depends on:
- the precision you want to obtain in your estimate, which also depends on
- the variability/frequency of the variable/event your are trying to estimate
Try run it a couple of times with reduced number of simulations and see by how much your estimates vary.
Consider construct a confidence interval for your estimate and see how much precision you obtain in your estimate. Based on a target precision/interval width, one can derive how many simulations you would need to run.
good luck
/N
Thank you all! These are indeed very helpful ideas, particularly about trying out smaller replication number. However for my problem, there has not been a rigorous justification for the asymptotic properties of the parameter estimates. But I will try to resort to bootstrap.
The question of how many simulations are required depends on the question at hand, because it also depends on the variability of the process you are interested. If you are interested on estimating a mean you might need only 30, if you want to estimate a maximum you might need 10000! So 200 might, or not, be enough! If you want to estimate more than just a single variable things get much more complex very fast. A potentially useful reference is Koehler, E.; Brown, E. & Haneuse, S. J.-P. A. On the assessment of monte carlo error in simulation-based statistical analyses The American Statistician, 2009, 63, 155-162 which presents an approach that solves exactly that question. The is an R package out there to implement the analysis described on the paper, but I believe it's no longer supported.
The number of replicate simulations you do should depends on how many are needed to provide a clear answer to the question at hand. I think about it this way: if I repeat the study from start to finish many times, I should get the same answer the great majority of times. If the answer to the research question (whether quantitative or qualitative) varies a lot among study replicates, I have not used enough SIMULATION reps in the study. Therefore, I think it is worthwhile to design a simulation study so that you can repeat the whole process many times to be sure the answer to your question is not driven solely by the intracicies of a single simulated data set.
Some times a question will be answered clearly with 10 simulation reps. Sometimes tens of thousands could be necessary.
You need to quantify how much better 200 is than 100, and based on that quantification you can decide when the diminishing returns isn't worth it. Supposing your goal is to estimate some value, the quantification could be the change in width of the confidence interval. This would mean choosing the number of replicates that gives you the confidence interval width that you want.
Exactly how to do that depends on the the analysis you're doing, but basically you solve the (distribution-specific) confidence interval equation so that it expresses the number as a function of interval width. For intervals around common statistics (such as mean or variance) on common distributions (such as normal, gamma, Poisson, Boltzmann, etc) these functions are well known, so all you need to do is confirm which one to use.
If you're using qqnorm I take that to mean you expect your data to fit a normal distribution. I would prefer a goodness-of-fit test that gives a numerical result (such as Pearson's Chi-squared test), because then you're not dependent on a subjective visual test and you can choose a target value for the goodness of fit. I'm not sure how such a target value would affect your sample size. A quick search gives me reason to think some tests might not work well with large sample sizes, but if your sample size is limited anyway that might not be a problem.
If you're doing something sufficiently original and too computationally demanding to set n high enough to get the results you want, then I think it's acceptable to set n according the available computing power and call your analysis preliminary - the aim would be to show that it would be worth funding a followup with more computing power (or more specialized software, or whatever will increase the ceiling on your replicate count).
The real answer to the question asked is "it depends on the study"; If you want to know how many is enough for your particular study, I think you'll have to ask a more specific question.
All are valuable answers. I talked with my supervisor and we just go with what we can afford. Many thanks
For an initial study 200 replicates may be adequate. However if you want to publish the research at least 1000 replicates are needed. This is because you want narrow confidence bands around your type I error and parameters.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis