When I execute an experiment, I measure execution time (minutes or hours), and since there is an environmental noise, the execution time is different. It is expected that the execution time complies with the normal distribution. I am wondering, is it defined or what is the number of repetitions that will be counted enough for statistical analysis and conclusion? 10, 50, 100, 1000?
You need to do a "Power Analysis", It allows you to determine the sample size required to detect an effect of a given size with a given degree of confidence. In R, for example, you can use the package "pwr". You can find some examples in the book "R in Action" by Robert Kabacoff
It depends on the precision you required. Generally, 3-4 number of repetitions are enough.
to be sure that your experiments are valid, make "online"testing. Observe the standard deviation of the outcomes of the experiments.
If somehow appears an outlier you may observe why it appears.
You may look here: "Statistical rules of thumb (STRUTS)":
www.nrcse.washington.edu/research/struts/chapter2.pdf
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics