For sample size needs for inference for continuous data or for proportions, William Cochran - Cochran, W.G(1977), Sampling Techniques, 3rd ed., John Wiley & Sons -
noted that we need estimates of population variances, and described four ways of obtaining this (for each variable of interest) in section 4.7 of that book. Among these four methods were a pilot study and other preliminary information, perhaps as little as the range and general shape of a distribution, which he attributed to Deming, W.E. (1956), Sample Design on Business Research, John Wiley and Sons, New York.
For a sample size adequate for reasonable inference, one might stretch resources too far such that greater nonsampling error (e.g., measurement error) may occur and then inflate the estimates of population variance. So that could be a limiting factor. (Note also that the formulations and estimates for sampling typically are only designed for sampling error (generally variance due to sampling), even though bias may be of interest, and nonsampling error can often be more important.)
There are other books with information on estimating sample size needs, such as
Blair, E. and Blair, J(2015), Applied Survey Sampling, Sage Publications. (Note that the example there which involves power is for proportions, and instead of estimating sigma, uses the worst case p=q=0.5.)
If you have a suggestion for obtaining information to estimate sample size needs, and/or an example to share, especially one you have used, I would appreciate your sharing that here. - Thank you.
Though stratification is generally very useful, both for design-based and for model-based sampling and inference, starting with the basics, consider one set (group/stratum/category) of data: In Cochran, W.G(1977), Sampling Techniques, 3rd ed., John Wiley & Sons,
on page 77, he shows a 'formula' for estimating sample size needs for simple random sampling.
In
https://www.researchgate.net/publication/261947825_Projected_Variance_for_the_Model-based_Classical_Ratio_Estimator_Estimating_Sample_Size_Requirements,
I have a very similar format for estimating sample size needs for a similar model-based situation. See page 2888.
In both cases, a sigma needs to be estimated before sample size needs may be estimated. For the Official Statistics application in the latter case above, highly skewed establishment surveys (frequent sample surveys, and less frequent census surveys) generally provide the right environment to obtain good estimates of sigma. But for new surveys, there may be some time before such useful data are available. In most other situations, such as one-time surveys, without a pilot study, there may not be much information to use to 'guess' sigma (for each key response variable!) without a pilot study. So it would appear that Official Statistics are in a particularly good position for estimating sample size needs whenever they are re-evaluated.
Another "easy" case would be estimating proportions under simple random sampling, IF you do not mind overestimating sample size needs by "plugging in" p=q=0.5, which is commonly found in "sample size calculators." But you can obtain better results if you can 'guess' p (and do not forget the finite population correction (fpc) factor if appropriate).
But what about other cases?
Conference Paper Projected Variance for the Model-based Classical Ratio Estim...
I have an underlying distribution.
From this distribution I generate a population of values.
From this population I withdraw a sample.
From the sample I calculate a mean and standard deviation/variance.
I then use these values in various statistical analyses.
The problem is that there is error in my estimate of the SD, and I have no idea what that error is. A part of the error is generated in the process of sampling. However, it may be that the way that I have conducted the experiment has also contributed to the error. It is hard to take biological measurements without influencing the biological system being studied.
Run the entire experiment with 5 replicates. Run it again and again, say 10 times. Then redo the experiment with 10 replicates, and again do this 10 times. Then redo the experiment with 20, 40, and 80 replicates and as usual repeat 10 times for each replicate level. Now you have a mean value for your SD, and an SD for your SD. If you graph this, you should find that an increasing sample size results in decreasing variability in the SD calculated for the sample. A log plot will give a straight line, and you can calculate an exact value for any desired level of accuracy.
This assumes that the sample mean and variance will converge to the population mean and variance as the sample size approaches the total population size. The method works best if the mean is achievable. If you take a sample of 100, and calculate a percentage, then there will be issues if the population value is 14.375% because that value is unachievable given your sample size. Likewise if the mean is 2.376 children per family and families are unwilling to chop the children into bits. There may be other conditions or limits that I have not thought of.
I used the method in my Dissertation, and in the 1998 paper in Journal of Economic Entomology on randomization tests which was from the same research. Start with heading "Number of randomizations" on page 764. The task was to justify the number of randomizations needed to insure accuracy in a randomization test.
Article Randomization Tests: Example Using Morphological Differences...
@Timothy,
Whatever you suggested is quite straight and correct and a justification for sampling.and statics, as well. However, the important concern of the post remains unresolved. As, your strategy could "stretch resources too far.
I agree, that my approach will require a large number of samples. If it turns out that the ideal sample size is 5, then you will have wasted an great deal of effort. On the other hand, people who wish to do similar work can now use your data to justify using a smaller sample size.
I used Excel to generate 36 random numbers using a uniform random number generator. (=rand()). I calculated the standard deviation for sets of six values. My standard deviation ranged from 0.27 to 0.36. Which value should I use in the calculation for sample size? If you just take one small sample as part of a pilot study, your results could be way off. So the real question is how badly do you want an exact calculation of sample size and what level of risk are you willing to assume by doing little more than guessing?
That last question may not be exactly fair. Some systems are very well studied and it is possible to have reasonable confidence that the guesses are accurate. So part of the answer would be dependent on the field of study. Certainly, such a plan as I originally outlined would not be advisable for human clinical trials. The best one could do in this case is to use a previously collected data set and try resampling that. The problem there is that the results will converge to the sample mean and SD, which may be different than the population mean and SD. In such a case, I think that you cannot reliably estimate a sample size larger than the sample size used to conduct the test.
My favorite method is to do bootstrap samples, take the average of the 16th percentile, 50th percentile and 68th percentile and infer the standard deviation from them. However, this method requires that the data is normally distributed, or at least that there is no peculiar issues ( multimodal distributions, strongly skewed distributions, etc.).
My opinion is that sample size should be greater than the minimum required for the underlying and hypothetical distribution that we (silently) accept for the process. Of course we have the 'curse of N>30', due to Normal Distribution, but it is not a 'cloth to fit for all' rule.
- Badges
- Science method
- Similar topics
- Mathematical Sciences
- Statistics
- Sampling Design