In the footnote of a paper published in the Journal of Economic Behavior & Organization in 2007, the authors Kuang et al. write the following two sentences: "Given the idiosyncrasy of random matching, such comparisons would be more meaningful with a larger sample. However, the fact that we get significant differences in spite of the relatively small samples provides further support for our results."
I have problems with the second sentence. Finding significant differences when sample size is small (possibly smaller as a sample size calculation would suggest) does not necessarily provide support, i.e. strengthen the significant findings, does it?
Hendrik -
"However, the fact that we get significant differences in spite of the relatively small samples provides further support for our results."
Translation: If a p-value is 'small' when a sample size is 'small' then the effect size must be 'large.'
The problem lies in the historically misleading word, "significance."
Please note:
Press release for the American Statistical Association:
http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
My letter in The American Statistician:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
Cheers - Jim
Article Practical Interpretation of Hypothesis Tests - letter to the...
I think that ,in this instance, you need to be thinking of the power of the test, rather than the size of the test.
For a given "real difference", and for the same size (significance level) of test, the probability of deciding that the sample difference is "real" goes up as the sample size increases. This applies for any case other than the null hypothesis. It may be that those authors are thinking of this sort of argument.
Potentially, a better approach would have been to provide a confidence region for the size of the difference. I think that this would inherently avoid the "sample size versus size of difference detected" confusion.
I agree with you that the second sentence is statistical nonsense. However, if for example, we have 2 states such as patient alive and patient dead, it doesn't matter what size the sample size is, as the state is clear and unambiguous. And the smaller the sample size then the less dead patients... As an aside with 3 measurements (as seem to be favored by pharma and the FDA), throw away the 2 outliers and one is left with a perfect measurement.
Short answer: As stated, the quote is statistical nonsense. The counter to the statement is to take it to extremes. Would the authors really argue that a sample size of two (per treatment) would provide the strongest evidence of a significant difference between two means should such a difference happen to be significant? I would hope not, but this is where the argument leads.
The observation that huge sample sizes result in a statistically significant differences that have no practical value is valid (as other comments suggest, this may be what was intended). In the quote, it is unclear if the sample size is typically 300 and the authors had only 175 replicates, or if the authors are trying to suggest that a sample size of 3 is better than having more replicates.
The authors seem to think that if sample size calculation requests m samples for p% power and they were able to get a stst sig diff with n
Hendrik -
"However, the fact that we get significant differences in spite of the relatively small samples provides further support for our results."
Translation: If a p-value is 'small' when a sample size is 'small' then the effect size must be 'large.'
The problem lies in the historically misleading word, "significance."
Please note:
Press release for the American Statistical Association:
http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
My letter in The American Statistician:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
Cheers - Jim
Article Practical Interpretation of Hypothesis Tests - letter to the...
I am working with some small data sets right now. I'm looking at student performance in different classes. For some of my classes, there are 30 students. In other classes there are 3,000. Even after looking at the results and the power of the tests, everything looks good and like certain new classes are better than the older classes. However, I don't believe it. The reason is I watch sports. I've seen hundreds of baseball players that look really good over a short period of time, then fade away to their usual levels.
I watch a lot of baseball. If the small sample sizes of a couple of my Detroit Tigers were actually "true", Then the Detroit Tigers have a pair of players with a batting average of over 400 right now. But, since anyone can get lucky for a short run, those averages have dropped to the "average" for each player.
Furthermore, if only large sample sizes can provide significant results, then your system probably isn't very meaningful. For example, I can prove that drinking coffee and shaving all the hair off of your body give "statistically significant" but practically irrelevant results for "losing weight". If I give you a large amount of coffee and weight you before and after using the bathroom, the water weight you lose and/or the mass of hair before and after shaving will be "significant" with enough participants. However, drinking 4 liters of coffee so you can excrete 4.05liters of liquid or shaving your body to eliminate 100g of mass is not practical.
For something to be meaningful, the sample size needs to be large enough to prove the results are not a fluke and the differences between groups needs to meaningful. "I lost 'significant' weight in only 5 minutes! Ask me how!" makes for a good headline. But, like most clickbait, it's pointless and stupid.
Some of the things you wrote make me think of the "health news" I hear on my local news stations. You know, "Eggs are good for you" then "Eggs are bad for you"...
Well, Andrew, yes, if a sample size is too small, it may not be very "representative," and that is an additional issue. You have to have 'enough' data to consider before doing anything. That is prerequisite.
Analogously, if you determine sample size needs for a population or stratum by knowing the standard deviation, and 'calculating' the sample size that may give you a desired estimated standard error of a mean or total, there has to first have been enough data, or another source, another survey or pilot study, to have a good 'guess' for the population standard deviation. It is a given that you need that much information to start, just as in the case of an hypothesis test, you have to have reasonable data first to go forward there as well.
That is usually a given before we consider power, etc, though perhaps, as I think you are suggesting, this is not given enough attention.
Cheers - Jim
- Badges
- Science topic