I have two data sets, one size is 123, I called it A, and the other size is 28, I called it B. and I also have a variable that I think it has an association with clinical outcome. I observe the association in the larger data set, but p-value > 0.05 in the smaller data set.
I randomly select the same size data set from A 100 times, and calculate association with clinical outcome, the results show that p-value > 0.05 occurred more than 50 times.
So my question is, can I say that this variable has no association with clinical outcomes in B data sets? Does the sample size of the B data set make people think that this conclusion is unreliable?
Anxiously waiting for answers, many thanks!
Ke Liu ,
I think this is the typical behavior of the p-value when we can not reject the null hypothesis. If you are working with continuous variables, the distribution function of the p-value is uniform (0, 1) under the null hypothesis. Then you can obtain p-values in equal length subintervals of (0, 1) with the same probability. I think this is the reason you are getting values with such variability.
Hi, Ke,
Indeed, the Type I error rate is affected by N. Given sufficiently large samples, trivial R-squared values such as 1% can become "highly statistically significant"--and require many zeros after the decimal to express. It is, in part, a question of statistical power and confidence intervals. Here are two articles, one on each of these issues, for researchers seeking models which specifically maximize predictive accuracy for the sample.
- https://odajournal.com/2020/02/08/statistical-power-analysis-in-oda-cta-and-novometrics-invited/
- https://odajournal.com/2020/06/11/generating-novometric-confidence-intervals-in-r-bootstrap-analyses-to-compare-model-and-chance-ess/
Intertwined issues are validity of the experimental and statistical methodologies, assumptions underlying the statistical test, and accuracy of model predictions.
- https://odajournal.com/2013/09/20/maximizing-the-accuracy-of-multiple-regression-models-using-unioda-regression-away-from-the-mean/
- https://odajournal.com/2019/04/14/regression-vs-novometric-analysis-predicting-income-based-on-education/
- https://odajournal.com/2016/09/19/novometric-analysis-with-ordered-class-variables-the-optimal-alternative-to-linear-regression-analysis/
- https://odajournal.com/2020/06/28/what-is-novometric-data-analysis/
The correlation probably has excessively low statistical power in the smaller dataset, making it questionable to rely on for hypothesis testing.
Saying no association between a variable and a clinical outcome based on data set B is misleading, though it is not possible for anyone to prove this conclusion is unreliable. The nice work that you have done (selecting the same size data 100 times from data set A and observing p value > 0.05 more than 50 times) itself shows that there should be a minimum number of data (sample size) is needed to prove (disprove) an association in a reliable manner. In fact the association observed based on data set A may itself is unreliable, unless it achieved the minimum required number. To avoid this type of problem it is essential to estimate the sample size required for the study to prove an association.
Paul Yarnold Yes, I have encountered this situation many times! Sometimes there is no obvious difference between the two groups, but the p-value is very small because of the large sample size. I am really not good at statistics, thank you very much for the recommended articles！
Kandasamy Ravichandran “ The nice work that you have done (selecting the same size data 100 times from data set A and observing p-value > 0.05 more than 50 times) itself shows that there should be a minimum number of data (sample size) is needed to prove (disprove) an association in a reliable manner ”
Sorry, I think I didn’t make my question clear, "I selected same size data" means that I selected 28 samples from dataset A, so I think that if I always get p-value < 0.05 in 100 results, then I can say that p-value>0.05 in data set B is not due to the small sample size.
And if I can not meet the minimum number of sample size, how can I make my results reliable? hope to get your reply!
Many thanks!
Still what you have tried is a good job. So ultimately you have analysed 101 data set of size 28 and more than 50 times the p value is > 0.05. On the other hand less than 50 times you have significant association. Whatever the result you got (either p > 0.05 or < 0.05) without adequate sample size is due to chance only. Thus, if you don't have minimum sample size there is no way to claim your results are reliable. It should be mentioned as limitation of the study.
Ke Liu
I wanted you to know that in my view, your self-appraisal of your statistical skills as "really not good" understates your innate understanding of statistical issues. As Dr. Ravichandran noted, you did good work investigating your issue. In my experience, the simulation you conducted is high-level work which is rarely conducted even by seasoned investigators, and it indicates that you have an intuitive grasp of important issues. This is a part of your cognitive constitution, and in my view it should be nourished.
I hope the articles I sent serve to feed your inner statistician, I'd love to read what you learn on your own, and/or working with like-minded colleagues...
Best wishes and regards!
Paul
If you use the correct methods the both p-values are reliable. If your sample size is less than 30, you shall use non-parametric tests.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size