If a biological experiment is set up correctly, and the data are statistically significant, but not reproducible, what should be done?
Any biological process - even at the level of a single cell - not to mention a whole organism, population or ecosystem - is a superposition of many periodic and quasi-periodic processes, with periods from nanoseconds to several years. Any result of the experiment is just a random point on the graph of this superposition. Therefore, the reproducibility of biological data is more of a miracle than a regularity. A related (but not identical) issue is discussed, in particular, in my article
Article Чудо косвенного измерения
(in Russian)So: if the data is received correctly and statistically significant and at the same time reproduced - rejoice and thank God. If, other things being equal, they are not reproduced - build and test hypotheses about the reason for this, but in no case (!!!) neglect the data.
There should be something wrong, otherwise why we do statistical analysis? Regards.
Dear .Medhat Elsahookie, by modern international standards, any experimental and observational data must be processed statistically. Only in this way is the sufficiency of the sample established, the reliability of differences between the control and experimental groups, etc. If statistical processing is not carried out or carried out by an inadequate method, such an experiment cannot be accepted even as a pupil's educational laboratory work, not to mention a serious study. Did you not know this until now?
Report that the results were not reproducible.
That a result is "statistically significant" is neither a proof nor a guarantee that it is correct or that it will be reproducible. It is only a "first line of defense" against being too over-confident in interpreting sample estimates.
Note that even when H0 is true, you have a 1:20 chance to get a p-value less than 0.05 what is typically considered significant. The fact that this can happen also implies that it, sooner or later, will happen.
Statistical tests are like a crude sieve through which we sieve gravel and diamonds; these diamonds are typically slightly smaller than the gravel, but they are dirty and we usually cannot identify them easily by looking at the pile of gravel. The hope is that the concentration of diamonds will be increased in the pile passing the sieve. A stone we find that passed through can still be a piece of gravel that happened to be small enough. If the gravel originally contains no diamonds at all, there will still be some smaller gravel stones passing the sieve, and there won't be a single diamond among them.
Remember too this statement by Fisher, with particular attention to the final sentence (emphasis added):
If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent. point), or one in a hundred (the 1 per cent. point). Personally, the writer prefers to set a low standard of significance at the 5 per cent. point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.
— Sir Ronald Aylmer Fisher
'The Arrangement of Field Experiments', The Journal of the Ministry of Agriculture, 1926, 33, 504.
- https://todayinsci.com/F/Fisher_Ronald/FisherRonald-Quotations.htm
- Badges
- Science topic
- Similar topics
- Filing