Assume having conducted an experiment. You find a marginally significant interaction effect (10% significance) but, regretting not having performed a pre-experimental power analysis, you judge the power of the statistical test of the interaction as too low.
You decide on getting some more observations of some experimental groups, in order to increase the number of observations in this group and presumably the power of the respective statistical tests.
Regardless of how this changes the test-statistics, is this a reasonable approach? I somehow feel that it is not, but I am not yet able to recall the argument behind it.
Additionally, in case one would have proceeded as described above: Would it be better to analyse data and to present findings from both experiments (1) together, (2) separately. Or (3) to exclude the data from the second "additional" experiment completely?
Thanks in advance for replies.
Continuing to collect data because of a promising statistical "trend" would be considered p-hacking, because you are chasing the significant finding.
One option is to use what you have learned in this experiment to design and run a new experiment (presumably with a larger sample to have power to see if that interaction effect is "real").
There is an excellent recent article in PLOS (among many others) about these practices, and how they can contribute to the problem of replicability in science.
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106
Hi Hendrik,
Definitely you can improve your experiment by conducting additional experiments. In statistics this called 'Augmented Design Approach' . It is reasonable approach.
The Augment Design Approach constructs additional runs in a way that optimizes the overall your current design plan. You can add more runs (experiments) to accomplish the following goals:
Hi Hendrik,
I would agree to the second answer. Augementation is common and effective. However, you should make sure that the entire array (previous + new runs) makes sense as one common experimental array. This means, the combined array needs to be orthogonal. "cherry picking" does not work and might distort the combined array such that you get unwanted correlations between model parameters. An easy and robust way to augment arrays is the so called folding. In this case you simply repeat the entire array, but with opposite signs (+ becomes - and - becomes +). It dramatically cleans up confoundings between main effects and interactions. Hope this helps. You might take a look at my script (free download in RG).
Depending on how you run the additional experiments, this might be called a meta-analysis.
Another way to approach this is to think of it as model checking. I collect data and build a statistical model of the relationships that I find. I then go back to the laboratory and gather more data. I now want to know if the new data is consistent with the old findings. In this context you would not need to rerun all of the treatments (though it would be better if you did). However, you have a problem with sample size. You need to decide the effect size that would be large enough to warrant rejection. Then choose a sample size sufficient that you could detect a discrepancy if it exists, yet not so large that a difference less than the critical effect size would result in rejection.
- Badges
- Science topic