Imagine I conduct an experiment where I compare outcomes in a Treatment (T) to a Control (C).
The study is under-powered, judged by observed power (which I was asked to calculate due to lack of an a priori power analysis)
I conduct an experiment with the same Treatment and Control (with different subjects), which is not under-powered (based on a priori power analysis), some time later.
Can I
a) use the information of observed power calculation from Experiment 1, in order to show that my a priori power for Experiment II is higher? Stated differently: Can I calculate the observed minimum detectable effect size of Experiment 1 and compare it to the a priori minimum detectable effect size of Experiment 2?
b) Can I use effect size of Experiment II as an informed prior for the effect size in Experiment 2? I am new to Bayesian Statistics but I was courious whether something like this would be possible.
Yes you can do this, you can either go for a full hierarchical model which means assigning distribution to hyper parameters or use an empirical Bayes approach and fix the hyper parameters for the priors using the data from the other experiment.
I would model both experiments together in one model and assign a distribution to the hyper priors. German 2013 has a chapter on hierarchical modelling and an example at the beginning of the chapter.
You may find some useful material and references and R code on my blog as well at Laplacebayes.WordPress.com
Its his book called Bayesian data analysis. You can try any of the editions if I recall there is a 2002, 2008 and 2013 edition
It seems to me that there is a bit of confusion.
An a-posteriori power analysis is logical nonsense. Power is a statistical concept of a decision strategy, not of a particular experiment or set of data. Your experiment does not have a power. Your strategy has a power.
To get a power for a strategy you need to assume a lot of things. Some educated guesses for reasonable assumptions can be taken from former experimence (literature, similar studies, or pilot studies). You use the information from your Exp. 1. You must specify a desired size (alpha) and power (beta) of your test strategy, and together with the effect size and the variance estimated from Exp.1 you get the sample size that would be required to allow you decisions on the desired size that will have the desired power. This sample size is used in Exp.2.
I would say that you can include the data of Exp.1 in Exp.2. For instance if your n in Exp.1 was 5 and the power analysis says you would need n=16, then you will need additional 11 measurements in Exp.2 to have a total of 16 when combining Exp.1 and 2. I can't give you a reference or proof this, but a simulation study shows that the power is held when "re-using" the data from Exp.1.
This is not and has nothing to do with Baysian statistics. "Power" and "Bayes" don't go together.
Dear Jochen,
thanks for your reply.
I agree with you that a posteriori (observed) power calculation is nonsense. However, editors of a scientific journal aksed me to do it. I read a lot about this topic and think I understand the intuition, albeit not all the mathematics, underlying this argument. At least I understand that observed power is a function of the p-value.
Do you think that the decision of whether I can pool data from both experiments together depends on the outcome of the second experiment? I mean, if the outcomes of experiment 2 conform findings of experiment 1, only then can I pool both, or does this not matter?
As for the Bayes related question, I may have not made myself clear. I did not want to do a "Bayesian power analysis". What I intended to do is to take the non-significant mean difference of experiment 1 as an informed prior for the respective mean difference in experiment 2. Is this a valid strategy, as Umar suggests?
Whether you can pool: yes, it has nothing to do with the confirmation of previous data. You may think that both experiments are part of on bigger experiment. Would you stop an experiment when you notice that a part of the data does not "confirm" what you might conclude from another part of the data? (it's a rhetorical question; the answer is: no! for sure). There is a "but": if the results/data are systematically different between experiments, then you have to take this "between-experiment-variability" into account, (e.g. as a fixed factor in the model). However, this is actually a bad thing, because it demonstrates that others may get systematically different results when they repeat your experiment. Not good. Maybe reconsider the assay / experimental design in general?
Bayes: Still a bit confusing. Bayes is not aiming at or interpreting statistical significance. The aim is to have a posterior that as good as possible summarizes our/your current knowledge about some fact (that is expressed as a feature/coefficient of some statistical model). So I wonder why you explicitly mention that something is "non-significant" here. That's irrelevant. The data obtained from Exp.1 modified your (starting) prior to get your "Exp.1-posterior". This states your knowledge after having seen the data from Exp.1. Now you want to improve your knowledge in a second experiment. This means you take your current "Exp.1-posterior" as "Exp.2-prior", include the information from Exp.2 to get the "Exp.2-posterior". Now your knowledge includes your starting prior plus the information from the data of both experiments. I already noted that this same result will be obtained by using your starting prior and update it with the data of both experiments together.
I think Jochen Wilhelm has tried to address your question, and I agree with him. My answer earlier is pointing you to the technical aspects, as to how perform a bayesian analysis (the references from the text book and the code on the blog), and combine the information from the two experiments. You can try a simpler approach for combining the two experiments by using a Random Effects model using the lme4 package in R.
@Umar, a note to your hint using a random effects model: as I see it there are only 2 experiments, so you will use one additional d.f. to estimate the between-experiment variance in a random effect. Problem: estimating a variance from n=2 is extremely vague/imprecise. It will also cost one d.f. when modelling the experiment as fixed factor - but no extra variance needs to be estimated, what I think is advantagous here. I therefore mentioned only a fixed factor approach. Do you agree or am I wrong somewhere?
- Badges
- Science topic