It sounds like a 1 factor ANOVA.

Once you detect a statistical diference between these groups, Wich ones are diferent?

One commonly used procedure is Tukey's comparison test.

@NipNip, I think you missed the question.

If I understand, the question is: If a researcher is interested in comparing the Combination treatment to the others, is it better to conduct all pairwise comparisons (e.g. Tukey) or conduct only those comparisons involving the Combination ("many-to-one", e.g. Dunnet).

I guess if you are really only interested in comparing to the Combination, you would use a many-to-one comparison. This is helpful when there are many treatments in order to limit the number of comparisons being made.

But are you sure comparing e.g. Therapy A to PBS isn't of interest? If you found that Combination isn't better than A or B, but also that neither A nor B are better than PBS in your study, would that give you some insight into the results? Maybe in that case your experiment doesn't have the power to distinguish between A and PBS, so it's inability to distinguish between Combo and A or B is likely also due to a lack of power?

Since you have only 5 treatments, for my money, I would go with obtaining more information and go with the pairwise comparisons.

The question was: "if the combinatorial therapy (composed of therapy A and therapy B) is better than either alone". This is answered by the interaction in a 2x2 model (therapy A (yes/no) x B (yes/no)). This is a single hypothesis that is to be tested. There is no multiple testing.

Fully agree with Jochen. Luckily, the experimental design posted by Adrianne does fit a 2x2 factorial design (with an additional sham surgery group). Therefore, the data can be analysed with a two-way ANOVA, not one-way ANOVA.

Considering the 2 x 2 anova model, I do think that you can use the test for the interaction as a test to determine if the Combination is greater than A or B, but I think it's trickier to interpret than one might expect.

See if I'm correct about the following.

An example. Excuse the simplification.

Let's say the Control has a value of 100*. When Treatment A alone is "yes", the value is 101, and when Treatment B alone is "yes", the value is 101. So for either A or B, going from "no" to "yes" has an effect of increasing the value by 1. The null hypothesis for the interaction is that the effect of B going from "no" to "yes" is 1, whether A is "yes" or "no". In that case the value for Combination would be 102.

Scenario 1: So, if the interaction is not significant, the value is 102. And the Combination is greater than either Treatment A or B.

Scenario 2: If the value for the Combination is 101, then there would be a significant interaction. In this case, obviously, the Combination is not greater than A or B.

Scenario 3: If the value for Combination is 103, then there would also be a significant interaction. But in this case, the Combination _is_ greater than A and B.

So, a significant interaction might mean that the Combination _is_ greater than A and B or that it _is not_ greater. **

I think in reality you could look at the plotted data to determine if the significant interaction means that the Combination is greater or not greater. But you'd have to be careful in interpreting the test.

To further complicate matters, imagine a different case where Treatment A has a different effect on the value than does Treatment B, and play out the different scenarios.

My takeaway: even in the simple case of a 2 x 2 anova, to avoid misinterpretation, I would still use post-hoc mean comparisons to have clarity as to which treatments were different than others.

I hope this example made sense. If this isn't clear, and people are interested, I can create examples with toy data and plots.

_______

* You can think of this as the mean for the treatment, and assume that there are multiple observations, and that there is some variability about the mean.

** You can substitute "different" for "greater" here. My point doesn't rest on the question being one-sided.

@Salvatore,

it seems correct what you write, but I don't see where's the big complication. The interaction can indicate a synergism (interaction coefficient is positive: effect of A and B together is more than the sum of the individual effects) or an antagonism (negative coef: A and B together give ess then the sum). If you don't consider the result significant for a negative coefficient, you do a one-sided test. That's no different to a simple t-test, where the mean difference could be positive or negative, too.

@Jochen,

My point was just that the hypothesis for the interaction doesn't map in a one-to-one way to the question, "Is the Combination different from Treatment A [or B or either...]". For me, this makes the interpretation of that hypothesis test for that question at least a little tricky.

But to make things more tricky, let me present some toy data I came up with. There are two data sets, one called Example Alpha and one called Example Beta. Analyzed as a 2 x 2 factorial. The results are presented in the attached images of the anova tables and plots.

At least by the way I'm doing the post-hoc comparison of the treatment means, the two data sets come up with very different answers to the question if the Combination treatment is larger than the single treatments. But I certainly wouldn't be able to suss this out from the anova tables and plots.

@Salvatore,

In both examples, the interaction was not significant (at any reasonable level), meaning that we cannot (reasonably) confidently interpret its direction (positive or negative: synergism or antagonism). Now you write " At least by the way I'm doing the post-hoc comparison of the treatment means, the two data sets come up with very different answers to the question". How do you do the post-hoc comparisons, especially that to conclude that the combi-effect is larger than the sum of the individual effects? You haven't attached this information. My gutt feeling tells me that there is somehow the interpretation of some non-significant difference involved...?

Correct me if I am wrong. I think what Salvatore is trying to express is the additive effect of the combined therapy. Basically, a non-significant interaction in a two-way ANOVA analysis simply tells us there is no synergism or antagonism between the two factors. Based on what I understand from Salvatore's comments, I think he is trying to mention that there could be some sort of additive effects between Treatments A and B, which makes the combined therapy better than Treatment A or B alone.

What I look back to Adrianne's question, it is indeed what she asks, although she did not mention if synergism is of interest. Hence, pairwise comparison is indeed what she needs to examine the difference between groups.

However, I think analysis like this should be done cautiously.Generally, if the interaction of a two-way ANOVA is not significant, we will look at the main effects to see if they are significant. If only one of them is significant (eg. Treatment A), then we will conclude that (most of) the treatment effect is contributed by Treatment A. In this case, a pairwise comparison would show a significant difference between the combined therapy and Treatment B, but it is illogical to conclude that the combined therapy is better than Treatment B alone when most of the treatment effect in the combined therapy is in fact, caused by Treatment A.

In my opinion, only when both main effects are significant, then perhaps, it could be meaningful to look at the additive effect with a pairwise comparison.

@Hong,

thank you for the clarification. I haven't thought that the question was simply "is there some additive effect at all" (not neccesarily a synergism!). In this light I think I now better understand Salvatore's post.

However, I still think that looking at the main effects individually doesn't help in finding if the data is sufficient to demonstrate any kind of addition, synergism or antagonism of the two treatments. The only correct way (as I understood) will have to interpret the interaction. If you want to show that the effects are really just additive, you must make sure to get a narrow confidence interval for the interaction coefficient and check that it doesn't extend into any "relevant" synergistic or anatgonistic region.* Showing that the positive interaction coefficient is significant allows you to conclude that the effect is at least not negative (however, you won't be able to conclude that there is no relevant synergism).

Even if both main effects are significant, this alone provides no information on the combined effect.

---

a non-significant interaction is no indication that there is "just an additive effect" (and neither synergism nor antagonism). It only says that the data is not very incompatible with a synergistic as well as with an antagonistic effect (i.e., we can not confidently claim the sign of the interaction). Only a sufficiently narrow CI may be used to rule out that there is any "relevant" non-additive effect involved (and this is actually a Bayesian interpretation taking the CI as a PDI, what is not really correct but acceptable, at least when the sample size is not too small and a flat prior is not too unreasonable).

Sorry, I wasn't trying to hide anything. I've attached the r code and post-hoc results for my examples. (emmeans in R used, with no FWER adjustments for the multiple tests). BTW, the error bars on the plots are standard error of the means.

One note: yes, I was looking at the mean comparisons on the interaction even though the interaction was not significant.

Ignoring the mean separations for a moment, I think my examples do show that using the anova table makes it tricky to answer the question about if the combination treatment is better. In both Example Alpha and Example Beta, the A-alone effect is not significant. Since the interaction is not significant, one might conclude that therefore the Combination is not better than B-alone. (For me, it's easiest to look at the plot to follow this logic). At least that's the logic I would use.

But I would argue that in example Beta, I would want to conclude that the Combination is better than B-alone. I would like to conclude this by considering 1) the 2 x 2 model, ignore the anova table, and run the mean comparisons on the interaction; 2) look at the plot; or 3) use a one-way anova and look at the anova table and post-hoc mean separations.

So... If I were giving advice to a student or colleague about how to determine if the combination treatment is better than the Treatment A alone or Treatment B alone... What I would I suggest? I think I would recommend 1) or 3) above.

The synergism question is interesting, particularly from a scientific perspective, but is a different question.

Yes, these tables do "make it tricky", because these tables do not adress that question at all. And you wrote "Since the interaction is not significant, one might conclude that ..." - that's what I meant: you use (implicitly) a non-significant result to make some conclusion. Although many people make such a mistake, it's still a mistake.

Regarding ex.Beta: the p-value of the interaction is 0.227. The MLE is 10.5, but the standard error of this estimate is huge, too (8.241), resulting in a low t-value (1.274) and the corredpondingly large p-value. The "conclusion" is that this data you provided (well, that you made up) does not provide reasonable evidence to conclude if the combination of A and B is better or worse than the sum of the individual effects of A and B. The data in inconclusive regarding the interaction (and it is also inclonclusive regarding both main effects, too).

Wow lots of answers! Sorry I didn't receive any notifications from ResearchGate and didn't realize I had gotten responses.

To clarify (if this helps), both Treatment A and Treatment B have been shown to be effective (significantly better than negative control) in many previous experiments. So I don't know that it's necessarily useful to compare them again in this case or not

Also I don't know if I should expect additive or synergistic effects from the combinatorial therapy.