Hi there,

just a small note: every time you use a t-test on the same data there is a chance for a Type I error (i.e. reject null hypothesis when it is true, which is approximately 5%). If you conduct multiple t-tests on the same data, Type I errors will be addictive. So for two t-tests the chance for Type I error will be ~ 10%. Even though you're not interested in one of the comparisons, you should use ANOVA and Post-hoc tests because in this scenario Type I error is controlled to be always 5%, even with multiple groups.

Cheers

Hello good friend.

The fact is, when you have several groups (more than two groups) to compare, use ANOVA.

And then use Post Hoc (such as LSD, Bonferroni, Duncan, ....) for double comparisons.

But everything goes back to your null hypothesis.

If your null hypothesis is not meanA = meanB = meanC

So, it's better not to use ANOVA.

You can also use the T test if parametric assumptions exist.

Or, if the number of data in each group is low, use the Mann-Whitney Non parametric test.

ANOVA is still necessary after which a post-hoc multiple comparison test may be used if the need arises. This would be more conventional and economical to use than two separate t-tests to compare group 1 and groups 2 and 3 respectively. If interest is not in the groups 2 and 3 comparison, this comparison may be ignored.

I thought it would be better to answer Ette Etuk's good friend.

As you know, when it comes to ANOVA we will have the F distribution test statistic.

In the calculation of this statistic, the data of all three groups are used together. In other words, while we only want to compare the group A with B, it has also been unconsciously contributed to the group C.

Similarly, when we want to compare only Group A with Group C, again, Group B intervenes in our calculations.

And that's not our favorite.

Of course, you know better than me.

Thanks Dr Etuk and Dr Ghoodjani for your kind reply.

So my understanding is that I can use two separate t tests to compare group 1 and group 2, group 1 and group 3. And I do not need to adjust the alpha value because there will be two null hypothesis (mean 1 > mean 2) and (mean 1 > mean 3) in separate t tests.

Hi Jiaqi,

When you have several means to compare, it’s not valid just to compare all possible pairs with t tests. Instead, you follow a two-stage process:

• Cheers, Frank

Hi there,

just a small note: every time you use a t-test on the same data there is a chance for a Type I error (i.e. reject null hypothesis when it is true, which is approximately 5%). If you conduct multiple t-tests on the same data, Type I errors will be addictive. So for two t-tests the chance for Type I error will be ~ 10%. Even though you're not interested in one of the comparisons, you should use ANOVA and Post-hoc tests because in this scenario Type I error is controlled to be always 5%, even with multiple groups.

Cheers

Hi all,

When you have sevrel means to compare

- Are all the means equal?

use ANOVA (case normality)

- If ANOVA shows that the means are'nt all equal ( use Post Hoc) for double comparaisons.

- Use Man-whitney non parametric test (if no information about normality)

Hello Jaiqi,

I appreciate your dilemma and think you have received some good advice above. Even though you are not interested in the difference between Groups 2 and 3, I think it might be good for the sake of "completeness" to check whether they might differ from each other or even confirm that they don't. Because of that, I'd do a one-way ANOVA. There would be no harm in that because a significant F value would indicate that at least two of your three groups differ from each other (possibly / probably Groups 1 and 2 and/or 1 and 3).

Furthermore, as pointed out above, it's not good practice to do multiple t-tests because of the risk of committing Type 1 errors. Using any of the post hoc methods (Bonferroni, Sheffe, Tukey's HSD, or LSD - each has slightly different characteristics) would help you avoid that in a legitimate way.

So, in essence, I'd do a one-way ANOVA with what you regard to be the most appropriate post hoc comparisons. (You might need to read up on the comparisons a bit, e.g., because Bonferroni can be a bit too conservative and LST not conservative enough.)

All the best with your work.

Robt.

Dear Jiaqi,

as you desire ‘only’ two specific comparisons, instead of ‘all’ possible three, there is no ‘need’ to use ‘post hoc’ tests. Instead, you ‘may’ use ‘planned’ or ‘a priori’ contrast-tests. In SPSS, instead of using, e.g.:

unianova DepVar by IndepVar /posthoc=IndepVar(Bonferroni).,

(which requests one of the many ‘post hoc’ contrast), you ‘may’ use:

unianova DepVar by IndepVar /contrast(IndepVar)=Simple(1).

‘Simple’ is a so called ‘planned’ contrast (in this case each subgroup vs the first subgroup), and therefore does not ‘require’ that the overall F-value is significant (although when ‘F’ is significant, usually one or more of the contrasts is significant and vice verse).

With three groups, you have two degrees of freedom, and are therefore ‘allowed’ to perform two specific or ‘planned’ contrasts, without any adjustments for alpha. The ‘power’ of ‘planned’ contrasts like ‘Simple’ is (somewhat) higher than the ‘power’ of ‘post hoc’ contrasts, as the alpha does not have to be ‘corrected’ for multiple comparisons. As a result, ‘planned’ contrasts are usually somewhat more ‘significant’ than ‘post hoc’ contrasts.

You will obtain even somewhat more ‘power’ (i.e. somewhat ‘more’ ‘significance’) with two independent t-tests, e.g.:

t-test groups=IndepVar(1, 2) /variables=DepVar., and:

t-test groups=IndepVar(1, 3) /variables=DepVar.

This option of using two independent t-tests is somewhat ‘beside the book’ and often frowned upon. There is no law against it, however, and you should certainly try it out as well, at least to get a feeling of the (usually small) differences between these various alternatives, and moreover to increase your understanding of both your own dataset as the area of statistics in general.

Success, Ernest

T test for separate groups is not appropriate for more than two groups. However, I think that the decision is between ANOVA and a nonparametric competitor (Kruskal-Wallis). Test for normality of your groups (e.g., Shapiro-Wilk) and also for equality of variances (e.g., Levene). If your data do not meet these criteria (and especially if your sample size is small) consider the Kruskal-Wallis. JASP will run the ANOVA (even a Bayes version), and also the Levene and Shapiro-Wilk. Kruskal-Wallis can be run on the VassarStats.net site (under ordinal) or with Statcato (my choice: just put each group in a separate column).

The lastest version of JASP will run a Kruskal-Wallis.

You may consider the "Planned Comparisons and Post Hoc Tests"

Hello,

The simple way to deal with the problem is to use Dunnett's test "all comparisons with a control", which is designed especially for this. It is in fact the two T-tests you're mentionning, with optimal multiplicity correction, provided that assumptions for the ANOVA are met [a version exists for non-equal variance, however]. It is a special case of the pre-planned contrasts method suggested by Ernest de Vroome.

Computations use the residual sum of square of the ANOVA, so running before an ANOVA is a convenience, but it is NOT needed. Performing post-hoc tests only when ANOVA is significant is « needed » only for post-hoc methods that do not correctly handle the multiplicity correction, like the basic LSD method, leading to the « protected » LSD (PLSD) method. Well designed post-hoc comparison methods can be done without a previous ANOVA — like the Dunnett's test. And that's even more true for pre-planned contrasts, where previous ANOVA is irrelevant, it's just a technical step to obtain coefficients estimates, covariance matrix and residual sum of square efficiently, but the F-test is here useless.

If assumptions are not met, you may go for non-parametric methods with suited multiple testing corrections, like two Mann-Whitney tests. Note however that technically, Mann-Whitney tests (and related) can be significant because of changes in variances or shape between groups, not only because of shifts. So interpretation of its results must be done with care.

Oh, yes, Emmanuel. This is the test I was thinking of in my previous answer. Duncan was a typo. Dunnett's test is the appropriate one for the problem mentioned by Jiaqi Zhang. :)

The t-test is simply a specific example of ANOVA with just two means to compare. ANOVA is by definition the same test except that three or more means are compared. Non parametric counterparts are Wilcoxon (Mann-Whitney) and Kruskal- Wallis respectively.

ANOVA uses an F-statistic but the t-test is simply an F-test with df (1,v) so only requires on value of the df compared to the two used by ANOVA. If the test in not significant then one is finished.

After finding that the ANOVA finds statistically different means, then it is time to do a series of t-tests ( there are a bunch of good ones, Tukey tests, Dunnett's, etc) to find which pairs are significantly different from one another.

Does anyone know if there is any non-parametric equivalent for the pre-planned contrast tests? In other words, how would you run a priori with a Kruskal Wallis test?

Ernest de Vroome

You may have a look to the Nemenyi test or the npcomp package for R.

Hello all,

I would like to ask a question about statistical analysis for group comparison. In my study, I have 3 COMPARISON GROUPS, group 1 vs. group 2 vs. group3. And my hypotheses is group 1 will be better than group 3 based on theory. And then could 3 separate t test to compare group 1 and group 2 as well as group 1 and group 3 group 2 and group 3? Shall I need adjust the alpha value?

I am looking forward to seeing anyone's reply! Thank you!

Best wishes,

Alanazi Amal : if your hypothesis is really « group 1 will be better than group 3 », the group 2 is superfluous and a simple unilateral t-test between group 1 and 3 will do the job.

If you really need to make some comparisons with group 2, then your hypothesis cannot be the one you gave.

Emmanuel Curis thank you for you answer. yes my hypothesis is group 1 is better than group3. In this case I will report mean for each group and p- value for group1 vs. group3 . my data is not normal distributed I will use Mann Whitney u test at one-tailed. what I understand from you I don't need to make bonferonni correction right

Alanazi Amal : yes, if this the only hypothesis that you test with this dataset, then you do not need multiplicity correction. But if this is really the case, I wonder why there is a group 2 in the experimental design...

Hello all,

I would like to ask a question about statistical analysis for two group comparison I have a directional hypothesis group 1 is higher than group 2 . I did separate analysis for 3 exams, exam1, exam2, and exam3 . because different content, Shall I need adjust the alpha value?

I am looking forward to seeing anyone's reply! Thank you!

Best wishes,

Thank you so much Emmanuel Curis , If I have three comparison group. And my hypotheses are that group 1 will be better than group 2 and group 1 will be better than group 3. I am not interested in comparing group 2 and group 3. there are2 groups is normal and one is not normal.distrubuited

And then could I use two separate t test or Mann Whitney u test to compare group 1 and group 2 as well as group 1 and group 3? Shall I need adjust the alpha value?

Yes to both questions, but consult a local statistician for more precise advice, and explain him/her in details your experimental design and your question.

Thank you so much Curis, I really appreciate your help

Amal

Depending whether the groups are.assembled by randomisation we could use ANOVA or ANCOVA, then Sheffee test ofcomparison may introduced to dètermine the strenght of the differences

You should use ANOVA because applying t-test for the the same data set multiple times approaches Type-I error.

Thanks for the recommendation

It depends on you. You can ignore the parts that you don't mind and keep away from reporting them. ANOVA is better due to its simplicity. On the other hand, generally, the posthoc tests of ANOVA and T-test conclude a similar result.

The trouble with not doing pairwise t-tests is that you have no details about the components of variance contributed by each group. If the results from the ANOVA for say 3 groups indicate significant differences among the groups, it may be that one pair is much more influential to the ANOVA results than the other two pairwise comparisons.