@ Asmat Ullah

Enclosed jpg file presents two examples of proper formulas for "pooling" the p-values of two independent samples, of size n1 and n2 into one - of size n12:= n1+n2. The first one relates to test of the mean value if the variance \sigma^2 is given; then the formula for the pooled p-value uses the sizes and the p-values of each sample p1 and p2. The second example shows, that the exact formula for the chi-square test needs additionaly the square of the difference between the arithmetic averages in the separate samples. Instead, a p-value is given for an approximating test. Regards, Joachim

Comment. Obviously your problem is much more complex due to lack of precise description of the statistical results used fro the obtained estimates. The above given text repeats my answer given to a question removed by the author of a question from RG (referred to with name @Fausto. However, some of other answers are obtainable from my Contributions page on RG

https://www.researchgate.net/profile/Joachim_Domsta/contributions

One can use the records within part labelled by "Answers" dated between circa 20 Dec 2017 - 15 Jan 2018. Let me suggest another researchers answers to that question (e.g. Jochen Wilhelm, Emmanuel Curis, Kåre Olaussen). Try searching answers to NOT-NAMED questions:) JoD

Dear Asmat

The hypothesis of the two test you have mentioned are different, and this case found also repeatedly in ANOVA (F-TEST) not significant, but found significant difference between means in multiple comparisons tests ( Duncan, Tukey, ...), by using SAS or SPSS softwares.

Good Luck

You are talking about a t-test of two groups and an anova of two groups? The results should be identical. See the discussion here: https://www.researchgate.net/post/In_situation_of_equal_means_of_two_groups_which_test_result_should_be_accepted_t_or_F_Whr_does_F-test_appropriate_for_2group_difference

It is possible, tho, that conducting the tests with software will be performing different tests. For example, if the software conducts Welch's t-test by default.

The problem is not related to the software applied, this is a problem of the relation between the critical regions of two tests performed on two different samples (this is assumed by Asmat). Moreover, the question is about the net conclusion. The examples from my previous answer are presenting two cases: the first gives a justified way of "pooling" the two p-values which was possible because of the specific fromula for the test statistics. The second example warns that not always such a stricly justified way exist if we do not have additional data related to the TWO samples (in the example this is the difference between the arithmetic means, which obviously cannot be deduced from the values of the chi-square statistics. The more, no strictly provable algorithm can be supplied from just knowing the final significance of the obtained data, when the values of the applied statistics are lost.

Acceptable algorithms, however, are used and could work pretty well. The possibility can be illustartes by the followig example: If X and Y are independent variables with given cdf-s F and G, and our critical values are above thresholds x(\alpha)= F^{-1}(1 - \alpha) and y(\beta) = G^{-1}(\beta), then the critical values of the pair (X,Y) with given probabability can be defined in many different ways. To be more specific, let me confine the next lines to the case of uniform distributions on [0,1].

Our choice nr one let be right-upper triangels of the square [0,1]^2. For given probability \gamma of it, this is a triangle with side size equal to \sqrt{2\gamma}. Thus if our observation x is greater then 1 - \alpha (significantly against the nul hypothesis) and the second observation y is below 1-\aplha (not opposing the nul hypothesis), then without any additional information we cannot state if the pair of the TWO results is in the critical triangle with probality \alpha. BUT if we know the p-values, which are 1-x and 1-y, respectively, then we can check their belonging to the triangle as follows: If their sum 2 - x -y is greater than \sqrt{\gamma}m then they are NOT in the critical region, and they are there in there, if the sum is less (for simplicity I am omitting the border, which is o propbability 0).

Our choice nr two let be sets of the form of right-upper squares of the form [u,1]^2. For level \gamma, the critical value u(\gamma) is common for both x an y and equals 1 - \sqrt(\gamma).

The two critical sets even with the same probability possess non-empty common part, there are pieces of the trianle outside the square, and there are parts of the square outside the triangle. This proves non-uniqueness of the inferred critical set for the joint statistical analysis of results of two independent tests. Indeed, in both cases, there are results (i.e. pairs (x,y)) which are in the critical set with the given probability (significance level) , even if NONE (neither x nor y) is above its own critical value with the same probability [simple example: assume \alpha = 1/100; then the square size in the second case is 0.1, whereas the critical interval for x and y separately is of length 0.01].

Hopefully this helps at least in treating any "wonderful" algorithms for "pooling" the p-values with very very great carefulness.

Good luck, Joachim

@Joachim,

I'll assume you are right about the question posed by the original poster. But can you clarify what the question means? What is "the P value of two sample groups is less than 0.05"? What is the F-test mentioned?

@ Salvatore,

Frankly, it does not matter. There were two tests performed - an F-test and a T-test. It is not clear for me if the two tests were performed on the same sample (consisting of two groups) OR these were two tests performed on two different samples. I have assumed the second case, which I have stated within the introductory assumptions. I didn't count on other reply on doubts which case should be considered, but by @Asmat. Namely him belongs the right and kindness' duty to explain his question. But once you are asking first, thus I have to explain what I had meant.

Let me add, that in the background of my answer is simultaneously hidden the answer that if this is the first case, then different test can give different p-value for the SAME sample. Therefore, I have resigned of wrighting separate analysis for that case. Much more important is, that there is no chance to advice about the net result when we have the two different values for different tests on the same sample. We can only report them.

Thanks for the question, best regards, Joachim

PS. For avoiding misunderstandings, I am using the following definition as I see it on the basis of some answers of another RGaters:

Definition: Let T is the statistics (dependent on the observed data) used for a statistical test and let us have a monotonically increasing family A(p) of critical subsets of the space of values of T such that under H_0 Pr{ T \in A(p)}=p. Then for given any observed value t of the statistic T p-level of the test is defined as the supremum of those q, for which t is not contained in A(q).

Accordingly, for continuously distributed statistics, the p-level is a random variable (dependent on the observed data) uniformly distributed over interval [0,1]. JoD

Well thank you all for the precious answers. I would like to add here the groups were same and were two in number. Group 1 (case) and 2 (group). I applied t-test for significance. The value i got was < 0.05. But when i applied F-test (Anova) on the same two groups, the P value obtained was > 0.05. I am taking this from the results that the difference between my case and control groups is significant in terms of T-test but not significant in terms of F-Test. My question is that WHAT IS THE FINAL CONCLUSION? SIGNIFICANT or NOT-SIGNIFICANT? Or i may put in the the statement that w.r.t T-test it was statistically found to be significant however w.r.t F-Test it was not-significant? Need inputs... Thanks

@ Asmat Ullah

One of the simplest ways of building the critical set is to choose the sum of sets (C1\times R) \cup (R \times C2), where C1 is the critical set for the first test and C2 for the second one. If both are of probability \alpha, then their sum (by independence of the two tests) is \beta = 2 \alpha - \alpha^2. Thus, this \beta is the significance of joint pair of the tests.

Conclusions:

If you just know that the p-level of one test was less than 0.05 and the second one was greater then 0.05, you are justified to claim, the the pair of tests reject the null-hypothesis on the level 0.1-0.0025=0,0975.

If you know more precisely the p-level of the first test, say 0.35, then you can claim that the significance level of the pair of the tests is 0.07-0.001225.

Obviously this is only one of the simplest ways of getting off the problem. Frankly, without knowledge about the value of the p-levels of both tests there is no widely acceptable way of calculating the significance of the results of joint treatment of the two tests.

Regards, Joachim

My answer above is correct. See the link there also.

t-test and anova (one-way with two groups) will give the same answer. However, your software may be using variants of the tests which give different results. For example, by default R uses Welch's t-test. This will match the results of Welch's anova, but not the results of the more common non-Welch anova. Likewise, student's t-test will match common anova.

As a final note, if your conclusion will be based on whether a p-value is 0.049 or 0.051, you may need to re-think how you are reaching your conclusion.

I agree with Salvatore.

Asmat, please tell us the mean, SD and sample size for each group, and the the results of your two tests. For the t-test, observed t-value, df and p; and for the F-test, observed F, numerator and denominator df, and p. With that info, someone will likely be able to tell you exactly what is going on.

Dear @Asmat, as you want to test whether the average value of two groups is equal or not. than you have to use the test which is specially designed for that purpose. F-Test (ANOVA) is not suitable test for this purpose as the basic condition for utilization of F-Test (ANOVA) is the number of groups must be greater than 2. so when you utilize F-Test on 2 groups, the results will not reliable. the suitable test for 2 groups is t-test. but t-test further has 2 types one for dependent samples and other for independent samples. if your two groups are dependent on each other than you have to utilize dependent samples t test. but if you have two independent samples than further you have two types of t test in this case. one is pooled t test and other is welch t test. if your 2 groups have equal variances than go for pooled t test but if they have unequal variances then welch t test will be efficient.

therefore, before utilizing any test, first check the assumptions of test. because, when your data fulfill the assumptions of test, it will give you reliable results.

Dear Followers,

A test with given significance level \alpha based on two independent statistics say T and S is any set A \subset R of the product probability

P_T\oproduct P_S(A) \le \alpha,

where P_t and P_S are the probability distributions (pd-s) of the statistics under the null-hypothesis H_0. The shape depends on the probability distribution of the same statistics under the alternative hypothesis H_1. Mainly due to commonly assumed requirement that the probability of this set should be as great as possible.

EXAMPLE. Let the statistics under H_0 have uniform distribution on [0,1]

A. If under H_1 S possesses distribution function F_1(u) = u^2,with a>0 and simultaneously T possesses distribution function G_1(u) = u^{3}, then the density of the joint pd of (S,T) equals

d(u,v) = 2u * 3v^2 = 6 u v^2

In such a case, the shape of the critical set with maximal probability under H_1, with given probability \alpha under H_0 is given by

A(k) = { (u,v) : u v^2 > k}, where AREA{ A(k) } = the_given_alpha

B. If under H_1 S and T possess distribution function

F_1(u) = G_1(u) = 1- (1- u)^1/2,

then the density of the joint pd of (S,T) equals

d(u,v) = [(1-u)(1-v)]^{-1/2}

In such a case, the shape of the critical set with maximal probability under H_1, with given probability \alpha under H_0 is given by

B(k) = { (u,v) : (1-u)(1-v) > k}, where AREA{ B(k) } = the_given_alpha

With respect to the question of this thread, it follows that without well determined alternative hypothesis we cannot give the form of the critical set with maximal power. Admitting any other power of the critical set leads to arbitrariness.

Let me suggest some other remarks from the paper: Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations, by Sander Greenland et al., Eur J Epidemiol. 2016; 31: 337–350. Published online 2016 May 21. doi: 10.1007/s10654-016-0149-3, available at

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4877414/

Regards, Joachim

Asad, there is no law of statistics prohibiting use of ANOVA when one has two groups. In that case, the F-ratio from ANOVA is exactly equal to the square of the t-value from a t-test on the same data. Find a data set and give it a try. HTH.

you are right Bruce, but my point is that when we have specially designed test for some conditions than why we will go for other test?

Well @AsadAli & @Bruce Weaver. Thanks for your so nice comments. I would like to add here that i have actually five groups. Can you please suggest that how will i use F-test (ANOVA) on it because till now i was considering F-test to be same for two groups. Please generalize. Thanks

Research Welch's t-test for comparing two independent groups: An Exce...

Welch's t-test is preferred even if you have equal sample sizes and variances.

If you want to compare the differences in outcomes between Student's t-test and Welch's t-test, please check out my calculator.

If the populations' variances are equal, the t-test is the most powerful (under normality assumption) even with different samples sizes. In these situations, I don't agree with Justin's comment. The preference of Welch test is based on general considerations in which the researcher has no knowledge about populations variances equality. From the Asmat's question, we can assume this is the case, and I agree with Justin's recommendation. If you have more than two independent groups you might use Welch anova.