First of all, you must note that your title contains a contradiction. For significance testing, for tests in general, confidence intervals are not more suited than p-values. p-values are for tests problems (that is, questions for which the answer is yes or no), confidence intervals are for estimation problems (that is, open questions on the values of a parameter of a model like "what is the value of…"). So the discussion is in fact not on the p-values, which are a very valuable tool for hypothesis testing, but on the relevance of hypothesis testing in many settings. Which takes two aspects: 1) is indeed a yes/no question the question you want to answer and if the answer is yes 2) is indeed the null hypothesis you choose for your test suited for your real question. Quite often, the answer is no for at least one of these questions. But if the answer is yes to both, then go for a test and use the p-value confidently.

In addition to Jochen's comments, note that the model used in the hypothesis includes a lot of thing including, as a probabilist model, assumptions on the joint distribution of the random variables modelling your data.

For instance, the null hypothesis of the Student's T-test is not "difference of means is 0", but « The Xi (i = 1 to n) random variables modelling the n experimental results (combining both samples) are independent, identically distributed according to a Gaussian (« normal ») distribution of expectation µ and standard deviation sigma² », and that any deviation to this hypothesis will distort the p-value distribution from an uniform distribution on [0,1] — which includes unequal variances, non-independent experimental observations [think about the paired case…], results with different distribution [outliers…], non-Gaussian distribution.

For some of the distortions, you may well have p-values with a right-skewed distribution, by the way.

Not also that the confidence interval/p-value equivalence only holds in simple cases. In more complex cases, you must be very careful in the definition of your confidence interval so that the equivalence still holds (think of the comparison of a probability to a theoretical value with the basic Wald test, for a very simple example), or you even cannot build a confidence interval. But from a confidence interval, you can always build a test, that may not be the more powerful.

Dear Kris,

Confidence intervals are not reliable when the data distribution (or the residuals) is non-normal. We would recommend bootstrapping and using the Bias Corrected Accelerated confidence intervals instead In this case.

@Jochen: no, the null distribution for the T-test is not uniform if indeed variances are not equal (and you use the classical, common variance, n1+n2-2 degrees of freedom). Neither if the values are dependant.

For case of unequal variances, this is the reason for the Aspin-Welch form of the test. Simulations show this easily.

For case of correlation between values, just imagine a case were data are prefectly correlated, all your observations are the same. You can have only one value of the test statistic (which will be a first glance undefined because of the null variances, but that's a secondary debate, and I guess it would be 0 once the undetermination 0/0 is removed), so definitly the p-value distribution is not uniform at all. Even if this single value repeated n times because of perfect correlation (X1 = X2 = X3 = ... = Xn) comes from a Gaussian distribution (X1 to Xn all follow a N(µ, s²) law, so all other assumptions of the T-test are met).

So the fact that the test is "not sensitive" to "small" deviations from assumptions simply means that the distribution of the P-value is _close to_ an uniform one, but not an uniform one.

By the way, I totally agree that M-W test is not a test for comparing median if no other assumption is done on the underlying distribution (typically, it is a test for comparisons of medians only if one assume that there is only a shift in the distribution between the two samples, which is not the case in your example).

Here's the code for the simulation in the case of unequal variances (an extreme example, so things are visible). That's not a mathematical proof, but hope will convince you that _any_ departure from the _whole_ set of assumptions makes the p-value distribution not-uniform. The departure from uniformity may be small and of little practical relevance, but it exists.

@Jochen : for case A, I guess your ratio of variance is not high enough, so departure from uniformity is tiny, and can't be seen with « only » 1000 simulations. Another point is that your sample size is "big enough" (between 110 and 200, under the null) and with a symetric distribution (and a normal one), so TCL/Slutsky theorem apply well and you are anyway close of the N(0,1) distribution beeing the samples either of same or different variances.

You may test the R file I attached, and you'll see the standard deviation ratio is 10000 and there is 10000 simulations, and the departure is still small. Sample sizes are also small (n = 5/group) to limit the convergence to the N(0,1) law and emphasize the departure to the theoretical _Student_'s law.

For case B, if the test does not produce low p-values, I would say that the null is right-skewed, no?

@Jochen: got your error: you forgot the "var.equal = TRUE" in t.test, so you are not doing a classic Studen'ts T-test but the Aspin-Welch test for unequal variances... Which explains why your test is indeed still uniform when unequal variances are at play! Note also that the case of equal variances (or the "identity" line) was absent, making difficult to detect small departures.

Other comments still apply, however, so to see the departure you should decrease the sample size and increase the ratio of variances and the number of simulations:

nreps

Thanks to everyone for taking the time to explain. Some of it goes beyond my current grasp of the topic, but I get the gist of what you are saying.

I do not care too much about significance tests, I prefer to look at the standard deviations of the estimated coefficients. I find that most econometric studies take the null-hypothesis too literally. They only test the sign of the estimate (if it is higher or smaller than 0). If this "significance test" is OK, they take exactly the estimated number - often with many digits after the decimal point without considering that the "true" value could be 10, 20 or even more percent higher or lower.

Already has great answers posted. Here is my 2 cents:

Statistical testing of null hypothesis (using any appropriate statistical tests: t- test, ANOVA, Chi-square, Fisher’s exact etc), you get a p-value. P value gives binary answer [Yes statistically significant (reject null hypothesis) or not significant (fail to reject null hypothesis)]. As you correctly pointed out, P value is affected by sample size.

Confidence intervals (CI), on the other hand, is more informative because it assesses the precision of the sample estimate. CIs include the point estimate or “the most likely value” and a margin of error around that point estimate (e.g., odds ratio (OR) = 2.1; 95% CI: 1.6 - 2.9). This margin of error indicates the amount of uncertainty of sample estimate (i.e., 2.1) that surrounds the population parameter (i.e., true OR).

For a specific point estimate (e.g., OR=2.1), a narrower confidence interval [e.g., 1.9 – 2.4] suggests a more precise estimate of the population parameter than a wider confidence interval [e.g. 2.6 – 3.9].

Apologies for typos on last sentence

......population parameter than a wider confidence interval [e.g. 2.6 – 3.9] ---> correct: [1.6 - 3.9]