Actually the procedure you describe, although it is quite commonly applied, is logically flawed, and this is the reason why anwers to this question don't make much sense, or why it happens that different people express different "opinions" in response to such a question. But let my clarify:

As you asked the question I assume you talk about "significance tests": one has some data and calculates the statistical significance of this data for some given model and a hypothesized value of the effect in the model (this is the hypothesis that is hoped to be "nilified" or "nullified"; therefor the name "null hypothesis", H0). The statistical significance is expressed as the p-value, which is the tail-probability of finding data that is "more extreme" (w.r.t. the tested hypothesis) than the data at hand. A low p-value says that such (or mor "extreme" data is very unexpected if it was generated according to the assumed model with the specified hypothesis. If the data is carefully collected and the model is well-thought, but the observed data is yet very unexpected, then this may be signify that the tested hypothesis is not credible - so one may decide to reject the tested hypothesis. This decision is a bit guided by the p-value, but is is not determined by the p-value. The decision must be based on the entire context of the experiment (what was the aim? how difficult is it to measure the response? how much data was available? is the effect size relevant? etc etc.). It is very important to understand that it makes no sense to define some arbitrary "significance level" and automatically reject any tested hypothesis whenever the value of p is below this limit. And even if this is done, the "level of significance" does not control a type-I error rate (as often stated), because the type-I error refers to the existence of a substantial (non.zero) alternative hypothesis, which has not been specified here (this might need a further explanation: the type-I error is seen as a "false-positive" result, as the erroneous rejection of H0 where the hypothesis is actually "true". But this is a mathematical formalism that makes no sense in reality. Practically, H0 is never true, so it is a priory known that any rejection will be correct, so a false-positive result is not possible and the type-I error rate is thus neccesarily zero, in a strict sense, no matter what "level of significance" one has chosen - Tthis is why I say that the contro of error rates simply makes no sense in the context of significance tests).

Then, please note that a failure to reject H0 (a "non-significant result") is not indicating any "support" for H0 per se. This interpretation again depends on many things, many of which are even not considered (what are possible different models? what is the cardinality of other models and other hypotheses under which the data would indicate an equally good -or even better- support?), so that "support" is usually nothing that can be derived from the p-value.

P-values and confidence intervals are related measures, and so they can be used in a similar way. If you would reject H0 if p

Dear Essam,

I just wanted to add a brief remark in addition to the other two respondents in direct answer to your question. A confidence interval represents a likely range of values of a population parameter based on a sample statistic. If the assumption that the sample data set comes from a parameter based distribution is in question then this concept is also called into question.

If it is possible to still use a parametric test in a robust way then the confidence interval may be biased. This can be corrected by using bootstrapping. In this case a revised confidence interval (such as bootstrapped corrected accelerated) might still be preferred.

If it is not possible to use a parametric test then a nonparametric one should be used. In this case I would rely more upon the probability value as a confidence interval is less meaningful.

Dear Peter,

a test is (well: should be) performed to test a particular, well-thought hypothesis. So it starts all with thinking about the adequate family of models and about a particular subset of this family that seems to be relevant for the research question. From this the hypothesis to be tested is formulated. After knowing all this one can look up which method one can apply to actually perform this test.

Different tests usually test different hypotheses. However, there are non-parametric tests that effectively test the same hypothesis like a parametric test (example: Wilcoxon and t-test) - if and only if the data is suited for the parametric test. This means: whenever one may substitute a parametric test by some non-parametric test, the parametric test could be used anyway - and it would have the greater power. In all other cases changing the test means changing the hypothesis that is tested. And here I become sceptical: if the hypothesis can be changed just like that it seems that the hypothesis and the entire family of models might not be relevant in the first place. So what is it all about?

(provocative:) If it is all about to calm down reviewers who will ask for p-values and "significant results", then you can simply choose some random numbers as p-values (in case the values will turn out to be larger than 0.05 you may repeat the procedure until you get some nice small p-value). If you really use random numbers from a uniform distribution over [0;1] the resulting p-values are absolutely ok and correct! This procedure guarantuees to get not more than 5% false-positive results. The power is also 5%, what is not much worse than for many real experiments, and given the tested hypothesis is anyway arbitrary a low power should not be a concern. Repeating the random number gereration until the p-value is smaller than 0.05 is clearly cheating, but this is what is also frequently done when experiments are repeated untit the result is "significant" and this last result is published, without mentioning that some or many others failed to reach "significance".