The multiple hypotheses I was testing can use two one-tailed tests (test two directions separately) or one two-tailed test. And I'm using false discovery rate (FDR) to control type I error.
For the two one-tailed tests, I could get statistical significance for the negative direction, but not the positive one. However, I did observe that there are few p values on the positive side and they were lower than the significant p-values on the negative side. Because there's only a few of them, it couldn't pass the FDR threshold.
Then I tried the two-tailed test, though the significance level was lower than the one-tailed test from the negative direction, it was still acceptable. The good thing about the two-tailed test is that, now I can see the few lowest p values from the positive side also can be considered as significant.
So it seems that the two-tailed test is a better choice for me. But how could I explain that the p values from the positive side won't be significant if using a one-tailed test?
I think the question should be what null hypotheses you want to test. Null hypotheses are not the same for one-tail test and two-tail test. In general, if you do not know to chose between one-tailed or two-tailed test, the two-tailed test is prefered because it is more conservative (null hypothesis is more general).
You use the one-sided test if your hypothesis is one-sided, i.e. if the positive association would not make sense in your model. It sounds like you are looking for any associations, positive or negative, which would call for the two-sided test.
It is not good practice to pick your hypothesis based on the data, so picking a one-sided test because of an observed lack of positive associations is not well motivated. Doing the statistics in multiple different ways and then going with the method producing the most attractive results is similarly problematic, Technically, it incurs an additional multiple testing burden, counteracting any advantage you might hope to gain.
A good rule of thumb: If you find yourself tinkering with your statistical method to obtain significant results, whatever you have is tentative at best.
A one-tailed test has a higher power to detect a directed hypothesis. Actually, a two-sided test is not appropriate for a directed hypothesis anyway. But is the directed hypothesis is reasonable? This is the question. One may formulate it differently: if I base a conclusion on a one-sided test, will the readers/reviewers be convinced that I would NOT have reported a "significant" result when the effect was to the opposite direction?
As many have suggested above, the choice of whether to use one sided or two sided test does not dependent on which one give you lower p value. Rather it depends on the question you ask. For instance, if you have two distributions A and B, and you want to know whether A is significantly higher or lower than B, then you should use a two tailed test. On the other hand if you a priori know that A can either be equal or higher than B (A cannot be less than B), and you want to test whether it is indeed higher or not, then you should use one tailed test.
Thanks very much for everyone's input. I read through all replies and they're all very helpful.
Let me put more details about what I'm testing:
I'm working on a project trying to identify genetic variants associated with a certain disease. The association has two directions, decreasing disease risk or increasing disease risk. So each genetic variant, it has two scores, a positive score for decreasing disease risk(protective) and a negative score for increasing disease risk (risky). The negative score is just for the purpose of using a two-tailed test and it also can be positive if it's in a one-tailed test. So the higher the score, the more protective; the lower the score, the more risky. We're testing hundreds and thousands of such genetic variants so we did hundreds and thousands permutation tests to achieve the permutation p values for each genetic variant. We're using the permutation p value as the final score for each genetic variant we're testing. So the best protective variants will have low p-values and best risky variants will have high p-values.
I thought the best choice would be using a two-tailed test. But when I read that a two-tailed test is testing if A and B are different I got confused. In my case, I'm testing if my genetic variant is associated with the disease, it can be protective and it also can be risky. I want to get the significance level.
Please note if you are going to use the R mt.MaxT package for permutation testing and you have computational performance issues then you should try the R sprint() package. This has parallel implementation of this that has been shown to produce good performance across a range of platforms from Linux PCs to supercomputers and the cloud.
You can find out more about its performance in the Concurrency and Computation journal article at
http://onlinelibrary.wiley.com/doi/10.1002/cpe.1787/full
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics