Hi, now I just read a paper from JASA named《permutation approach to testing interaction for binary response by comparing correlations between classes》, it says permutation can lower the FDR rate. However, in its supplementary material, it neglect an alternative case? So how can they prove they can improve the accuracy? Maybe I don't know something throughly rather than their careless, please tell me. Thank you!
This seems to me a zillion dollar question, a chicken-and-egg one, dealing with improvement factors in classes that are not yet defined, while the classes definition depends on attributes that include the "improvement" factors.
In my solution there's an algorithm that deals with it.
Edith Ohri
Home of GT data mining
1) The higher stringency of a test/procedure, the higher is the FDR.
Note: a higher stringency means that it is less likely to get a low p-value (given the same data). This means: higher stringency -> lower power.
2) The fewer information is used, the more stringent is the test.
Note: it is usually a bad idea not to use any information that is available. However, some information is "soft-coded" and enters the test in form of assumptions. If these assumptions are reasonable they are a greatly boosting the power. If the assumptions are unreasonable they are rather harmful (they can decrease the power or introduce bias).
3) Permutation tests use only the available data and thus are methods that use as little "extra-information" (assumptions) as possible. Therefore they will be more stringent (but have less power) and give a lower FDR than methods that make use of (reasonable) assumptions.
Note: the difference will vanish when the sample size increases (the information from the data becomes much larger than the information provided by the assumptions, so the "effect" of the assumptions becomes negligible). For small samples, however, it makes a large difference.
A (small-sample!) example:
You have 2 groups with only 2 values each:
A: (1, 3)
B: (5, 8)
You may want to get the p-value for µ(A) = µ(B).
The typical parametric approach is to assume an independent and symmetric distribution of the the residuals (µ-x), what is used by the t-test. The result is p=0.087. If the implicit assumption of homogeneous variances is skipped the p-value is a bit higher: p=0.158. The mean difference is estimates as 5.0 (8.3; 18.3).*
Adding more knowledge: Given we know that these values are counts, then the expected values could be estimated as the parameter of a Poisson distribution. Doing so gives a considerably lower p-value of p=0.009. The mean difference is estimates as 5.0 (1.3; 9.4).**
Using less knowledge: The only assumption of a permutation test is that, under the null hypothesis, the assignment of the values to the groups does not matter. So all possible permutations of assignments are equally propable under H0. Now there are 6 different ways to assign the given 4 values to 2 groups. The observed mean differences for these 6 cases are -5 -2 1 -1 2 5, the absolute differences are two times each 5, 2, and 1. This is the empirical distribution under H0. The absolute mean difference of the given samples is 5, what is expected with p=0.333 given H0. This p-value is now considerably higher.
--
* Values in brackets are the limits of the 95% confidence interval.
** I used the identity link function to directly assess the mean difference. It is more typical for count data to infere the fold-change instead, what would be achieved by using the log-link. The estimated fold-diffference is 4.3 (1.4; 18.9).
A permutation test tries all possible ways to arrange the data.
If you randomly select different ways to arrange the data you get a randomization test. This is usually done when the number of permutations is large, and will therefore consume considerable computer time. A randomization test will allow the possibility of getting the same arrangement of the data multiple times. However, this is an unlikely occurrence if the number of permutations is large.
The way I would prove accuracy is to use a data set generated with a computer random number generator. Use several distributions. You can even generate a population of values from an underlying distribution and then sample this population. At any rate, as a "fake" data set you know the true values and accuracy is determined by how close the sampled result is to the true values.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis