No the chi square stat depends on the entire table. Look at Pearson's original derivation. Perhaps you can modify it some how. Best, David

You could consider loglinear models for your data. Here are some examples for SAS: http://support.sas.com/kb/22/565.html

It makes no sense to limit the independence hypothesis to a fraction of the cells of the contingency table. It is never done. The entire table must be considered.

You can consider the adjusted standardized residuals, for each cell, that calculate the usual statistical programs

Agree with Timo Törmäkangas

consider log-linear models - besides SAS there is glm() in R

you may google "log linear models" directly to get some tutos

Thanks for the answers.

In fact this is just part of my problem.  The whole problem is:

Suppose we have 100,000 discrete variables, each of which has three discrete values  A,B and C. (They are defined over 500 objects).

I need to find pair of variables that are dependent in a very special way, with some confidence (p=0.001, say):

Given two discrete variables X and Y with three values each, A,B, and C,

I am only interested to reject independence in three cells of the nine cells of the contingency table, (A,A),(A,B), (B,A), and I need that  the observed counts be greater than the expected in these three cells.

So, I could generate all possible contingency tables for all pairs of variables, and calculate the usual chi squared score, but instead of adding over all 9 cells, I sum over only 3 cells and only if the observation counts are above the expected ones.

Then, I use the distribution of these 10^10 scores to calculate the p-value of a table: given a table, I calculate the score using the three cells,

and see what proportion of tables has this score or greater (more extreme).

In other words, I see the top scores, that are among the top p= 0.001 percentile, for instance.

Is it correct?

For your setting the elaboration of a permutation test may be tricky and I don't feel too comfortable with your proposal. I would prefer to buid up a likelihood ratio test completely adapted to your situation. I don't think it is too complicated - see the attached doc.

Best.

Thanks a lo Jean Maccario!

We will study this possibility.

Thanks for your quick answer, Jean.

I see you recommend to use the G test instead, and to sum only in the three cells with only 1 degree of freedom.

It sounds very reasonable to me.

I still wonder if we can sum a cell only if the observed counts are greater than the expected ones.

Also, shall we divide the p-value by the number of pairs to do a Bonferroni correction for multiple tests?

On the other hand, the permutation test I suggest can be improved: 

For each random variable X of the 10,000, we could generate randomly the values for the 500 objects with the same frequencies of X. And then consider all possible contigency tables for the pairs and for each calculate the score of the G test for only the three cells. The empirical distribution of this score could be useful to calculate p-values

(Without correction for multiple tests).

What do you think?

And thanks again for your kindness and time.

If you can send to us some references, it would be very helpful.

Salve Jaume,

your answer contains 3 questions

1 - the proposed G test is actually aimed at testing _differences_, testing superiority is possible but necessitates a specific elaboration yet to be done after the choice of a proper H0/H1 alternative;

1.5 - G test is commonly described in the literature, the denomination G is (to my knowledge) due to Bishop, Feinberg and Holland : Discrete Multivariate Analysis Theory and Practice (see C5) but is known elsewhere as likelihood ratio test in many books;

2 - halving the p-value is a short-cut I would not recommend although it could be correct and/or efficient -- to be checked;

3 - your elaboration of a permutation test should lead to a chi square distribution if your data are not to far from asymptotic behaviour.

Best.

We find that the derivation of the  log-likehood  ratio test is correct. Thanks Jean Macario.

However, we wanted to compute it empirically with a Monte Carlo procedure.

We generated 1,000,000 random tables of two fixed marginal distributions and do not get a chi squared distribution with one degree of freedom.

The way we did is as follows:

For each of the two variables  (with three values) we generate 1,000 random permutations of the 400 samples.

Then we considerer the contingency tables for all possible pairs and for each table, we sum only the values of the G-test  (2 O_ij ln  (O_ij  /  E_ij))  for the three cells we are interested in (which is equivalent to what you suggested) instead of the usual 9 cells. 

And we got the histogram we include.

There are thousands of negative values which is estrange, we confirm they are well calculated.

What do you think it could be the problem?

Thanks in advance.

It's impossible for a G-test to give negative values, it is a very general result of mathematics : the supremum of a function in a given space is larger or equal than the supremum in any subspace.

Your computations are probably (ou certainly) correct, but the impossibility remains.

It is not always clear however to see what region(s) of the experimental result space a permutation test explores. Without more details I can't tell.

Regards.

Thanks Jean.

This is what we have now:

Let X and Y be two discrete variables with three values each, A,B, and C.

We are only interested to reject independence in three cells of the nine cells in the table,(A,A),(A,B), (B,A). So we have three null hypothesis: H_AA :  Pr_XY(AA) = Pr_X(A) P_Y(A) H_AB :  Pr_XY(AB) = Pr_X(A) P_Y(B) H_BA :  Pr_XY(BA) = Pr_X(B) P_Y(A)

Given a sample of 500 subjects for the  two variables X and Y, we construct a contingency table T.

Assuming that the table follows approximately a multinomial (instead of the true hypergeometric) distribution with fixed marginal counts, the probability of T is the usual:

Pr(T) = (factorials) P_AA^{n_AA} \times P_AB^{n_AB} \times P_BA^{n_BA} \times \cdots \times  P_CC^{n_CC}

ln Pr(T) = log(factorials) + \sum_{i,j} n_{ij} ln P_{ij},  i,j\in {A,B,C}

If we calculate the Pr for H_0 and H_1 and take the differences of the logs, as usual, we get the usual:

log odds =   \sum_{i,j \in {A,B,C}} n_{ij} ln P_{ij,1}/ P_{ij,0}.

Then if we assume that the null is only on three terms, only those terms survive and we get that sum on only three terms.

But, this is a mistake: If the AA and AB cells are independent, since the marginals frequencies are fixed, the AC cell is independent. And also CA, so we should use 5 terms, not only 3 terms. We use 5 terms and still we get negatives statistics!

So then we realized that if the independence is assumed in 5 cells, there is some sort of independence also in the other four cells, out of the 9 cells.

To make it more precise, we try to get the value in four values of the right-bottom square as small as possible:

min  \sum_{i,j \in { B,C} n_{ij} ln P_{ij,1}/ P_{ij,0}

instead of assuming that the sum just above is 0, which is not.

We can vary only one of the four values in the 2x2 square because with marginals fixed, all the other 3 can be calculated. So we calculate the value x for n_BB, say, such that the sum just above is minimum, assuming that the counts for the 5 independent cells are fixed and the marginals are also fixed, and the counts for the cells are positive. This reduces to a minimization problem with a variable, x=n_BB, say, and restrictions of positivity, that can be solved  numerically (because it has a sum of logarithms that should be equal to zero).

So given a table T, we could recalculate the counts for 4 cells (bottom-right square) to make the table as independent as possible (the statistic as low as possible), and calculate the usual G-test on the table over the 9 terms.

If we do so, how many degrees of freedom would it have? Would it follow a chi-squared? Do you like this calculation? Now it uses lots of parameters, all of them .....

We did an empirical Monte Carlo test, in which we recalculate the 4 cells but not with the minimum (we were lazy) but we fix a value with the greater count n_DE to the expected value as if it were independent, and calculate the other 3 cells.

We did one million permutations and did not get negative statistics! It looks like a chi-squared of one degree of freedom (like an exponential) but it is elongated, so it is not a chi-squared.

Thanks.