Dear colleagues in the research community,
As we know, there are two approaches to hypothesis testing of cross-tables: testing for independence and testing for correlation between variables. In both cases, for exact probabilities, we ask the same question: what is the probability of getting "this table" and the "more extreme tables". For independence tests, the traditional exact test is the (dominant) Fisher-Freeman-Halton (FFH) statistic, and for correlation tests, the Mehta-Patel (MP) algorithm is a widely used solution. In some cases, especially when the table is sparse and ordinal, these algorithms give conflicting, if not opposite, inferences. I recently faced a table where the exact probability by FFH could be p = 1, while MP was p < 0.001 because of high correlation. In the attached note, I ponder this issue and compare their strategies. It seems that FFH's result is confusingly wrong, and the reason is the way the FFH algorithm treats tables with the same probability as the one of interest. This claim is strong, and it calls for a larger discussion within the research community about FFH: Should we change the logic of FFH to avoid confusing results? If we should, why? If we should not, why not?
Preprint Note on the deviating exact probabilities. Should we change ...
I had posted these comments on the paper itself, but this is probably a better spot for discussion.
This is an interesting paper. I'm still trying to understand it all.
A few comments:
- I do find the introductory premise of the paper a little confusing. Of course if one test is treating the categories as ordinal, and another test is treating the categories as nominal, they can come to different conclusions (different p-values). For the rest of the paper, I had some trouble disentangling this idea from the idea of the different methodologies.
- I've never heard of the Mehta-Patel algorithm for ordinal tables. Is something that is implemented natively in SPSS ? Or was it something you had to program ?
- I'd be curious to compare the Mehta-Patel algorithm for ordinal tables to, say, the linear-by-linear test, Spearman correlation, Kendall correlation.
Hi!
You are correct in that the interest in analyzing the table HAS to have a role in the interpretation. My concern/confusion comes from the fact that both approaches (FFH and MP) see the same table, but the other one infers that the variables are totally independent (FFH) and the other says that they totally dependent (MP). And, yes, they both are correct, but FFH seems to loose all the information of the table if we could assume even a slightest piece of ordinality in one of the variables.
I revisited the note after I realized that the real problem lies in the fact that FFH is bound to veeery ineffective "discrepancy measure" in comparison with the MP. FFH uses Chi-Squared statistic (or any other nominal-nominal statistic) as a proxy for the probability while MP uses much more specific proxy (some of the ordinal correlation, maybe, like D or G). That is: if a table produces equal or higher chi squared value (or correlation in MP) its probability is equal or smaller than the probability of the table of interest.
My proposal in the renewed note is that we could come into middle of these, and use Freeman's theta (your name is mentioned in the paper...) which is a "semi ordinal" measure. If assuming that even ONE of the variables is ordinal in nature (note that dichotomous variable always is ordinal) and using theta as a proxy instead of chi squared, the interpretation this kind of "ordinal FFH" would be the same as with correlation approach. As known, theta is a spacial case of Somers D, and I noticed by your package that theta is, factually, an absolute value of D(gX) which looks the table in the direction that the variable with the wider scale (X) explains the order in the variables with the narrower scale (g). That is, it assumes that only g is ordinal. This is, actually, true when g has just two categories. Then, using theta as a proxy seems to give a correct result always when one variable has just two categories. Seems so. I feel this interesting.
- Well, I think the starting point has to be the analyst deciding whether the categories are going to be treated as ordinal or nominal. And sometimes it makes sense to treat a variable as ordinal in one analysis and nominal in another. (Because you can test either hypothesis, and either might make sense).
- I really don't like the term "semi-ordinal". Freeman's theta and the Cochran-Armitage test --- and really Kruskal-Wallis --- are for ordinal x nominal designs. It's just about the analyst deciding whether to treat their variables as nominal or ordinal, and then choosing the correct hypothesis test and effect size statistic.
- I don't think the difference has anything to do with the wider or narrower scale. You could have 5 nominal categories and 3 ordinal categories.
- One thing I've never quite understand about Somers' D is that it has you designate if it's (row | column) or (column | row). I mean, I just don't really understand when you would use this vs., say, Kendall correlation. Not a criticism of the statistic. Just something I don't know when you'd use this vs. another effect size statistic.
- Right, a dichotomous variable can be treated as nominal, ordinal, or continuous. It doesn't matter. You can perform Pearson correlation on two dichotomous variables. We call this phi, but the math is the same. And so on.
Sal Mangiafico Good to continue the discussion. Obviously, I agree most points. I need to think about this "semi-ordinal" twice. My point was that (unlike KW test), theta is a coefficient of CORRELATION, however, not fully, I think. Hence, "semi". But I'm not bound to that.
When it comes to the direction, we are starting to talk about important and crucial matters. I came up with the topic when I studied Somers D (Article Somers' D as an Alternative for Rit and Rir IJEM 2020 FINAL
) and noticed that only direction that makes sense in the measurement modelling settings (my area) is the one where the score X (i.e., the variable with the wider scale ) explains the behavior in the test item (g), that is, the direction DgX or "g given X" which usually is notated in software packages as "X dependent". This is the same direction as the standard eta squared in the GLM settings (Article Artificial systematic attenuation in eta squared and some re...
), the traditional Pearson-Bravais product-moment correlation coefficient R (Article Directional nature of the product–moment correlation coeffic...
), and Goodman-Kruskal gamma (Article Directional nature of Goodman-Kruskal gamma and some consequ...
), although the latter two are traditionally taken as symmetric measures because they produce only one estimate regardless of how the variables g and X (or X and Y) are declared. Kendall tau-a, tau-b, and tau-c are truly symmetric measures, and hence, they are not comparable with DgX, GgX, etagX, and RgX. In D and Eta, we can choose the direction but G and R give only one direction: X explains g. The direction of R is easy to verify by eta because algebraically R equals only ONE direection of eta. G was more tricky to show. To me, both relations were quite a mindblowing findings.
The term "correlation" --- like all natural language words --- is "vague on the outside and muddy on the inside".
To some people "correlation" can only refer to Pearson, Spearman, or Kendall correlation. Others might add those appropriate for dichotomous variables (like phi).
I would use the word "measure of association" for things like Freeman's theta. Also for Cramer's V. And probably things like Kendall's tau is I'm dealing with ordinal variables.
I think in naive English, "correlation" just means "association" in some vague way, like "ANOVA showed there was a correlation between height and gender."
I don't place much importance on these words. But I've seen the use of "correlation" cause confusion, if people have in mind a limited definition.
I think, ultimately, it comes down to the analyst deciding if the variables are being treated as nominal, ordinal, or interval/ratio, and that an appropriate measure of association is being used.
I agree. As not being native in English, I do not hear the small nuances in these words. I thought the matter from the etymology view point: 'correlation' comes from 'co-relation'. From this viewpoint it may come close to 'association' i.e. Latin associare ‘to unite, ally’. In some of my writings, I use association instead of correlation and a referee pointed that I should use correlation. I still have some problems to swallow it because my mind is indoctrinated to associate 'correlation' with Pearson and Spearman because they are based on 'covariance' i.e. 'co-variance' while the rank correlations are based on probability (or proportion of observed to maximal value). From that point of view, 'association' may be better for those (including theta).
But then we have Chi squared and other nominal-nominal statistics... Definitely not 'correlation'? 'Association' maybe? This was my problem in the note: how to separate the correlative and dependence statistics in the text. Hence, 'independence' and 'correlation' (and not 'association'...)
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training