One-sentence summary: "Results from analyses are different if you apply a continuity correction or not."

On a quick perusal, they created "gold standard" results by calculating the tests "by hand" which actually meant in Excel!

Given that Excel is famous for the correctness of its statistical calculations – the these hand calculations would surely have involved some reliance on Excel's ability to calculate densities of distributions, for example – I'm not at all convinced. But comparisons between packages are interesting, and they have made all the code and output available. The initial discussion also looks very useful.

But I need time to give it a good read.

Ronán Michael Conroy , I know. I was really hoping that "by hand" meant by hand, and then looking up p-values in a table.

I gave the article a pretty cursory read, but here's what I got.

To me, the main results are in Tables 4, 5, and 6.

(2) is kind of interesting, but I don't care.

(3) probably means nothing, and is probably a result of the software's ordering of the groups differently, like in a Wilcoxon-Mann-Whitney test. Like, if they're alphabetized or taken in order presented. It could be a result of one software package not following the convention that a negative t or z value implies that that the second group has larger values than the first group. This would be annoying, but can't imagine caring much about this.

On (4), I might be missing point. But if not, this almost entirely silly. It just has to do with the fact that different packages apply different default options. Granted, this could lead to meaningfully different results. But this kind of assumes there should be some correct default approach.

So, who is spearheading the letter to the editor on this ?

Ha! It sounds like you're volunteering, Sal Mangiafico! I still have not had (or taken) the time to read the article properly, but I'll get back to you when I have. ;-)

Not surprising. I'd note:

1) There's almost never a reason to use Fisher's exact test or the continuity correction in any realistic case. (This is one of the few R defaults that is very annoying - it's a terrible default for the 2x2 chi square.) The issue is that both assume fixed marginals which is unrealistic in almost every practical application of a 2x2 contingency table.

2) Chi square tests are parametrc

3) Excel calculations as the correct baseline seems ... peculiar.

4) There are more versions of the 2x2 test I would guess than any other test ... certainly I've come across at least 30-40 variants andI suspect there's 100s. They make different assumptions or have other properties that make it obvious I think that p values will differ.

I still have not finished reading the entire article, but I've read enough to find several things to pick on. E.g., in table 1, the authors describe parametric "equivalents" to the non-parametric tests/procedures they chose to examine. They also promote the (mistaken) notion that the WMW and KW tests test null hypotheses about medians (rather than means).

Table 3 purports to show/describe output for "default statistical procedures". The Stata command shown for analysis of a two-way contingency table is:

tabulate v1 v2, chi2

IMO, this is not a default. Stata has been described as type a little, get a little. The basic, dare I say default tabulate command is:

tabulate v1 v2

This produces a contingency table showing the observed cell counts, period. If you want more, you have to add a comma and whatever options you want. Arguably chi2 is the most frequently chosen option for those who want to report a test, but there is nothing default about it, and there are several other options. Here are a few of them:

• chi2: report Pearson's chi-squared
• exact[(#)]: report Fisher's exact test
• gamma: report Goodman and Kruskal's gamma
• lrchi2: report likelihood-ratio chi-squared
• taub: report Kendall's tau-b

Like others, I am very perplexed at why Excel was used to perform "hand" calculations. I am not familiar with (SAS) JMP, so I do not know if it could be used for "hand" calculations. But I'm sure they could be done using SPSS, Stata, or R. In addition to avoiding suspicion about Excel's computational accuracy, this would allow one to document exactly what was done as code.

At this point, I don't know if I'll bother reading the rest of the article. I'm just not sure it's worth the time, as the main message seems to be what I said in my original post: Users of statistical software should Read The Fine Manual so that they understand what the output is showing them.

Having said that, the authors also encourage checking results by carrying out tests/procedures in 2 or more packages. I do agree that this can be a useful practice. But in my experience, when two packages have given different results, the reason for the discrepancy is generally found by RTFMs. ;-)

One of my courses allows students to use JASP or R, and they have different defaults for the 2x2 chi-sq. So of course the assessment question has data that is one side of p = .05 with one, and the other side with the other. It is one they have to put the data in themselves, so until they look carefully they think the other person must have entered the data incorrectly. (ps. have not read the BRM paper, but this is a nice thread)

Conducting nonparametric analyses with SPSS might lead to complex and valid results, but using R will be perfect as it will provide a simple valid result compared to the other statistical software results.

Fajinmolu Abdulbasith O , why do you think R is "perfect" (and what even is "perfect." Here is a counter example

chisq

@Daniel Wright, I think R is perfect by using some advanced library packages like pgirmess and nparLD in conducting nonparametric analysis in R. (perfect = providing justifiable results).

What I notice in other threads like that of #Thom Bagulay is that they compare the default programming of R with other software.

I fear that we will miss what the paper actually shows --- rather than what the paper thinks it shows ---.

These different software packages all return the same results for these kinds of tests if you conduct the same test--- with a caveat for the few minor points I listed above in bullet points.

As Bruce Weaver says, you have to read the manual. (And really, you just need to read the short documentation for the function you're using).

I think the paper authors make one salient point, which --- as Daniel Wright points out --- if you just use the defaults from the software package, you may get different results.

But really, we have to expect users to have minimal understanding of what they're doing. If the R results say, "Pearson's Chi-squared test with Yates' continuity correction", it's really on the user to say, "Huh. I wonder what Yates' continuity correction means".

Adding to the point about defaults, statistical software users (even useRs, I daresay!) need to understand that the defaults are not necessarily the best choices. Thom Baguley already gave the example of R reporting Chi2 with Yates' correction. But there are countless other examples of this. E.g., FACTOR in SPSS and PROC FACTOR in SAS both use the so-called "little jiffy" method by default (IIRC). I.e., they use PCA to extract "factors" followed by Varimax rotation. As Preacher & MacCallum explained so clearly in their 2003 teaching article, this approach is not the one that most factor analysts want.

https://quantpsy.org/pubs/preacher_maccallum_2003.pdf

Uzair Essa Kori, CopyLeaks.com flags your response as AI-generated content (see attached image). If you did indeed use ChatGPT or some other AI-bot to generate your response, why in the world did you not say so at the top of your post? Do you not think that would be the ethical thing to do?

Uzair Essa Kori , I'm sure you are aware that your post is not all relevant to the discussion here. It doesn't address the question at all.

I assume your post is generated by ChatGPT or other AI-generation tool. I'm sure you realize this is not helpful to anyone. I doubt the reader needs a paragraph explaining what "conducted in" means.

Your profile lists that you have a Master of Philosophy in Statistics. Likely you have something useful to contribute to this conversation.

Perhaps Uzair Essa Kori can say why even with a masters in the philosophy of statistics he does not feel he can add something useful, and why he wants to present himself in this way on ResearchGate.

Sorry for any kind of interruptions, No problem at all! If you're looking for a brief idea or concept, feel free to ask, and I'll provide you with a concise answer. If you ever need more detailed information or explanations in the future, don't hesitate to ask as well. I'm here to assist you with your questions and requests.

I used AI just to get a quick idea because of time constraints, otherwise, I can provide detailed answers on my own.

Uzair Essa Kori, I repeat my earlier question, which you have not yet addressed: Why in the world did you not say clearly at the top of your post that it was generated using AI? Do you not think that would be the honest and ethical thing to do?

Uzair Essa Kori :"I used AI just to get a quick idea because of time constraints, otherwise, I can provide detailed answers on my own."

Doubt.....

Okay , Acknowladged

Uzair Essa Kori – you should have explained, then, that you used AI, which was quick, because you had not enough time to learn statistics and programming which would have allowed you to write the answer yourself. And, of course, to know if it contained errors, which it seems to do. ;—})