Providing an example with toy data would be helpful for us to see what you are trying to do.

Hello Kayleigh Paas. I agree with Sal Mangiafico that it would help if you provided a small dataset illustrating what you want to do.

But meanwhile, I do have some comments on what you have written so far.

1) For your quantitative variables, paired t-tests are only useful for assessing whether there is a mean difference between the raters. Often, people are also interested in the strength of the linear association between the raters, and maybe also some measure of absolute agreement. To capture all of those things, I like to show the scatter-plot and report the following (with 95% CI for each of them): mean difference, Pearson r, ICC. You can see an example in this MedStats thread:

• https://groups.google.com/g/medstats/c/YR-2uLLacrc

2) How many levels are there for the categorical variables you mentioned? Are they ordinal or nominal?

Thanks for clarifying.

Thank you for your answers! We did an ICC for the continuous variables as well, I will take a look at the thread.

The categorical variables are nominal, with values of: 0 no fit and 1 fit OR 1 to 5 with no order (so nominal as well).

We are trying to compare Rater1 and Rater2 so for example:

R1 R2

0 0

0 0

0 0

0 1

1 1

1 1

0 0

We used cohen's kappa to analyze agreement, but we got the advice to look at systematic differences as well by using chi-square. I'm pretty new to SPSS so I hope this clears it up a bit.

For matched pairs of raters, you'll want McNemar's Chi2, which you can get via the CROSSTABS command. IIRC, SPSS will automatically give you the Stuart-Maxwell test of marginal homogeneity when the paired variables have 3 or more categories. But you can also get it via Analyze > Nonparametric Tests > Legacy Dialogs > 2 Related Samples, as shown here:

• https://community.ibm.com/community/user/ai-datascience/discussion/stuart-maxwell-test-of-marginal-homogeneity-paired-samples

Back to the 2x2 tables. Kappa is often described as measuring agreement beyond chance. A problem with kappa is that as the marginal distributions become more extreme (i.e., further from a 50-50 split), chance agreement goes up. So when the marginal distributions are extreme, it becomes difficult to get even a moderate value of kappa, even if raw agreement is very high. In Chapter 19 in Users' Guides to the Medical Literature (link below), McGinn et al. suggested using a transformation of the odds ratio. Unfortunately, they called it the phi coefficient, apparently not remembering that this is the name for the well known Pearson correlation between two (true) dichotomies. A better name for it would have been Yule's Y, aka., the coefficient of colligation (2nd link below).

• https://jamaevidence.mhmedical.com/content.aspx?bookid=847§ionid=69031494 (with any luck, you will have institutional access)
• https://en.wikipedia.org/wiki/Coefficient_of_colligation

HTH.

Before application of chi square test you have to prepare contingency table for given categorical variables and then proceed further for chi square statistics.

Kayleigh Paas ,

I assume in the example that each row is an item, or other blocking variable.

In that case, yes you want a paired test, like in your example, likely McNemar's test.

But you should also think about what you mean by "systematic difference". For example, if you had:

R1 = (0,0,0,0,1,1,1,1)

R2 = (1,1,1,1,0,0,0,0)

McNemar would not find that to be a systematic difference. But also, R1 and R2 would not be correlated.

I'm not sure if any of the comments address your question. For example, I don't know what you meant by "this is not possible since the outcomes are constant".

Shriniwas Bharati , maybe. A lot of software allows you input data in long format, and analyze it in that form.

Note that Kayleigh Paas mentioned SPSS in one of her posts:

"I'm pretty new to SPSS..."

In SPSS, you need one row per subject (i.e., the independent units of observation), and the paired values are in two separate columns (or variables). You can see relevant screenshots here:

• https://www.statology.org/mcnemars-test-spss/