If the value is less than 100%, you can reject the null hypothesis that it is 100% (at any level of significance, providing 100% is the max possible). So your significance test is trivial if your null hypothesis is 100%.

Whether the difference is meaningful depends on the context and is up to you to define. If you think being lower than 95% is meaningful, then do a test of proportion for this value or calculating the confidence (or some other) interval and see if it is different from you smallest meaningful amount. I will wait to see how you define meaningful before addressing specific procedures.

It depends upon the number of measures you have.

If you have 10 measures, 9 in agreement and 1 not (this is about 90% agreement), the Gtest for the difference between observed and expected (100% agreement) is about (after Yates correction): G=5.249, df=1, p=.02205

If you have 100, 90 and 10, G= 112.22, p

Professor Daniel Wright is totally right. I must add in this case, your result is not statistically significant, it is mathematically certain. Your result is 100% right.

I am wondering whether you could calculate the correlation between de 2 methodes.

If it is exactly 100%, any proof is left over; otherwise, the aforementioned "t" test would be adequate, in addition to the "Chi square" test.

Thanks so much to all of you for your insights.

Daniel Wright, I do believe 95% would be as appropriate of an percentage agreement cutoff as any other to represent meaningful lack of concurrence. It is somewhat arbitrary, but not any more arbitrary than a p-value of .05, I suppose, and I can find no guidance to suggest another level. In practice, folks behave as if the scoring methods were interchangeable (which is why the original thought was to compare the concurrence level to 100%) although, if asked, I don't think anyone would state that they believed the concurrence level to actually be 100%. Also, a 90% level as a cutoff seems too low, but again, it feels very arbitrary.

Jos Feys We just have two different scoring measures for one instrument so I'm not sure if this has any implications for the use of the Gtest you mentioned. It is not one with which I'm familiar. Regarding use of a correlation coefficient, such a coefficient is sure to exhibit statistical significance with a sample size of 298,000. Thus, it would suggest that there is an association between the two methods (and of course there is), but I don't believe it would necessarily answer our question about whether the two different scoring methods for the measure exhibit a meaningful rate of disagreement.

Two things. First, you likely want to take into account both variables (e.g., with an odds ratio), but if you are interested just in % agreement, then suppose you have 268587 of 298000 in agreement. In R

prop.test(268587,298000,p=.95)

gives the confidence interval and p value. Of course with a sample that size a point hypothesis is probably not warranted. Is that the sample size you have.

If you square the correlation coefficient you can report the proportion of variance explained by the methods on each other.

https://en.wikipedia.org/wiki/Coefficient_of_determination

You will have your values in form of 0 and 1 and then you can apply one sample t-test that will work as a test of proportion.

Daniel Wright , thanks so much again. I just ran the test of proportion, and the results are exactly what I was looking for. And thanks again to everyone for your insights. Everyone made great points that were helpful in thinking through this analysis. This community is great!

Melinda Gaddy , glad to help. Just one final note. When I was saying you might not just want % agreement, but diagnosticity/association, two measures can have 95% agreement but not be associated:

Rater 1

N Y

rater 2 N 0 2

Y 3 95

This is an old issue that has been in the literature since at least the 19th century. THere is was predicting tornadoes. But if you are just after raw % agreement then prop.test does what you want.

I'm a bit late to this thread, but I did wonder if something like Cohen's kappa (with a 95% CI) might be more interesting and useful than raw agreement.

Melinda Gaddy, it sounds like you may be looking at a 2x2 table showing Yes/No for Method 1 vs Yes/No for Method 2. If so, can you show us that table (with the observed counts)?

Cheers,

Bruce

Bruce Weaver , absolutely. It is below:

Method A = 0 Method A = 1

Method B = 0 52796 12006

Method B = 1 13516 220535

Well, the spacing I included wasn't maintained. Imagine the "Method A = 0 Method A = 1" row of the text above shifted to the right and aligned over the figures. Lol.

My suggestion would be to use percentage of positive and negative agreement. I've been persuaded by John Uebersax that statistics like kappa can be misleading because they can mask low positive or negative agreement (especially if most agreement is positive or negative).

I wrote up some R functions for this (also contains the link to John Uebersax' web pages):

https://rpubs.com/seriousstats/popana

Thanks so much, Thom Baguley ! I will be checking this out as well.

Thom Baguley, if I understand your concern, another way to address the problem of having most of the agreements being either positive or negative is to use Yule's Y, aka., Yule's coefficient of colligation.

• https://en.wikipedia.org/wiki/Coefficient_of_colligation

Some folks I worked for many years ago used this measure in place of kappa when most of the agreements were either positive or negative, but unfortunately, they called it phi. (I suppose they must not have remembered that phi was already used to refer to Pearson r computed on two dichotomous variables.)

I used two user-written commands for Stata (-kappaetc- and -similari-) to compute Cohen's kappa and Yule's Y for Melinda's table. Full results are in the attached text file. (The output would be unreadable if I pasted it here. But it should be fine if you view using a fixed font.) Here are the key results:

Cohen/Conger's Kappa = 0.7507 (95% CI: 0.7478 to 0.7535)

Yule's Y = 0.7888 (95% CI: 0.7863 to 0.7913)

So for Melinda's data, it does not make all that much difference, IMO. YMMV. ;-)

I'll have a look. Scaling as a percentage is good for interpretability (so much better than kappa I think), but having separate figures for +ve and -ve seems useful extra information to me.

Specifically you can have good overall agreement if +ve or -ve agreement is poor.

So for the sample data in Graham & Bull (1998):

Yule's Y is 0.7959966.

$`95% CI for overall agreement`

[1] 0.8851847 0.9367334

$`Proportion of positive agreement (popa)`

[1] 0.7891892

$`95% CI for popa`

2.5% 97.5%

0.7176887 0.8469847

$`Proportion of negative agreement`

[1] 0.9463549

So I think the popana approach is more informative. Here overall agreement is dominated by the -ve agreement (which is more common) but considerably lower for +ve agreement.

Hi Thom Baguley. I also ran the Graham & Bull (1998) data, and can confirm your Yule's Y result:

Yule's Y = 0.7960 (95% CI: 0.7193 to 0.8535)

Cohen's kappa = 0.7360 (95% CI: 0.6578 to 0.8142)

In your R code, I found these computations for positive and negative agreement:

popa

Yes - the CIs use a Bayesian approach as apparently that has better coverage in small samples. Its sort of equivalent to adding a small constant to the cells if the default prior is used.

I don't see why researchers always want to perform a hypothesis test in situations like this. What would be benefits of applying tests rather than giving a performance/agreement measure? Do they think they would be proving something meaningful by this way?

There is a plethora of such performance measures defined so far, some of which are outlined below, in addition to what are suggested above

https://en.wikipedia.org/wiki/Sensitivity_and_specificity

Uebersax' web pages also contain some precious information.

I agree, Mehmet, that simply providing the descriptives would be as helpful as anything. I imagine, however, that the journal to which I'd like to submit would criticize the work for not fitting into a format conducive to hypothesis-testing. I have seen many an article criticized for posing a research question, but no specific, testable hypothesis.