You can also calculate the 95% confidence interval of the proportion of subjects that scored inconsistently, i.e. 11 out of 38.

The 95% confidence interval (CI) is calculated as p ±1.96√[p(1-p)/n], where p is the proportion of 11/38 = 0.29 inconsistent scores, and n the sample size of 38 subjects.

I calculate a 95% CI of [0.14, 0.43]. The confidence interval does not contain the value 0 (zero), so you can conclude that there is a 95% probability that the proportion of inconsistent scores is larger than 0.

Either way, it does not look that your two tasks are performing perfectly. Or, interestingly, the two tasks do not measure the same trait, and perhaps the subjects performing inconsistently are a subgroup worthy of further investigation!

Here is my file with the 38 participants and the 11 ones who do not present the same behaviors across tasks.

I do not know what actually measures you are dealing with, but it seems that they are dichotomous. I think that the best way to deal with this is to do a McNemar test, which does exactly this. it checks for differences on dichotomous variables (usually in before-after designs)

I guess your answer is already in the data: the two tests give inconsistent results in 30% of the subjects tested!

You have a nominal independent variable, i.e. the task presented (two categories: "Task 1" and "Task 2"). The dependent variable can be taken to be the subsequent test scores: "consistent" (DD and GG) or "inconsistent" (DG and GD), and that can be measured as a frequency (number of subjects).

You can, a priori, state some accuracy level of the two tests' consistency, say 99%, meaning that you would expect in 99% of the subjects a consistent score of DD or GG. You can then perform a Chi squared goodness-of-fit test where you test the observed frequencies of 27 "consistent" and 11 "inconsistent" against the expected frequencies of, respectively, 0.99 * 38 = 37.62 and 0.01 * 38 = 0.38.

I then calculate a Chi squared of 299.8, which at 1 degree of freedom has a probability of 3.64E-67, i.e. as good as zero, of occurring under the null hypothesis of an expected 99% accuracy. So it would be virtually impossible that your results were obtained with two tasks that are 99% consistent.

When I run the test again but then with an expected accuracy level of 80%, I obtain a Chi squared of 1.901 at P = 0.168. So your results would be compatible with two tasks that are 80% consistent.

You can also calculate the 95% confidence interval of the proportion of subjects that scored inconsistently, i.e. 11 out of 38.

The 95% confidence interval (CI) is calculated as p ±1.96√[p(1-p)/n], where p is the proportion of 11/38 = 0.29 inconsistent scores, and n the sample size of 38 subjects.

I calculate a 95% CI of [0.14, 0.43]. The confidence interval does not contain the value 0 (zero), so you can conclude that there is a 95% probability that the proportion of inconsistent scores is larger than 0.

Either way, it does not look that your two tasks are performing perfectly. Or, interestingly, the two tasks do not measure the same trait, and perhaps the subjects performing inconsistently are a subgroup worthy of further investigation!

Thank you very much Peter. , you just saved the life of my paper !

How can I report that in the manuscript ? Would something like :

"The 95% confidence interval of this proportion (from .14 to .43) falls above 0. We can therefore conclude that the proportion of participants who are not classified within the same category using the two methodologies is significant at a p < .05 level".

And Peter : Is it possible to report power analysis fort that kind of statistics ???

I think I would write something like: "The proportion of participants that score inconsistently on the two tasks is 0.29 (95% CI [0.14, 0.43]). This is significantly different from 0 which value would be expected when the two methodologies' scores are perfectly consistent, i.e. "DD" and "GG"."

You can do a power analysis with the free software G*Power, but you have to define an effect size that you want to detect a priori. You can also do a post-hoc analysis: when I do a quick calculation with G*Power on a group size of 38, alpha = 0.05 and a "constant proportion" of 0.01 (so assuming that the two tasks are inconsistent in 1% of the subjects) I get that this experimental setup could have detected an effect size (i.e. proportion of inconsistent scores) of 0.10 with a power of 80%.

I am not sure what this power analysis would add to your findings, though.

http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3

Thank you again Peter. A lot !!!

you may get the request by reviewers to run a simulation or bootstrapping, i.e. draw 1000 times, how often would you get 11 subjects being inconsistent. As Peter mentioned, if the two tasks have (according to literature) only 80% correlation, your 11 subjects are not significantly deviating from drawing a random sample. The two tasks you are using should have a reliability and validity score which cannot be 1, more likely something .80ish. Would be careful to over-interpret the 11 subjects, esp. on a dichotomous scale (Binomial test!)

just do a Qui-square independence test for paired samples (aka McNemar test)

McNemar's test is for paired proportions, for instance to test whether two diagnostic tests come up with the same diagnosis equally frequently in the same sample.

Catherine's question (as I understand it) was not whether the two tasks gave the same result per sample (i.e the same proportions "G" and "D" in the two tasks), but whether they consistently gave the same result per individual (i.e considering only the combined scores "GG" and "DD" for the two tasks).

According to the data Catherine provided, Task A scored 12 "G" out of 37 (32.4%). Task B scored 15/37 "G" (40.5%). These proportions are already quite the same. I calculate a McNemar's Chi squared (without Yates' correction) of 0.818, with P = 0.366. Not a statistically significant result, the proportions of "G"-scores in the sample do not differ between the two tasks, but it is no indication of whether the two tasks perform perfectly at the individual level.

Exactly Peter. Thank you again. I have (re)sent the paper, which was under revision.

As a matter of fact, I started with chi-squared but realized that it did not reflect what I wanted to show. Exactly as Peter stated it shows that the two distributions are the same or not but it does not say anything about the individual level matching.

It seems to me that you cannot address this problem properly without knowing the repeatability of both of the tests. In other words, what is the chance that the apparent inconsistency is a consequence of the "poor" repeatability of one or both tests? As I remember it, handedness tests have only moderate repeatability, with some people being strongly one-handed, some people are ambidextrous, and many are somewhere in between. Thus, some people will reliably give you the same result (the strongly handed) while others will often flip (the ambidextrous). (as an aside, for ocular dominance it is a mess)

The McNemar change test is considered a paired test, though the equation appears to not make use of the pairing. However, if you think about the expansion of the contingency table, it produces unique pairing. So, the pairing is really there in the contingency table. I made up some data (3, 1, 10, 24) and got p=0.007 from the McNemar test. Another approach is conditional logistic regression for matched pairs. Using that same data (transformed into the long form) I found p=0.03.

Even so, I think that the repeatability of each test is the underlying issue here.

Yeap, I believe Russel Woods is correct!...

Now, for some thinking out of the box: if you wan't to see how individual differences work out, you could do a Latent Growth Model (LGM). The problem is that you need at least 3 repeated measures per test, so if you could apply the test again, you could run a LGM on binary data and see how variance around the intercept mean (proportion) would turn out. If it was significant than you could conclude that there was significant variation between subjects. You could also see if the slopes of of the 3 measurements vary significantly between application. If the variation around the mean slope was significant, that would prove Russel point that there is a question of the reliability (repeatibility of the tests.

I tested the repeatability. One of the test is quite reliable and the other not.

Sounds like you have found your next study. You can either try the process again with another set of respondents to determine if the un-reliable test is truly un-reliable, or choose another test (or develop one of your own). Even the best laid plans of mice and men.....

Don't get discouraged if your data does not turn out as anticipated. Re-read all the responses above, there are some very good observations you can glean from them. You never know, you might find something interesting within your data set that gives you inspiration for an outstanding future study.

May your 'p-values' always be low, as your RG score continues to grow! :D

I've already developed a test of my own Jared :-) I'm now waiting for the feelings of the Reviewers about it . Thanks for your encouragment.

Dear Peter (and others if you can help me)

I've received the decision from the editor and he is happy with the confidence interval analysis. So thank you very much. However, as I expected, he would like a power analysis. I have downloaded the G-power tool but I do not know exactly what to enter in the different fields and I do not know how to report the results. Can you explain step by step how you reached your conclusion ?

Many thanks in advance.

Dear Catherine,

Congratulations on the Editor's decision!

I attach a screenshot of the G*Power window, hope this helps.

Salut,

Peter

As the sample size is small you should not use the binomial approximation to the normal test. You could use the sign test.