Hello Snæfríður Sól Árnadóttir. You need to say about your data--providing a small example of what the data file looks like would help, for example. Also, you are talking about True Positives, False Positives, etc. That suggests that there must be a "gold standard". Is it one of the three repeated measures you described? Or is there a gold standard and 3 competing diagnostic tests? What question(s) are you trying to answer with your analyses? Thanks for clarifying.

Method of what?? It's unclear if your intent is to determine which method has a higher success rate or to determine which method is better at diagnosing something. Your methodology will be different depending on which of these you are testing. If you want to assess whether the proportion of successes is the same between methods of treatment, then Cochran's Q may be appropriate. It will test whether the proportion of successes is the same between the three methods.

On the other hand, if you are testing diagnostic ability (e.g., the underlying condition being diagnosed is true/false or positive/negative and the "method" is trying to accurately determine which), you will want to report things like positive predictive value, negative predictive value, sensitivity, specificity, etc., but these all assume you are comparing each method to a "gold standard" method (assumed to determine positive/negative correctly) to see how often you get the answer correct with each of the other methods.

I agree that we need some clarification, both as to what your independent and dependent variables are, and with regard to what your three methods are.

Hi, thank you for replying.

My data would look something like;

Sample nr. Method A Method B Method C

1 + + -

2 + + +

3 ....

I am actually working on two different projects with the same type of data. One has a reference method as gold standard determining whether a result is true or not. Here I have calculated sensitivity, specificity, PPV and NPV.

The other one is smaller and simpler where a positive results for any of the three methods is considered a true positive and a negative results for all three methods represents a true negative. (Is this a huge error maybe?)

I have calculated the sensitivity of the methods here but the data is not able to determine the specificity since I did not test everything but only what I considered "likely" to be what I was looking for. (Bacterial colonies on agar media).

The question I'm trying to answer is whether there is a difference in performance between the methods, and if so which method is the most suitable.