Well, the four by four table has rows and columns labelled TP, FN, TN, FP for True Positive, False Negative, etc. The upper left 2 by 2 table within the 4 by 4 table is what you need for sensitivity and the lower 2 by 2 table within the 4 by 4 table is what you need for specificity. You may have to do a bit of recoding to convert the actual test result to TP, FN, etc. (example TP if test is positive AND gold standard is positive).

For each of the three diagnostic tests, you will get a count for TP (True Positive), FN (False Negative), etc. Compare the TPs and FNs for two of your diagnostic tests in a two by two table. The upper left corner of the two by two table would be the number of patients in which both tests were TP and the lower right corner would be the number of patients in which both tests were FN. You're interested in the off-diagonal elements of the two by two table, where one test is a TP and the other is a FN. Compare these using McNemar's test.

The Likelihood ratios are a bit trickier, but you can get tests for these as well. If you show that Sn and Sp are the same across all three tests, perhaps you don't really need to show it for the LRs.

Thanks for the response Stephen. But i am not able to prepare a 2x2 table for the data. When i compare the test results of two of the tests I get a 4x4 table. I have enclosed a comaprison table of two of the three tests in the attached excel file. Kindly go through it and help me out. Thanks.

On each trial it either can be a target, or not, the diagnostic test may say target, or not, it is from one of 3 tests, and there is patient number. So, I am imagining the data file to have 4 columns and 150 rows. Is that how it is formatted? Also, if a patient is a target, are they always a target (i.e., are these tests for the same thing?). I will assume these are for different things.

The standard 2x2 method with TP, FN, etc, (also called Hits, False Alarms, etc) can be written as a regression (in psychology often a probit regression, in medicine often a logistic regression, these are similar except at the ends). So, if you had just one set you could do

logit(is a target) = ln(is a target/is not a target) = B0 + B1 (diagnostic test says target)

Most signal detection methods books will describe this (for a good paper see http://www.columbia.edu/~ld208/pub.html ... and http://www.columbia.edu/~ld208/ has lots of resources).

Since you have multiple sessions you could run as a repeated measures logistic using mixed (also called multilevel) software. With three tests you'd probably include test as a fixed factor.

There are several ways you might use to compare the tests. My first thought is different from yours. I might ask if the B1 from above is different. The interaction between the test and the B1 above will be one way to explore if the tests discriminate similarly. From your question you say sensitivity and specificity. If you are interested in these on their own, you would only do this for part of the data. Comparing LRs is also different.

ps. Re-reading the question, it looks like there is one gold standard and 3 diagnostic test. For this, just have 50 rows, and the variables would be the gold standard and the three tests. Predict the gold standard from the three tests. In standard software the coef associated with each test will be after conditioning on the others. So, the p value will be whether each adds more. You can test them in pairs. If you use a forward stepwise method, the software will add the one with the highest LR, but this won't mean that it is significantly better.

Well, the four by four table has rows and columns labelled TP, FN, TN, FP for True Positive, False Negative, etc. The upper left 2 by 2 table within the 4 by 4 table is what you need for sensitivity and the lower 2 by 2 table within the 4 by 4 table is what you need for specificity. You may have to do a bit of recoding to convert the actual test result to TP, FN, etc. (example TP if test is positive AND gold standard is positive).

Stephen the 4*4 I had attached was a comparison of two of the three tests I have used. But your latest response has given me a way out. Thanks. I will use the McNemar for th

e 2*2 as you have suggested and see if that solves my problem. And I'll surely get back to back to you here. :)

So you've got four columns of data. one is the gold standard and the other three are the diagnostic tests. If the gold standard is + and a diagnostic test is +, code it as a TP. If the gold standard is + and a diagnostic test is -, code it as a FN, etc.

Stephen, are you suggesting taking the 50x4 table and then creating three separate 2x2 tables and then comparing the statistics based on each of these in the same way as you would if these were from three different groups of 50 patients, or have I missed something in the discussion?

Dear Inaamul Haq,

To compare the LRs, you can use the results of the manuscript "Roldán Nofuentes. Comparison of the likelihood ratios of two binary diagnostic tests in paired designs. Statistics in Medicine 26, 4179-4201".

It is possible that the sample is very small. The method can not be applied if there are many frequencies equal to 0.

Good luck

Good idea Paul!

You use a variant of McNemar's test, because there is dependence between observations on the same sample.

The definitive test establishes your truth standard, You can then compare each pair of putative tests by coding them as correct or incorrect. When working with sensitivity you only use the true positives (positive on the reference diagnostic). For a pair of tests, T1 and T2 you have a table as follows

                                  Test 2 correct      Test 2 Incorrect

Test 1    Correct           a                              b

Test 1    Incorrect         c                               d

To test sensitivity this table is just for true positives. To test specificity the table is just for true negatives.

Now, your analysis focuses on the discordant values b and c. If the sensitivities of the two tests are equal then you should have an equal proportion discordants correct for Test 1 and for Test 2. Your test of equal sensitivity is then a test of the hypothesis that b follows a binomial distribution with denominator (c+b) and probability 0.5. For large numbers you can use a Chi square approximation, but you can also use an exact test (which is pretty conservative). You might like to read this paper:

"Fagerland, M. W., Lydersen, S., & Laake, P. (2013). The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Medical Research Methodology, 13, 91. http://doi.org/10.1186/1471-2288-13-91"

You can do this to produce pairwise tests for each pair of diagnostic tests. You can use a similar approach for specificity. You can generalise this in lots of ways (there is a very nice non-inferiority approach due to Tango).

Pepe's book Statistical Evaluation Medical Classification Prediction, OUP, 2003 has an easy section 3.3 on this topic.

You can find the book on Amazon at  http://www.amazon.com/Statistical-Evaluation-Medical-Classification-Prediction/dp/0198565828

This isn't the only approach, but it is simple. Because the only information is drawn from discordant numbers, power can be fairly low when the tests are mostly in agreement (you get few discordants then). Good luck.

Thanks Mervyn. Thanks Stephen, once again. I went by your advice and have now completed the analysis and the comparison. So I now write in the 'methods' section of my manuscript that I used this particular method for comparing sensitivity and specificity. But I need some reference for this. Can you suggest a few references? 

Hi

If you have used the Mcemar test approach, then quote 

"Fagerland, M. W., Lydersen, S., & Laake, P. (2013). The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Medical Research Methodology, 13, 91. http://doi.org/10.1186/1471-2288-13-91"

Yes I did use the Mcnemar approach and I am reporting the mid-p values. But the paper you are quoting does not tell me that this is the way sensitivities can be compared.??? Or is that portion just understood and i am being foolish?? I feel stuck:(

Not foolish at all :). The McNemar test is for paired binomial comparisons, and that is exactly how you are comparing sensitivities. So just comment that you compared sensitivity and specificity using McNrmars test for paired binomial responses, and quote tge paper. Good luck :)

Thanks Mervyn. That has been great help.

Multiple comparison in Area Under Curve (AUC) or Receiver Operator Curve (ROC) (it´s the same thing)SPSS output generate tres AUC graphs on one and it generate an sensitivity and specificity values list where you can visually inspect and select the "best" (best balance between sensitivity and especificity) reached by every test, I recommend you to check the chapter "Diagnostic tests", If you cannot access IBM-SPSS download EpiDat 3.1 (WHO-Spain Health Committee, 2006) and learn how to use it for multiple comparisons (it´s easy) in "diagnostics test" (simple and multiple comparison) and this software have a good help (in spanish, unfortunately) or use it to calculate LR´s, Youman index, IC95% for S, E, Predictive values, etc.

Norman. Spanish is a problem for me ;). I tried googling EpiDat but all i get is Spanish websites. But your answer here has interested me. I dont know how to compare AUCs using SPSS. But i did run AUC/ROC with my data (binary, yes/no) and to my surprise the AUC was low even at sensitivities and specificities of over 60-70%. Why does that happen and what does that tell me about my data?

You can compare AUCs in SPSS introducing the tree tests variables in the "bigger box" (submenu screen ROC Curve in SPSS) it will accept "a lot of" variables, and your dependent variable in the small box down...check for "ROC Curve" and check "With diagonal reference line", "Standard error and Confidence Intervals" and check on "Coordinate points to ROC Curve",; EpiDat 3.1 installation give you language options (including english), because report mutliple comparison and determine statistical difference between test through non-parametric statistics (p

It really depends where you are in your research, and on the nature of the test. If your putative diagnostics are marker concentrations (or even a multivariate  index assay), then what cobstiyutes a psitive result depends on where you set the thresholds. If you are at an early stage, you may well be better comparing the ROC aucs, or partial AUCs. If yiu are at a later stage of research, with either a defined threshold or a categorical test you need to compare sensitivity and specificity.

ROc auc comparisons are probably easiest tperform with a bootstrap. You bootstrap the difference in aucs between diagnostics as you resample the units.

wharever analysis you use must take accouht of the fact that you are measuring all diagnostics on the same subjects.

Stephen, Mervyn : the comparison of the three "experimental" tests to the gold standard ARE NOT INDEPENDENT ! Your estimation must account for possible (probable) correlation between experimental tests ; your  test procedure (if you believe in hypothesis testing) must account for this if you want to control the global error rate. The paper pointed by Nofuentes (its author, BTW:) is in the right direction.

Going to a logistic regression model is a good idea. I'd probably go for a Bayesian estimation...

Comparing paired AUROCs is also a good idea ... if on only if one can build such a curve, either explicitly (i.e. the test results is whether some numeric variable is above or below a given threshold) or implicitly (the test procedures give you a way to classify your sample for "less likely" to "most likely"). Nothing in Inaamul's question hints at this.

 1. To check for agreement between tests, use kappa statistic for multiple tests with dichotomous outcome (+, -)- quantitative. https://www.graphpad.com/quickcalcs/kappa1/?K=2

2. Then use Cochran's Q test (SPSS is best): if there is no statistical difference between the methods, stop here.

2. If there is statistical difference, use postHoc McNemar's test as pair-wise between methods (such 1 & 2, 1 & 3, 2 & 3) to check which test is making the difference. PosHoc Dunn's can alternatively be used.

See: Article A Comparison of the Efficiency of Three Sampling Methods for...