The study involves 3 diagnostic methods all performed on the same sample: Test A, Test B, and a reference test. The purpose is to compare Test A to Test B, against the reference. Possible results of the tests are positive, negative, and unassessable. The proportion of unassessable tests is roughly 15% by each test (with significant overlap across tests).
Reported outcomes are sensitivity and specificity with CIs and p-values comparing Test A to Test B. When calculating sensitivity and specificity for Test A, we only need to exclude results that are unaccessable according to Test A OR the reference. However, to calculate a p-value using a pairwise statistical test like McNemar's, you would need to exclude results that are unassessable by any of the 3 tests. Which is the best approach?
a) Report sensitivities/specificites using the more complete data and then report p-values after applying the additional exclusions (the p-values are highly significant and the sensitivies/specificities differ
Hello Mark,
Given that you don't wish to include the unassessable instances as part of the analysis, I'd lean towards your first proposed method (a).
If you absolutely, positively insist on running the dependent comparison, then I'd opt for complete case results.
Good luck with your work.
I would disagree since the tests WERE DONE . . I would say this should be a three group analysis. Otherwise the results are considerably biased by ignoring the third group. If you had a sample of of ten trials with 8 unassessable would you want to risk saying the two tests were the same? I think not. I would suggest a multinomial logistic regression with three groups. Google that topic to see how to run it in your favorite package. Best wishes David Booth
BTW I should have said the x in your regression would be the reference value. Best wishes David Booth
I agree with Dr Booth, excluding unassessable will biased your tests toward optimistic accuracy findings.
A 2 step approach can be used:
- logistic regression explaining variables associated with the probability of an unassessable test
- then looking for test accuracy. A sensitivity analysis could certainly be done looking for the impact of missing data. I'm pretty sure your audience would be also very interested in knowing why some tests do not give interpretable result (this per se is a very interesting result to target animals (samples) with highest chances of being usefully processed by your test).
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA