Let us suppose we have a new cheap and simple diagnostic test we want to evaluate against the expensive and complex gold standard for a highly lethal disease.
The gold standard test is dichotomous (positive or negative), but the new test returns two continuous results: let's call them "Result A" and "Result B".
Assuming the disease can be accurately diagnosed with the gold standard test, we want to
1) estimate the posterior probability of disease given the prior and the new test results A and B, i.e. P(D+|A,B)
2) define the best threshold values for both A and B
Given the high lethality, we're more interested in avoiding false negatives.
Let's suppose we have data like the ones in figure 1 (randomly generated data). Big red dots and small grey dots are patients whose gold standard test did result respectively positive and negative.
Which is the best model to evaluate such a test?
Logistic regression and ROC curve?
Clustering in machine learning?
Other?
Thank you.
In this case, I would use a simple procedure where potential cutoffs for the new test are assessed more or less manually.
(This would be somewhat related to a Cate-Nelson analysis, though not quite.)
If you break both Result A and Result B into two categories each, you would have four quadrants, each of which would have a count or proportion of correct New Test results. From there, you have correct negatives, correct positives, false negatives, and false negatives. Essentially, a confusion matrix for each pair of A cutoff and B cutoff.
You could use a procedure that steps through the potential cutoffs and creates essentially a confusion matrix for each pair. And then order the results by the statistic of interest (% correct, % false positive, etc.).
The reason I recommend doing this more or less manually are a couple. 1) The procedure I describe is entirely nonparametric. 2) You mention that you are interested in avoiding false negatives. This will require some judgement in what cutoffs for A and B are chosen. You may not want to just maximize the correct predictions for the New Test because this might have a relatively high number of false negatives. However, you probably can't just minimize false negatives as this might just lead to the New Test almost always returning a positive result, and thereby returning too many false positives.
You could also break Result A and Result B into three areas each to yield results that are more akin to "positive", "negative", and "indeterminate".
Dear Max Pierini
The availability of a "gold standard" - a reference standard against which to measure the filter's performance – is required for filter validation.
The most often reported indicators of test effectiveness are sensitivity and specificity, which show the likelihood of a positive or negative test result provided that the patient has or does not have a certain ailment.
Kind Regards
Qamar Ul Islam
Getting the posterior probability of the disease is based on logistic regression (and the prevalence of the disease in the target population, which should be a reasonable prior).
The performance of the new method can be evaluated by a ROC analysis. You may decide on a useful combination of sensitvity and specificity you can get (standard indices like the Youden index or similar are ignoring all practical consequences of false positives and false negatives and should not be used).
Having determined a reasonable "cut-off" for the diagnosis, you must estimate the positive and the negative predictive value (PPV, NPV) in the target population. You should do this on an independent data set (data not used for the determination of the cut-off).
Jochen Wilhelm Sal Mangiafico
Thank you for your answers.
I tried different methods and it turned out that, in this case, result "A" is not needed but, instead, the best cut-off involves "B" only.
I took 500 replicates of ROC curve and AUC, randomly splitting the features into 50-50 train-test, to obtain 95% confidence intervals.
ROC curves and AUC of (A,B) and (B) only are not significantly different, so we can use "B" only, and the best B cut-off is 2.
With this cut-off, supposing a prior probability of 0.1 (in the real world, the prior should be estimated carefully for each patient), we'd get a PPV ~ 0.448
I'm curious, Max Pierini: By what criterion did you decide that the "best B cut-off is 2"? Earlier, you said you want to avoid false negatives. To avoid false negatives, you need good sensitivity. What are the sensitivity & specificity values with B=2 as the cut-off? Thanks for clarifying.
I think you can form a 2*2 table (which is common in clinical trials and tests). Based on the table and different cut points, you can calculate sensitivity, specificity and errors and then optimize them.
Dear Bruce Weaver I was trying to choose the best tradeoff between SE and SP, the black X in the ROC curve where SE=0.81 and SP=0.88
But, to minimize false negative rate, would it have been better to maximize SE till SP reaches a defined acceptable minimum?
Let's say SP should not be less than 0.8 (is this reasonable?). In this case, I'd choose SE=0.83 and SP=0.8 and the B cut-off would be 1.725. PPV would be ~0.32 (with a prior probability of 0.1).
What criterion do you suggest? Thanks for your help.
Dear Babak Jamshidi thanks for your answer. How would you optimize SE and SP to get the best tradeoff in this case?
In my opinion, you can perform like selecting the most powerful tests( type I and type II errors); it is possible to set a bound for one of them and optimize the other. For instance, if based on evidence and requirements, you need sensitivity over 90 percent, you can use the cutoff providing sensitivity over 90% and highest possible specificity.
Babak Jamshidi Thank you.
In this case, having SE>=0.9 would mean SP~0.33, so a good true positive rate but a very high false positive rate too. B cut-off would be 0.807 and positive predictive value ~0.13 with a prior of 0.1
Would this be acceptable even with a low SP?
Just curious, what happens when you use Cohen's kappa as some loss function to determine the cut-off? Sort of searching for the point where "random guessing" balances of between TP and TN as you seem not want to focus on either TP or TN. Or perhaps this is complete nonsense?
Mathematically, this is a good decision, but it is on you to find a specific -related to the field- rationale for it as well
Wim Kaijser thank you, interesting question.
Calculating kappa as in figure (is that correct?), the best kappa score ~0.648 is for B~2
Just my two cents (because I think there have been good ideas so far): if you have an idea, how much more important false positives vs false negatives are or if you can quantify how much more important sensitivity compared to the positive predictive value is, you could try
a) weighted Cohen's kappa to penalize FP and FN differently
b) Fbeta-Score to weight Sens and PPV differently.
Maybe this helps!
Now I am doubting if Kappa is the right thing to do. For covid-self-tests FN might not be an issue. In the worst case I sit at home for two weeks without having covid, or do another two tests to make sure (?). So, favoring more FP over FN. For other issues, where it might be more severe to have a FP (e.g, death penalty, I dont want to be put to death will not guilty).
If you mention "highly lethal" disease I tend to favor FP over FN (as this seems similar to the death penalty?). Then again, no idea, not an medical person. You suggest it is a "cheap and simple diagnostic test" than why not favor FP over FN (better to be safe than sorry). If you test P (not knowing TP or FP) then do another test or go for a more "sensitive" test.
Personally, I focus on Kapa, simply because I can't find any reason to care whether I have more FP or FN for the occurrence of duckweed. For me they are equally important, but it is more an exploration of what the model highlights (I mean, nobody is going to use them anyhow).
I am also a bit confused by the cutoff along B (also in "cutoff1.png"). Because this is not the model prediction, is it? As the response (prediction) should range from 0 and 1 (assuming a classification issue), where the cut-off is set at i.e., 0.5. As for now, you have determined where, along B, Cohen's kappa is highest not where the cut-off on the model response maximizes Cohen's kappa (something like ->). This also confuses me in "kappa.png". Yet, again I don't think Cohen's kappa (at least unweighted) has any relevance for the question.
Wim Kaijser That's why I proposed two alternatives to Cohen's Kappa, but you have to evaluate the relative importance of one measure (e.g. FP vs FN) over the other.
@Rainer Duesing, yes you did, I agree. Just expanding on it with some extreme examples.
Wim Kaijser thank you. Yes sorry, I showed kappa and B but not kappa and regression predictions.
I'll try Rainer Duesing suggestions about weighted kappa and F-beta score.
Maybe, in a case like this, is not so important to define a cut-off? Would it be better to provide a table of SE, SP and PPV (or LR+) for each discretized value of B and let the clinician decide, based also on patient anamnesis, risk factors, etc?
Hello Max Pierini. In trying to find the best balance between Se and Sp, you are using Youden's Index (or some variation on it). Let me remind you of what Jochen Wilhelm said in his reply (with emphasis added):
The performance of the new method can be evaluated by a ROC analysis. You may decide on a useful combination of sensitvity and specificity you can get (standard indices like the Youden index or similar are ignoring all practical consequences of false positives and false negatives and should not be used).
The only change I would make to what Jochen said is that Youden's index (or similar) should only be used when false positives and false negatives are deemed equally costly. But you want to limit false negatives (because of lethality), so trying to strike a balance between Se and Sp does not make sense. Rather, you need a cut-point that guarantees whatever level of Se you deem necessary (IMO). HTH.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size