I am looking at the diagnostic accuracy of a questionnaire in diagnosing a specific type of shoulder pain. For several questions as part of the survey responses can be given as (yes, no, not sure).
How do I then assess specificity, sensitivity etc. if it is not just a yes no answer?
I was thinking of doing a best and worse case scenario but I was wondering if there is a better way to incorporate the impact on accuracy of respondents answering with 'not sure'
Thanks
For multi-class classification, you may use one against all approach.
Suppose there are three classes: C1, C2, and C3
"TP of C1" is all C1 instances that are classified as C1.
"TN of C1" is all non-C1 instances that are not classified as C1.
"FP of C1" is all non-C1 instances that are classified as C1.
"FN of C1" is all C1 instances that are not classified as C1.
To find these four terms of C2 or C3 you can replace C1 with C2 or C3.
This will enable you to assess specificity, sensitivity etc for each class. Then, you can compute the average for the desirable metric.
https://stackoverflow.com/questions/55635406/how-to-calculate-multiclass-overall-accuracy-sensitivity-and-specificity
If I understand correctly, you have a confusion matrix with two columns (actual positives (AP) and actual negatives(AN)) and three rows (estimated positives (EP), estimated uncertain (EU) and estimated negatives (EN)).
So, in addition to true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN) you also have uncertain positives (i.e., actual positives that were estimated uncertain, UP) and uncertain negatives (i.e., negatives that were estimated uncertain, UN).
If so, you can compute indicators as usual. For instance, recall=TP/AP and precision=TP/EP.
However, you should consider the following.
When there are no uncertain estimation, thus you have a 2x2 confusion matrix, it is TP=AP-FN.
Thus recall = TP/AP = (AP-FN)/AP.
So, you can assume as the definition of recall both TP/AP (the proportion of actual positives that were correctly estimated) or (AP-FN)/AP (the proportion of actual positives that were not incorrectly estimated). Of course, with the 2x2 confusion matrix these definitions are equivalent.
However, when uncertain estimations are allowed, you have TP=AP-(FN+UP), hence AP-FN=TP+UP.
So, while the first definition of recall is still TP/AP, the second is now (AP-FN)/AP = (TP+UP)/AP. Clearly, these possible values for recall are different.
So, which value should you use? Well, it depends on how you want to interpret and use you results. If you are going to treat (e.g., cure) only the estimated positive subjects, then TP/AP is the right value. But is you are going to cure also the uncertain cases, then (TP+UP)/AP is probably more suitable.
In conclusion, when uncertain evaluations are possible, the traditional performance metrics have to be 'reinterpreted' carefully, not just taken as-is.
If not "positive" nor "negative" how do you consider the result "uncertain"? Which is the diagnosis in that case?
- Badges
- Science topic
- Similar topics
- Statistics
- Survey