I am trying to evaluate a test against the gold standard.
The gold standard test has 5 categories (no effect, small effect, medium effect, very large effect, extreme effect). I am trying to determine the best cut off value for the new test that correctly classify subjects to each of the 5 categories of the gold standard.
I created five different variables,
1- no effect goldstandard
2. small effect goldstandard
3. medium effect goldstandard
4. very large effect goldstandard
5. Extreme effect goldstandard
I coded each variable of these five as 0 "not in this category" 1 " in this category"
I ran the code to generate the ROC , graph and details for each of these categories against the continuous /ordinal observations of the new test that range from (0 to 15).
When I run the ROC command I get cut-off scores in terms of = or > 0 , = or >1,..etc till..=or > 15
I would like to generate the ROC for cut-off scores in terms of = or less than the cut off score.
Can I do this?
If I can't, can I just generate an inverse ROC?
If sensitivity for a cut-off point= or >4 =90%, can I conclude that sensitivity for a cut off
Although there is an ambiguity in what codes you are trying to perform on which software, I will try to give an answer supposing that your aim is to find the optimal cut-off for a 5-category response.
When you have more than 2 categories for a gold standard, you could use the ROC Surface to find the optimal cutoff value.
Could you explain more what you are doing?
Mehmet, thanks for adding an answer.
I am using SAS.
I am trying to find out the cut-offs on a continuous score that closely relates to each cut-off on a gold standard.
The gold standard has 5 categories.
There is yet a method to be found to fit gold standards with more than one cut-offs to categorize "disease stages", therefore I am trying to deal with each of these 5 categories at each of the 5 cut-off scores as a separate "gold standard" and match each to the "test".
I am trying to find best cut-off scores on the test to match the cut-off scores for the gold standard.
You can find the best cut-offs by considering the ROC surface. You need to consider true classification rates for 5 categories and a 5 dimensional surface would be constructed. In R you can find optimal cut-offs by this way.
I am not aware of another method and do not think that there is one as simple as you are trying to find. I am curious about your analysis plan and willing to help you more.
Thank you for your answer.
I only have one problem. For the ROC, as you can see in the attached picture, it models a cut-off score with (more than), I need to reverse it to model the cut-off score with (less than).
This is because the scores range from 1-15 for the entire sample, I have 5 categories (3 or or = I can't do this, or this is what I believe for now!
So, do you know of a way to make the ROC model the < or = not the > or = probability?
And also hence, this question:
f sensitivity for a cut-off point= or >4 =90%, can I conclude that sensitivity for a cut off
OK I understand now. This is a coding problem. Namely, If you want to model >= then you call those values >= some cutoff as 1's (namely diseased or correct classification). If you model < then you call those values < some cutoff as diseased. This does not make an essential change except your new ROC curve will be symmetric to the old one with respect to main diagonal and your previous sensitivity values will become a specificity values for the new one.
If you believe that you need to call those patients diseased where your marker is less than some cut-off then you are allowed to do this. Actually this is the very nature of ROC curves and depends on the marker.
Are you saying that I need to reverse my coding for the cut-offs?
so category 1 (=2 =1 and all other values =0?
Do you have an example code that works for any program R, STATA,or SAS?
Actually your situation seems not to be handled with standart ROC curve. You should use a ROC surface because you have a multicategory output.
What I meant is this, suppose you have a score or a biomarker to diagnose a disease. If the score (or biomarker measure) is greater than some cutoff you call patient diseased and if less then healthy. This way you obtain a ROC curve.
If you do the other way around, namely you call him healthy if his score is greater than cutoff and diseased if his score is less than cutoff, then you will obtain another ROC curve.
But these two ROC curves will be symmetric and sensitivities for the former will be specifities for the latter and vice versa. So in fact coding in different manners will not have an effect to ROC curve and sensitivities etc.
I do not have a sample code.
- Badges
- Science topic