Imaging an assessment scale with multiple cut-off scores, mild, moderate, severe symptomology, and it is used as the gold standard versus a newly developed scale. How do you run the ROC/AUC analysis ? considering also that since this is the first study with the new scale, you may be limited in sample size.
Amir Sepehry , how small is the sample size? How many observations do you have? How balanced are the categories of your (binary?) dependent variable? (e.g. 50% of zeros and 50% of 1's?)
Early 30s for sample size. and if you are referring to prevalence, within the sample, its 30%. 1 time only observation.
Hello Amir,
Assuming that the new measure yields continuous scores, you'll need to:
1. Determine whether misclassifications are equally costly across all three levels (and, if not, which is more costly and by how much). For example, is under-rating a case which should truly be "severe" as "moderate" more costly than under-rating a case which should truly be "moderate" as "mild"? Or, under-rating a case that truly should be "severe" as "mild"?
2. You could use a combination of classification/regression tree method with utility weighting (based on your response to #1, above).
C/R trees are very opportunistic, however, even with "internal" validation (e.g., k-fold method), so trying your chosen thresholds for the new measure with new/additional data would be highly recommended.
Good luck with your work.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size