I am working to develop a scoring system for predicting an outcome.
The score contains 4 items.
Each item has a score of 1 or 2
So each patient had a score of 4 to 8
I did ROC curve to calculate sensitivity, specificity and AUC
Do I need further tests to validate this score?
Need? Depends upon what your goal is. If you submit to a low tier journal, you probably have enough. If you want to submit to a higher impact journal, or have people actually use your method (not the same thing), you should probably:
A) Make sure you use different training and test sets. You don't say if you did that. If all you did is train a model, nobody is likely to care. You need to show that it's able to accurately predict outcomes for samples that were not used in training.
B) Test on multiple datasets. Show that your training model isn't totally dependent upon the way one experimental group recorded their observations.
C) Compare to other, similar models. Show that you're providing an improvement of some sort on what other people have already done.
Also: you might consider changing your ranking to '0 or 1' rather than '1 or 2'. That would give a final score between 0 and 4, which is easy to mentally convert to 'how many categories were they positive on'. Similarly, computer science people tend to start at 0, rather than 1, simply because that's what most programming languages expect. It's also the default numbering scheme for 'false' or 'true' in binary applications in e.g. Excel. Etc. etc.
Dear Dr. Ahmed,
to validate this scoring system in addition to Roc curve first ,you need to proof that, is there a association with this system and the gold standard or the clinical status or not? Then the association is +ve or _ve and it is significant or not.The type of the test will depend primarily on the types of variables used in two sides [the predictor and the outcome]. Also, try to validate each group with a specific scoring system and calculating confidence interval CI this more precise than P value.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs