I am having problem understanding how weka calculates ROC curves. As I am generating machine learning models by 4 methods-j48, random forest, naiive bayes and when i save the data for roc by each of them number of instances vary. for some there are only three instances or some like hundred. Should I not be getting almost same number of instances in all the cases irrespective of the model. I understand that ROC is plot of True positive rate vs. False positive rate but how does number of instances come into picture in weka.
It is possible to get multiple ROC curves on a single plot by using
the KnowledgeFlow. Set up a flow as follows:
ArffLoader ---dataSet---> ClassAssigner ---dataSet--->
ClassValuePicker ---dataSet---> CrossValidationFoldMaker ---
trainingSet/testSet (i.e. BOTH connections) ---> Classifier of your
choice --- batchClassifier ---> ClassifierPerformanceEvaluator --->
ModelPerformanceChart
Next configure the ArffLoader with a data set, and then configure the
ClassValuePicker by selecting which class value you want to treat as
the positive class. Then you can start the flow running by selecting
the "start loading" action from the ArffLoader. The model performance
chart will show the ROC curve.
Subsequent ROC curves for different algorithms can be displayed on the
same plot by either deleting your first classifier from the existing
flow and inserting a new one and then running the flow again, or,
setting up a new flow identical to the first one (but with a different
classifier) and connecting this one's ClassifierPerformanceEvaluator
to the first flow's ModelPerformanceChart.
For subsequent curves to appear on the same plot the dataset, class
attribute, positive class value and evaluation method (eg 10 fold
Xval) must be the same as for the first curve.
I have tried this by installing a plugin 'JFreeChart Renderer'. Multiple ROC curves of different algorithms can be plotted in a graph. It is easy to compare. However, I could not find the configuration to choose or set the different colors for different algorithms, thus it would be able to compare algorithms along with different datasets. Problem here is multiple graphs of different algorithms in different datasets are different.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Bioinformatics
- Bioinformatic Tools