How can I judge which image segmentation method performed well using ROC or perfcurve in MATLAB?
Any method of analysis (classification say) is a tool. It should be treated the same way as any hardware tool. Get results from part of a data set (define parameters), and predict segmentation in the rest of the data. Then in the validation data set, examine the error statistics from say a distance square criteria, compared to the true segmentation (gold standard). In other words, any methoid should first be applied to an artificial data set with and without noise, using characteristics of experimental data. It should perform well on this data set, before you try on expt data. In this validation of method, the errors in segmentation, border curves, should have zero mean at the very least, and gaussian statistics if possible to allow use of confidence intervals. Otherwise, confidence in border locations would require using bootstrap or Monte Carlo methods.
Another separate step in validation of an algorithm is the 'model' complexity applied for the definition of segmentation shapes. The previous method validation needs to be repeated for each possible model complexity...
Using ROC, the 'optimum' parameter is that that produces the point on the ROC closer to the top left corner. That is the optimal combination of sensitivity and specificity.
Because you explicitly asked for perfurve/ROC I dare to give a small example:
label = [0 0 0 1 1 1 1]; % your ground truth data
scores = [.1 .2 .7 .9 .4 .3 .8]; % your estimated scores (from your algorithm)
[tpr, fpr, thresh] = roc(label, scores); % The ROC function
[fpr2, tpr2] = perfcurve(label,scores,1); % The perfcurve funtion
plotroc(label,scores); % calculating and plotting altogether
figure(); plot(fpr, tpr); title('ROC'); xlabel('False Positive Rate'); ylabel('True Positive Rate'); % Plotting the ROC results
figure(); plot(fpr2, tpr2); title('perfcurve'); xlabel('False Positive Rate'); ylabel('True Positive Rate'); % Plotting the perfcurve results
Here I assume that you provide a score for each pixel which you can compare to your ground truth labels. perfurce and roc can help you with that.
The thresh parameter at the roc function provides you the thresholds for each ROC point. Why? Because nobody but you can tell at which point you want to operate. The more false positives you accept, the more true positives you will gain. But this tradeoff is up to you and your specific application.
Thank you, how about computing ROC for binary classification using svm?
If you have a binary classification, you can plot a single dot in the ROC figure only.
To generate a ROC you need a technique that classifies events/measurements as positive or negative and the technique depends on the value of a parameter. You also need a gold standard, i.e. situations for which you can apply the technique, but you also know if they are positive or negative. To generate one point on the ROC you choose a value for the parameter and then you classify the universe of the situations you have (and for which you know the true classification). You can count correct decisions as well as false positives and false negatives, therefore you can derive sensitivity and specificity. Do that for a wide range of the parameter and you will be able to plot the ROC curve. You can then choose a point of operation based on the ROC curve. Normally one would like to maximise both sensitivity and specificity, as I indicated above, but you might want to have more sensitivity sacrificing a little specificity or vice-versa, depending on the problem.
The confusion matrix parameters are true positive, true nagative, false positive and false nagative. we can easily take the ROC curve by the use of cross entropy method by training artificial neural network simulation.
- ROC Curve
- Originally used for distinguishing 'Noise' from 'Not noise'.
- ROC curve is an evaluation metric for binary classification.
- Gives Trade off between True Positive and False positive.
- In ROC curve more area under the curve better is the Model. larger the dataset more the area under the curve.
ROC would assist the us in choosing his/her operating point in terms of false positive and detection rate.
test_labels=double(nominal(imdsTest.Labels));
% ROC Curve - Our target class is the first class in this scenario.
[fp_rate,tp_rate,T,AUC]=perfcurve(test_labels,posterior(:,1),1);
figure;
plot(fp_rate,tp_rate,'b-');hold on;
grid on;
xlabel('False Positive Rate');
%ylabel('Detection Rate');
ylabel('True Positive Rate');
- Badges
- Science topic