Hello all. My question is quite similar to one posted by Ahmed Ibrahim in March 2016 ("ROC vs Confusion Matrix for classifier evaluation?"), but I was hoping to ask a couple additional questions about utilizing these tools within ENVI 5.3. I've worked with confusion matrices in IMAGINE, but have never worked with ROC Curves in any program.
Background: I am attempting to quantify that my new classification method is an improvement upon an old one. I used the Decision Tree tool to find pixels of interest on the ground, and I have been trying to use the Confusion Matrix and the ROC Curves tools in ENVI to tell me if the new classifier truly is working better.
Confusion Matrix Tool issues: Initially, I received the error message: "Problem: Input file must be of type: ENVI Classification." I changed the file type of my Decision Tree output in the ENVI header to ENVI Classification and re-ran the Confusion Matrix tool with the "Using Ground Truth Image" option, but I'm not receiving results I would expect -- for example, I receive 100% for the unclassified class and 0% for my pixels of interest. I'm wondering where I went wrong -- can I not use the Decision Tree results and compare them to an ROI? Is there something else I've skipped along the way?
ROC Curves Tool issues: Since I was getting stuck with Confusion Matrices, I decided to try ROC Curves. As I've never worked with this tool before, I'm unsure of how to choose ROC Curve parameters (like the min/max threshold range and what should be chosen for points per ROC Curve). I believe, though, that my min/max should be chosen based on the min/max values in my classified image (in my case, 0 and 255, respectively).
I've looked through all of the ENVI help docs and wasn't able to find something that helped me move forward. I'm hoping that someone might recognize the issues I've run into and can provide some assistance. Or, if anyone can recommend another tool that will help quantify my results, I’d greatly appreciate it. Thank you very much!
Attachments show results from Confusion Matrices and ROC Curves.
Thank you very much, Arjun! I will check the link and try It out.
In case your problem is binary classification, you can go as mentioned for R, this is much easier. You can use the package mentioned by Arjun R. In addition, you can use the Caret package, which is is a mother package for using many other machile learning tools. Thus for the Caret package, you will need do define the method parameter in the train function while using caret to point out for the library you want to use ( tree, gradient boosting, random forest or adaboost trees). In addition, in the Train function in the caret package there is a paramter called metric, if you place "ROC" in the metric parameter, it will automatically report the sensitivity, specificity and AUC. Note that the AUC calculated is named in the output ROC, yet it is the AUC. Regarding the confusion matrix you still can calculate for the test data by calculating the prediction errors for each class over a sample of the test data. But the sensitivity, specificity, and AUC are the most important to report over the training/validation data.
The following post might be useful to your case:
http://blog.revolutionanalytics.com/2016/05/using-caret-to-compare-models.html
I don't know about the tool you have mentioned. But I have work experience in Matlab. Its a better tool for ROC and Confusion matrices analysis. Surely ROC and Confusion matrix give better result and comparison of new and existing methods of classification.
Thank you very much, Saleh! I will definitely look into the Caret package, and will start with the link you shared to learn a bit more.
Thank you very much, R! I have a tiny bit of experience with Matlab, so I’ll try utilizing that instead. I appreciate your suggestion.
- Badges
- Science topic
- Similar topics
- Mathematics
- Algebra
- Matrix