I'm working on a project that involves the classification of satellite imagery into benthic habitat maps. I will need to produce accuracy assessments of my maps relative to field collected reference data and I will need to compare the accuracy of habitat maps derived from different classification procedures. I like to work with open source software whenever possible and I do most of my work with QGis, Python, and various Python libraries.
I have started writing my own Python scripts for creating confusion matrices (or error matrices or whatever you'd like to to call them) with user's, producer's, and overall accuracy. I will also add in some sort of kappa value or, more likely, something like overall quantity and allocation disagreement measures as suggested by Pontius & Millones, 2011. I also hope to package up my script and make it available as a QGis plugin when it's done.
Now that I'm about half way through this process, it's occurred to me that I ought to check and make sure I'm not reinventing the wheel. I've dug around on the internet a bit but I haven't been able to find any open source software for accuracy assessment. I've seen some references to Erdas Imagine having this capability but, aside from that, I haven't seen much about propriatary software doing this stuff either. Can anybody out there tell me if I've missed something?
Thanks,
Jared Kibele
Pontius, R. G., & Millones, M. (2011). Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. International Journal of Remote Sensing, 32(15), 4407–4429. doi:10.1080/01431161.2011.552923
Have a look at http://www.riks.nl/mck/, the map comparison kit might not be open source but is free to download once you have registered on the website
Thank you Riyad. I had seen some references to the map comparison kit (MCK) software but somehow hadn't managed to find that web-site. It looks like the MCK only runs on Windows. I generally work on Ubuntu and Mac so that's a pretty big draw back for me. I'm about two thirds of the way through building a QGis plugin that will do what I want. I'll post back here when it's done in case anyone is interested.
I'll also be taking a look at the references to fuzzy comparison methods that I found on the MCK page. Perhaps I will add some of those comparison methods to my plugin.
Thanks again,
Jared
Just in case anyone is reading this...
I've put up an early version of my QGIS Accuracy Assessment plugin. The easiest way to get it is through the Plugins / Fetch Python Plugins menu item in QGIS. It's called 'Accuracy Assessment' and will probably show up near the top of the list. More information can be found here: http://plugins.qgis.org/plugins/accassess/ and here: https://github.com/jkibele/acc-assess.
I plan to make some improvements at some point but the basics are there now.
I had not seen r.kappa before. It sounds like it does 90% of what I want. However, now that I've started with the QGIS plugin development, I'll probably still continue on with it and add a few more features. I tend to like working with the QGIS interface more than I like GRASS and hopefully others will find the plugin useful as well.
Thanks for pointing out r.kappa. At the very least, it will give me something to check my results against to make sure my plugin is working correctly.
I love GRASS and QGIS is a very good interface. I am looking forward to use the plugin! However, I see a great potential in R for GIS raster analyses. I use it all the time now, and you can easily combine it with python using ipython!
In case you are still interested in other tools for accuracy assessment, try Orfeo Toolbox and Confusion matrix Computation
Jared, thanks for your work! Is it possible to compare my classified map with shapefile points reference?
Hi Joao,
My plugin only takes raster input at the moment but you can use the gdal_rasterize command to convert a point shapefile into a raster. The documentation for gdal_rasterize is here: http://www.gdal.org/gdal_rasterize.html.
If you don't feel like using the command line, you can use this same function through QGIS. It's accessible in the raster menu as "Rasterize (Vector to Raster)" under the "Conversion" sub-menu.
Just make sure that the extent and resolution of your new raster match that of the raster you're trying to compare it to, and it should work fine.
Hello,
You might be interested by my project https://github.com/scls19fr/pandas_confusion and its Pip package https://pypi.python.org/pypi/pandas_confusion
With this package confusion matrix can be pretty-printed, plot. You can binarize a confusion matrix, get class statistics such as TP, TN, FP, FN, ACC, TPR, FPR, FNR, TNR (SPC), LR+, LR-, DOR, PPV, FDR, FOR, NPV and some overall statistics
Kind regards
That looks great Sebastien. I'm up to my neck in some other work at the moment but I'll have a look at providing a pandas pickle output option for my plugin if I get the chance. Your outputs look a lot nicer than my .csv files.
- Badges
- Science topic
- Similar topics
- Geoscience
- Remote Sensing