Have a look at http://www.riks.nl/mck/, the map comparison kit might not be open source but is free to download once you have registered on the website

Thank you Riyad. I had seen some references to the map comparison kit (MCK) software but somehow hadn't managed to find that web-site. It looks like the MCK only runs on Windows. I generally work on Ubuntu and Mac so that's a pretty big draw back for me. I'm about two thirds of the way through building a QGis plugin that will do what I want. I'll post back here when it's done in case anyone is interested.

I'll also be taking a look at the references to fuzzy comparison methods that I found on the MCK page. Perhaps I will add some of those comparison methods to my plugin.

Thanks again,

Jared

Just in case anyone is reading this...

I've put up an early version of my QGIS Accuracy Assessment plugin. The easiest way to get it is through the Plugins / Fetch Python Plugins menu item in QGIS. It's called 'Accuracy Assessment' and will probably show up near the top of the list. More information can be found here: http://plugins.qgis.org/plugins/accassess/ and here: https://github.com/jkibele/acc-assess.

I plan to make some improvements at some point but the basics are there now.

GRASS GIS and the r.kappa command works fine

I had not seen r.kappa before. It sounds like it does 90% of what I want. However, now that I've started with the QGIS plugin development, I'll probably still continue on with it and add a few more features. I tend to like working with the QGIS interface more than I like GRASS and hopefully others will find the plugin useful as well.

Thanks for pointing out r.kappa. At the very least, it will give me something to check my results against to make sure my plugin is working correctly.

I love GRASS and QGIS is a very good interface. I am looking forward to use the plugin! However, I see a great potential in R for GIS raster analyses. I use it all the time now, and you can easily combine it with python using ipython!

In case you are still interested in other tools for accuracy assessment, try Orfeo Toolbox and Confusion matrix Computation

Jared, thanks for your work! Is it possible to compare my classified map with shapefile points reference?

Hi Joao,

My plugin only takes raster input at the moment but you can use the gdal_rasterize command to convert a point shapefile into a raster. The documentation for gdal_rasterize is here: http://www.gdal.org/gdal_rasterize.html.

If you don't feel like using the command line, you can use this same function through QGIS. It's accessible in the raster menu as "Rasterize (Vector to Raster)" under the "Conversion" sub-menu.

Just make sure that the extent and resolution of your new raster match that of the raster you're trying to compare it to, and it should work fine.

Great, Jared!

thanks

Hello,

You might be interested by my project https://github.com/scls19fr/pandas_confusion and its Pip package https://pypi.python.org/pypi/pandas_confusion

With this package confusion matrix can be pretty-printed, plot. You can binarize a confusion matrix, get class statistics such as TP, TN, FP, FN, ACC, TPR, FPR, FNR, TNR (SPC), LR+, LR-, DOR, PPV, FDR, FOR, NPV and some overall statistics

Kind regards

That looks great Sebastien. I'm up to my neck in some other work at the moment but I'll have a look at providing a pandas pickle output option for my plugin if I get the chance. Your outputs look a lot nicer than my .csv files.