Xiao Feng and Monica Papes - any ideas?

I outline how to do this under Topic 4: Balancing model complexity and transferability on page 41 in this guide: Technical Report A training manual for habitat suitability and connectivity m...

Hope that helps!

Hi Paul Glover-Kapfer , thanks for the reply. I'm assuming the method you outline and the calculations in your Excel spreadsheet mirror the verbal explanation Warren and Seifert (2011) provide for calculating AICc and BIC.

I am hoping that someone has tweaked the code for ENMeval (or similar package) to automate these calculations across a number of Maxent models (i.e., varying the regularization multiplier). However, you do bring up a helpful point that I could write some R code to calculate AICc independent of raster data.

I stumbled onto some older posts in the Maxent google group that provide R code to calculate Warren and Seifert's implementation of AICc (link below). At first glance, I believe this code could be tweaked to fit my needs, similar to your manual calculations.

https://groups.google.com/forum/#!topic/maxent/EsXZKlpvdTI

Hi, Andrew. Have you considered to rasterize your reaches using every in-channel attribute and running the optimization package as usual?

Hi J. Andres Olivos , thank you for the response.

What I worry about with the approach you mentioned is altering the grain size of modeling environment (i.e., now multiple raster pixels may represent one stream segment).

One way your suggestion could work is if there was a way to neatly coerce my tabular samples-with-data format into a rasterstack object. Then I could run the optimization packages and presumably extract the model predictions and link them back to my stream segments in GIS. In theory this approach should work, but I do not know the coding tricks that would be needed on both ends to make the data usable.

Alternatively, since posting the question and the initial response by Paul Glover-Kapfer, I've been working on R code that is essentially a simplified version of ENMeval for non-raster datasets. If anyone is interested in this, or has been working on something similar, please let me know.

This is coming a bit late, but I think the reason most implementations use rasters when calculating AICc for presence-only models is because you need the predicted values for the entire background extent. If you sampled this extent completely (not randomly), then you could feasibly use the predicted values from a table instead of the rasters themselves. I'm working on a new version of ENMeval and am thinking of adding this option, with some caveats.

Thanks for the reply Jamie M Kass . I was able to calculate AICc per Warren and Seifert (2011) as you said, from a table of values that included the entire study extent as "background". In stream-segment applications such as my own, I believe the norm is to include the entire system as background (perhaps unless you are modeling on a national or continental scale and computational power is a limiting factor). I've talked with the lead author of the ENMeval paper a bit on this subject, and myself and several coauthors have a paper (currently in review) with the R code we used to address spatial sampling bias and model complexity (along with Discussion that non-raster tools are needed for these applications). I am encouraged to hear about the work you are doing. I hope that you will consider non-raster modeling environments and continue to work to add flexibility and/or alternative options to the raster dependencies in the existing workflows. Thank you!

This is also a late reply, but I'll post it here for posterity. In Wright et al (2015) they cite Dan Warren in a personal communication who said if you are using non-random background sites then the AICc calculation should be performed across them (vs. the entire background--meaning all cells). When I read this it occurred to me that it is actually really strange to calculate AICc across all cells on a raster--after all the model itself is not presented with data on all cells (unless there are

Adam B Smith , thank you for the reply. I agree with your assessment that AICc should be calculated across the study area extent, including all of the cells (or in my case, stream segments) within it, but excluding cells situated outside of study area boundaries.

Adam B Smith has a good point -- AIC is traditionally calculated on data used for model training, and using predictions of data points that were not used for training to calculate AIC is a bit strange. I could not easily find a good argument for this in the original Warren & Seifert paper (2011). The original quote from Wright et al. (2015):

"The choice to normalize over the entire geographic extent in the original method was made to ensure that the full climate space available to each species was sampled, something that might not occur if background data were drawn randomly. However, in our case, calculating the likelihood over the data used to train the model is appropriate because our background data are not drawn randomly and our taxa are well sampled in the study region (D. Warren, pers. comm.)."

I guess this makes sense in order to avoid uncertainty, but it is still a strange to include non-training data into the calculation of AIC. Background samples should be good samples, in that they are representative of the environmental variability (see Guevara et al. (2018)). If you are using a bad sample (where the background environmental density curves change drastically with each random draw), then your model results are not consistently representative of the study extent. Thus, if you have a good sample, using just the background data points used for model training to calculate AIC should be relatively stable with regard to variability even if they were drawn randomly.

Hi all,

Andrew T. Taylor, it looks like you already solved your problem, but here my answer just in case you or other researchers want to explore this option.

I agree that it is weird to calculate AICc values from an entire raster layer if the N of the background used is different than the total number of pixels in the layer. When this happens the total sum of raw values in the raster layer can be a lot larger than 1, but the sum of raw values in the background will always be 1. I don't know how much this can change the final AICc value of a model and how this can affect the comparison of multiple models, but it may be worth exploring it.

Here is an R function to calculate AICc from numeric results and not rasters. I use it when I work in SWD format in maxent. I hope it helps. The arguments are as follows:

occurrences = all occurrences used to create the model (columns: longitude, latitude)

prediction = all points used as background (columns: longitude, latitude, raw prediction)

n_parameters = number of non-zero parameters in the lambdas file that maxent produces.

A requirement is that all points in occurrences are included in the background. They are identified by searching for the exact match of longitude and latitude.

aicc

Thanks Marlon E. Cobos , that looks very useful!

Your comment also reminded me to post this recent article on the thread, wherein we tried to tackle issues of spatial sampling bias and model complexity (similar to ENMeval functions) using non-raster data with NHD stream segments. Portions might be useful to others. My hope is that someone will eventually work non-raster formats into a holistic R package with some of these standard functions.

Article Spatial sampling bias and model complexity in stream‐based s...

Jen Tinsman - thanks for adding another voice to the need for non-raster functionality. I noticed an R package recently that does the Maxent MESS analysis with SWD functionality (I believe dismo and others were raster-dependent). https://rdrr.io/cran/modEvA/man/MESS.html

Hopefully more flexible packages arise!

Jen Tinsman - Confirmed: ENMeval v2.0 will have SWD functionality and some new features for plotting MESS maps of occurrence partitions. It will also calculate AICc without rasters as an option.

NICE! Thank you for incorporating this Jamie M Kass .

That will be huge for our modeling efforts in segment-based riverscapes.

UPDATE: This has been implemented in ENMeval 2.0.x, now on CRAN. Please see the paper for detailed description of all the new functionality: https://doi.org/10.1111/2041-210X.13628