In niche models, overfitting has been traditionally assessed by large difference between training and test AUC or other model performance metrics. Are there other methods to assess model overfitting than this? MORE IMPORTANTLY, how do you assess model overfitting if you do not partition data into independent training and test data? Recommendations and their implementation (preferably R scripts, compatible with BIOMOD2) would be greatly appreciated.
To estimate overfitting you need to assess your model on an independent dataset. Or use cross-validation
If you're not splitting your data in an independent training and testing set, I'm assuming you are using internal cross-validation? In that case you can compare the cross-validated AUC (or another cross-validated accuracy metric) with the AUC of a model where you use all data for training and those same data for testing. See this paper of Boria et al. where they do this: Article Spatial filtering to reduce sampling bias can improve the pe...
Cross-validating your ensemble models with spatial blocks (blockCV package for R) also helps to reduce overfitting
You could also visually check your response curves and restrict the types of model features used to fit to the data (i.e., keeping it simple to linear and quadratic functions). You can also use "smoothing parameters" in the model fitting process. These can help ensure a more "biologically realistic" fit of the data so that, in theory, you also avoid overfitting your data. Neither is a straightforward test to "check for overfitting", but both can help avoid it. [Especially if you have a low number of occurrences and can't afford to split them into testing and training sets.]
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training