I've never done this myself but I've collected quite the amount of resources on spatial analysis with GIS and R in the past year...
Maybe is not useful in your case, but at least I can point you in the right direction: for what I know you need to use adehabitatHR (https://cran.r-project.org/web/packages/adehabitatHR/vignettes/adehabitatHR.pdf) and then use a UDOI ( Utilization Distribution Overlap Index ) or a percentage of the overlapping area.
Reference for the UDOI: https://gis.stackexchange.com/questions/44389/how-to-calculate-utilization-distribution-overlap-index-udoi-in-arcgis-desktop
Reference for the percentage: https://conservationecology.wordpress.com/2013/03/03/calculating-percentual-home-range-overlaps-in-r/
Finally, in this paper (page 23 of the link below) the author compare two model (2D and 3D) and the formula for the last model is the same of the UDOI used in adehabitatHR, so I think it can be useful.
Sorry for reporting only others examples and models, and to not answer with a direct, concise and practical explanation! If someone can provide some new inspirational way to analyse the overlap, I'm interested in it too!