As such, there is no explainable AI(XAI) tool or package is readily available mainly because of the following reasons,
It is mostly application-specific and secondly, this domain has not explore much due to the fact that we are more focused on what results we are achieving through CNN or another Deep Neural model rather than XAI.
But, still, there are many research journals available in this regard.
I have used the Captom.ai library, designed for Model interpretability for PyTorch. It has many exciting features that you might find helpful as an explanation.