Deep learning is often called as an block box model researches in medical imaging usually use XAI to explain their results .There are few techniques like , Gradient based or even textual based but significantly on the confidence of an algorithm such as pre-modal explanation that focuses on the dataset particularly. Looking forward to learn more and would be really grateful for resources that could help me out .

Thank you and Have a great day )