This is a question that requires a lot of deep knowledge of how ML algorithms work. In deeper nets, it would get even more complicated. There have been some attempts to clarify how each layer performs some tasks. There is a trend called "concept-whitening", which is used to do this. So, if you want to know why, for instance, some particular parts are misclassified while others not, you have two options (1) Interpret things mathematically, which requires the mentioned knowledge and concept-whitening (2) Seek physical explanation. Explain that this part for example has some special features that make it behave this way. You have to then know your data very well.

Trying to interpret machine learning models based on statistical patterns is difficult, and research is still ongoing towards more interpretable and explainable models. Given the knowledge of the input features, a common practice is to use feature selection techniques or regularized models to "weigh" the importance of the features, i.e. how much or not they influence the final decision of the model. For example, a classic decision tree would put the most important features at the top of the tree.

Great ......

You can experiment with the network's behavior upon varying the input sample. Observe upon the model's state when it correctly classifies something, and then do its contrastive analysis with when it classifies incorrectly.

Secondly, the domain of Explainable AI has numerous other methods/techniques which presently is being studied intensely in the ML community. Refer to https://christophm.github.io/interpretable-ml-book/ for one such document.

I think you need a validation dataset. Tc