A lot of papers are published recently on explainable artificial intelligence (XAI). However, there seems no clear meeting point on what researchers define as XAI. Can we have an engaging talk around this issue?
As I understand it in a simple way, the traditional AI model has limitations, and sometimes its outcome is not reliable and constant. The same AI model gives different performances depending on which machine it's working on or other criteria. Explanable AI, or XAI, tries to explain the internal function. For example, you get better results using an optimizer. So what's the specific work of that optimizer on that system? Why does it work better than others? You have to understand and explain. So the deep learning model will be transparent rather than a black box.
Too many terms seem to be used these days on RG that are not fully defined or understood so that discussion produces nothing useful. Or is that just my degree in mathematics speaking? James
The idea is that XAI models do not only produce predictions, but also justification of how/why those predictions were produced. This is important because understanding how the model arrived at its predictions can increase trust in the model, or may provide clues on how to improve it. Some models are intrinsically explainable, e.g in decision trees there are rules to take one branch rather than another, and those rules can play the role of explanation. But many models such as deep neural networks are too complicated for that, so they need some additional mechanism for explanation. In an example, a classifier was doing a good job by identifying when an image contained a horse in the development environment, but failed to do so with new images never seen before. What was going on in here? The mystery was solved when an explanation method revealed that the model was making its predictions based on the presence or absence of a small watermark that appeared in the training images of horses. This not only revealed how the model was making the right predictions, but also what caused it to make wrong predictions.
Miguel Lerma and Esmot Ara Tuli, I appreciate your input to the discussion.
I love the fact that we are setting the tone of the discussion. Quite insightful indeed.
- Badges
- Science topic
- Similar topics
- Phytochemistry
- Extracts