I would like to know if there is a method, or sets of method that can explain why a RL agent has chosen certain decision, after the agent has been trained. I'm not looking for different RL architectures or reformulations of the problem which try to be more transparent and self-explanatory... I know the RL agents try to maximize future rewards, based on their current states. The first and obvious approach that comes to my mind are data-driven test cases with human in the loop... I have also read about secondary and simplest agents evaluating the decisions of its complex counterparts...
May be this content can help you : https://arxiv.org/pdf/1903.04110
Have a look at https://openreview.net/forum?id=S1xitgHtvS
https://www.aaai.org/Papers/FLAIRS/2007/Flairs07-027.pdf
https://ai.googleblog.com/2020/03/massively-scaling-reinforcement.html
I am not sure how to interpret the 'inference' in your question. In RL, the actions, i.e., decisions of the agents are chosen in the training phase. Simply put, an agent learns to take that action that maximizes the long term reward. Therefore, based on the reward function, in each state and time step, an RL agent learns to make optimal decisions. We train such a model over numerous episodes and finally store the trained Q-table. That said, this Q-table indicates the quality of each action-state pairs.
In the testing phase (or performance evaluation phase), in each state (and time steps), we look at the trained Q-table and choose the action that gives the maximum reward. For each test episode - at each state, we simply perform argmax Q(state,:) like operation. So, after an agent is trained, it always chooses the action set based on the reward function we trained it on. In other words, a chosen action can be interpreted as the best shot of the agent that will maximize its long term reward.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence