I am relatively new to reinforcement learning and have been experiencing with a reinforcement learning model to make decisions based on human activities (dynamic environment). Appreciate if someone can help me in understanding how best to evaluate a reinforcement learning model for performance before the model goes into production and when it is in production.
You can start preparing test cases. Those that have not been used in training.
What exactly does your RL Agent do? What do state, action and reward look like?
It will depend a lot on your problem and environment. Could you please give some constraints and details of the environment, what data you have from the real environment, alike you have a simulator for the environment, how you trained the agent (given it was not learning in a real environment), and what kind of algorithm you used?
Dear Muhammad Ali,
I suggest you to see links and attached files on topic.
Metrics to Evaluate your Machine Learning Algorithm
https://towardsdatascience.com/metrics-to-evaluate-your-machine-l...
11.3.5 Evaluating Reinforcement Learning ... - Artificial Intelligence
https://artint.info/html/ArtInt_267.html
RL — Reinforcement Learning Algorithms Quick Overview - Medium
https://medium.com/.../rl-reinforcement-learning-algorithms-quick...
Ch 12.1:Model Free Reinforcement learning algorithms (Monte Carlo ...
https://medium.com/...machine-learning.../ch-12-1-model-free-rein...
Best regards
Roman Liessner The data is in the form of images (sensor data) representing the activities in a smart home like turn off the light, turn off the TV .etc.I have not trained the agent yet.What i want to do is to make decisions according to the situation in a dynamic environment.Regarding algorithm,i am planning to use Q-learning model like DQN or DDPG(Deep Deterministic Policy Gradient)
Regret is a very commonly used metric. At each time step you take the difference between the reward of the optimal decision versus the decision your algorithm actually took. You then sum this up for cumulative regret. The minimum regret is 0, with the optimal policy, the smaller the regret the better an algorithm has performed, see slide 15 for more details:
http://www.it.uu.se/research/systems_and_control/education/2017/relearn/lec3.pdf
- Badges
- Science topic