I am using reinforcement learning to control my system. The problem that I have is that two control signals are dependent to each other and their value should satisfy a constraint something like this :
u1-u2>b1
u1-u2< b2
How can I apply this constraint on the output of agent in the reinforcement learning?
I am not sure whether I understand your question properly or not. You may want to try implementing your constraints in the reward function. The goal is to avoid unpleasant scenarios in training by having less reward for those cases.
In some cases, constraint actions are modelled by a precondition function. Maybe it help you Ali Molavi
Control constraints are always quite tricky to implement in RL. In this case, you can have a change of variables to make the constraints easier to work with as
u1' = u1+u2
u2' = u1-u2
Such that the constraint read
-inf
While there should have several ways to deal with the constraint, in general, I think the following steps are helpful:
- Whenever the RL agent chooses an action that violates the constraint (or leads some other parameter setting that violates any constraint), the reward function should return a huge negative reward.
- This shall be sufficient to teach the agent that the chosen actions that lead to negative rewards should be avoided.
Thank you all guys. Your proposed methods are very interesting. I am trying to use your ideas to solve the problem.
Hi Ali Molavi, I think there are three methods to solve your question:
- 1. adjust your reward function to penalize constraints violation by giving a huge negative penalty and/or stop the current episode. However, that might provide suboptimal actions if your reward factors are not well-scaled.
- 2. reformulate your action to handle your constraints, as Gerben Beintema answered above, i.e. use u1' & u2' instead of u1 & u2. this method is the best choice, however, it would be insufficient if the main actions u1 & u2 are bounded.
- 3. use constrained enforcement as discussed in https://www.mathworks.com/help/slcontrol/ug/constraintenforcement.html and some examples can be found. In brief, new action u should satisfy Bu
- Badges
- Science topic
- Similar topics
- Automotive Engineering
- Machines