Whenever I have a sequential decision problem — a problem where previous choices influence future choices — and when some parts of the problem are unknown (reward function, transition function between states), I have a reinforcement learning problem.
If all my samples are independent (not sequential) I don't have a reinforcement learning problem.
The agent observes the state s, takes action a, gets reward r and the state is transitioned to s'.
However, let's suppose that s' isn't necessarily the state where the next action takes place.
- Can the agent still learn how to maximize the discounted reward? Or, in other words, is this allowed in reinforcement learning?
Let's suppose I have an agent and two (finite) queues. Each queue can offload data to the other queue, in order to avoid being saturated. The agent, whenever a new data packet arrives to one queue, can decide to offload it to the other queue, or keep it for himself. So the transition from s to s', given action a, is deterministic. However, in the next step, the environment observation s may not be s'.
This is due to the fact that, before the next packets arrives, one or both queues might have processed some data.
My concern is that the agent cannot learn to predict what the next state will actually be. For sure, given state s and action a, the agent can easily predict what s' will be, but at the same time s' is not the "actual" next observed state.
According to my knowledge agent does not need to be able to predict the next state s' but the agent must get feedback from the environment/ system that its previous decision was right or wrong.
In your described environment, suppose in one queue the action taken by the agent was offloading. Immediately after taking this action environment will change and your reward function should be designed to be able to let the agent know that whether the decision was right or wrong. Suppose at state s Queue1 had 5 packets and it reached its limit. The right action on that moment is to send a packet to Queue2. So logically, in next state Queue1 should have 4 packets but as data was processed in the mean time Queue1 was filled up with 5 packets again. If that is a confusing scenario for you, then I suppose RL agent will still learn. But the reward system has to be good enough.
Thanks Syed Tasnimul Islam for your help. Let me quote one of your last sentences: "Logically, in next state Queue1 should have 4 packets but as data was processed in the mean time Queue1 was filled up with 5 packets again."
From your point of view, is this (next) state the one you "feed" the agent (a.k.a. s' in (s, a, r, s') ), or does this state just happens to be the next observed state (a.k.a. s in the next (s, a, r, s'))?
Raoul Raftopoulos pleasure to share my perspective. From my point of view in an iteration, you feed the agent s which results in some changes in the system and you get the next observed state s'. In an ideal condition, this s' is supposed to be fed to the agent as the current state s on the next iteration. But in your case that is not possible if I truly understand because of your observation time window.
Still that won't be a problem for the RL agent to learn.
- Badges
- Science topic
- Similar topics
- Psychology
- Mental Processes
- Learning