I'm new in reinforcement learning and I don't know the difference between value iteration and policy iteration methods!
I am also very confused about categories of methods in reinforcement learning. Some studies classified reinforcement learning methods in two groups: model-based and model-free. But, some other studies classified reinforcement learning methods as: value iteration and policy iteration.
I were wondering if anybody help me to know the relation between these classification, as well.
.
this presentation does a nice job at presenting the various options for RL and reminding their relative merits given the applicative context :
http://www2.econ.iastate.edu/tesfatsi/RLUsersGuide.ICAC2005.pdf
.
Actually, I don't know whether the classification of RL approach (i.e., policy iteration & value iteration) is just for model-based approach or is about model-free approach as well?
Dear Fabrice: I've found your link very helpful and I've learned a lot form it. But, I couldn't find anything related to my problem in your file. I were wondering if you can introduce me another suitable resource for this purpose as well.
.
in a nutshell, at the end of learning
- with a model-free approach , the agent knows how to act, but doesn’t explicitly know anything about the environment (think of Q-learning)
- with a model-based approach, the agent has built a correct model of the environment and therefore can simulate it so as to find the right decision (think of E3)
.
.
Thanks dear Fabrice,
Do you know policy iteration & value iteration methods are just related to mode-based approach or are related to model-free approach as well?
I am not sure whether you could understand about my explanation.
Policy iteration requires two step
1. Policy evaluation : you have to try all possible state and action pairs from your transition probability Ps,a,s' based on the predetermined policy "pi"
2. Policy improvement : the it is not converge, you have to run the new policy "pi" for the next iteration.
Then, do it repeatedly until it is converged.
Value iteration can be done via by using the Bellman equation where you can evaluate progressively state-by-state until the solution converges.
I hope it helps.
.
everything goes !
you can have model-free or model-based value iteration algorithms etc ...
.
by the way, the Book is on line
https://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html
(see II.4 and III.9 in particular)
.
and you can even have a mix of the different approaches : see Dyna-Q in the book above
.
To understand this, you need to read about Dynamic Programming and the Hamilton-Jacobi-Bellman Equation !
In my Mater Thesis there is a simple approach from Model-based to Model-free decision-making processes; I think it will help you.
https://www.researchgate.net/publication/281712242_AMT_MASTER_THESIS_presentation (slide 4)
https://www.researchgate.net/publication/281631089_Reinforcement_Learning_on_the_Lego_Mindstorms_NXT_Robot._Analysis_and_Implementation (section 2.1 pages 11-14 )
For further (and formal) information I also recommend the Sutton/Barto book:
https://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html
Data AMT MASTER THESIS presentation
Thesis Reinforcement Learning on the Lego Mindstorms NXT Robot. Ana...
The best way to learn about both methods, similarities and differences, is the book by Russell & Norvig. Some people might differ, but it is considered by majority of AI researchers to be "the" reference book for AI topics. It explains very well (in chapter 17 "Making Complex decisions" I believe), both topics within a few pages. The pdf version is also available online, for a quick glance, but I highly encourage you to find it in your local library etc.
According to my view, policy iteration PI and value iteration VI are two methods that are used to solve the Bellman's equation in on-line fashion.
They have following difference,
1) PI required admissible initial policy whereas VI does not required.
2) PI called full backup solution, it can take significant computations whereas VI partial backup solution and take less computations.
3) VI is a recursive method, it use previous policies to update new policy.
hi negin?
https://faradars.org/courses/fvrml120-reinforcement-learning
Algorithms that purely sample from experience such as Monte Carlo Control, SARSA, Q-learning are "model free" RL algorithms
Policy definition and computations needed are the main difference. Depends on what you are trying to solve. Some good references cited by previous respondents.
In policy iteration algorithms, you start with a random policy, then find the value function of that policy (policy evaluation step), then find a new (improved) policy based on the previous value function, and so on. In this process, each policy is guaranteed to be a strict improvement over the previous one (unless it is already optimal). Given a policy, its value function can be obtained using the Bellman operator.
In value iteration, you start with a random value function and then find a new (improved) value function in an iterative process, until reaching the optimal value function. Notice that you can derive easily the optimal policy from the optimal value function. This process is based on the optimality Bellman operator.
In some sense, both algorithms share the same working principle, and they can be seen as two cases of the generalized policy iteration. However, the optimality Bellman operator contains a maxoperator, which is non linear and, therefore, it has different features. In addition, it's possible to use hybrid methods between pure value iteration and pure policy iteration.
Source: stackoverflow
- Badges
- Science topic