Hi all:
I met a problem in an RL project. We use an searching method to produce the data and then use deep neural network to train the value function. It is an off-policy method. We have a large data buffer to keep the data, and every time we sample the data and feed them into the neural network(replay). Although the average reward increases, I found the loss function never decrease and even increase!!! Attached is the plot for loss function. Did you meet this problem? How did you solve it? I thought the reason lies in the lack of important samples...
Thank you
Hi Chen
If you want that people help you here, you first need to clearly describe the problem you are working on:
What is your problem
How did you modelize it (I guess you are implementing a DQN algorithm):
- state input vector
- output vector (usually you are working with the Q value not V)
- what kind of neural net
- target for policy evaluation (Monte carlo, time difference with n-steps bootstrap or not...)
- E-greedy exploration or not? exploration versus exploitation ration
- replay buffer
- batch size?
- what is your loss function
- and finally your optimizer...
Then, I think that you can find somebody out there who will be able to help you, otherwise it is impossible to answer your question.
Regards
I have encountered before that the weight coefficient of L2 regularization is too large and the loss function goes up
Yes, it combines the DQN and PG. The training data is updated by the policy and we sample them in the data buffer to update the policy. Thank you
- Badges
- Science topic
- Similar topics
- Cognitive Science
- Thinking