Hello,
For most learning algorithms, we make I.I.D. (Independent Identical Distribution) assumption for the dataset. This assumption is both reasonable and useful (https://ai.stackexchange.com/questions/10839/why-exactly-do-neural-networks-require-i-i-d-data). In deep RL, we learn from the experience tuples of (s_t, a_r, r, s_(t+1)). During training, these tuples are sampled in batches. Are we making I.I.D. assumption at this step? If yes, how do we defend it because clearly, there is a system dynamics that control transitions? Any discussion/pointer on the topic is much appreciated.
Thanks!
Two simple approaches exist that might form the basis some revised methodolgy.
(i) Weighting. Suppose that, out of all you input points, there is one pair of points that are extremely highly dependent. Then it would be reasonable to assign each point a half-weight in any overall objective function. Similarly, if there are many pairs or groups you could devise a sensible weighting scheme.
(ii) Analogy with linear regression. As an extension of (i), you could look at what is done for regression with dependent model errors. There, a sum-of-squared-residuals objective function is replaced by a quadratic form involving the inverse of the covariance matrix of the errors. To implement this you might just guess a covariance matrix to give the right effect in order to judge how important dependence is, or you might be able to justify some particular structure for the covariance matrix, possibly based on data analysis. For example, your notation using t suggests that time is a consideration here ... in that case you could use covariances generated by some time-series model.
You need to be careful in your use of the term "optimal". In the regression case, under weak assumptions, a linear OLS application will converge to "true" value even if some correlation and/or heteroscedasticity is present .. and so might be said to estimate the optimal answer. But a GLS application will give estimates converging to the same value but with smaller variance ... and so might be said to converge to the optimal answer, optimally quickly.
Hi Namrata Gupta
!When optimizing neural network, the training data should be independent and identically distributed. This is done to prevent action values from oscillating or diverging catastrophically. However, sequence of experience tuples (e.g. (s, a, r, s') and (s', a', r', s'')) can be highly correlated. The learning algorithm that learns from each of these experience tuples in sequential order runs the risk of getting swayed by the effects of this correlation. For this reason, mechasisms such as experience replay, in which experience tuples are stored and randomly extracted for the optimization, are tipically used.
Hope I've been helpful
- Badges
- Science topic