This issue has been explored in a NIPS paper by David Pfau and Oriole Vinayls (researchers at Google).

[1610.01945] Connecting Generative Adversarial Networks and Actor-Critic Methods

Intuitively, what the paper shows is that GANs are closely related to a model of reinforcement learning proposed by my long-term UMass colleague Andrew Barto called an “actor critic” (AC) method. In AC methods, the agent separately keeps track of a policy that guides behavior (usually as a parametric distribution giving the probability of choosing some action a in state s, or p(a | s; \theta) where \theta is a vector of parameters specifying the distribution (e.g., mean, variance etc.). The agent also keeps track of the value function specifying how good the policy of the actor is. However, there is much more to the story than is told in this NIPS paper. There remain deep differences between the two approaches, summarized below.

The critic in AC is like the discriminator in GANs, and the actor in AC methods is like the generator in GANs. In both systems, there is a game being played between the actor (generator) and the critic (discriminator). Each starts out not knowing very much. The actor begins sort of bumbling around the state space, and the critic has no clue how to evaluate the sort of random behavior of the actor. And yet, out of these hopeless beginnings, order emerges from chaos. Hard to believe, but it can be formally proved that in any Markov decision process, AC methods will eventually converge to the optimal policy. The proof was non-trivial and shown decades after the original AC method was published by Barto, Sutton, and Anderson in 1983 in a classic paper. This original paper can be viewed as the formal beginning of work in computational reinforcement learning. Fast forward 35 years, and Google Deep Mind used the same AC algorithm combined with deep learning (a simple combination of convolutional networks and feedforward networks) to solve the difficult Atari games from raw video input.

The proof that AC methods converge was shown in a brilliant Masters thesis by Konda (Indian Institute of Science), who later came to do his PhD at MIT (under John Tsitsiklis). Konda’s PhD thesis on actor critic methods is a tour de force of mathematics, and is not for the faint hearted, but is a beautiful example of the deep mathematics in reinforcement learning. Konda’s MS thesis appeared as a journal paper in the SIAM Journal of Control and Optimization (along with co-author Vivek Borkar, who pioneered the two-time scale stochastic approximation theory that underlies the actor critic algorithm).

Actor-Critic--Type Learning Algorithms for Markov Decision Processes

Getting back to the GAN side of the story, sadly there is no proof of convergence of the GAN architecture. No one has yet shown that GANs, as originally formulated, will converge to an equilibrium (and practical experience bears this out, as numerous people who implemented GANs have discovered, they are notoriously fickle).

One ray of hope is provided in the idea of using multiple generators and discriminators, unlike the original GAN, and in modifying the original distance metric to the so called Earth Mover’s metric (or Wasserstein metric). Arora et al. have shown that a so-called MIX-GAN architecture with multiple generators and discriminators can reach an approximate equilibrium under fairly tight conditions. In practice these conditions are not usually adhered to.

[1703.00573] Generalization and Equilibrium in Generative Adversarial Nets (GANs)

Two PhD students of mine showed in an earlier paper that keeping around multiple discriminators does indeed improve empirical performance (they called this architecture a “GMAN”, for generative multi-adversarial network). This work was published this year in ICLR 2017.

[1611.01673] Generative Multi-Adversarial Networks

Geting back to the connection between GANs and RL, it would be interesting to explore whether there are analogous “multi-generator” and “multi-discriminator” type AC algorithms.

Finally, intuitively it is quite amazing that AC methods (and empirically GANs) converge at all. Think about it for a minute. At the beginning, the actor knows nothing, and bumbles around. The critic is equally clueless. It’s like the “blind leading the blind”. Both struggle to learn and teach each other, but amazingly enough, the combination eventually does learn to behave optimally in complex tasks (as Google Deep Mind has shown).

Do we have actor critic architectures inside our heads? Is there a part of the brain that corresponds to an “actor” module, and a separate part of the brain that acts like the “critic”? There are some reasons to think this architecture is biologically plausible. It is an extremely clever idea, and is an example of the highly innovative thinking of the early pioneers in reinforcement learning, who were exploring ideas decades before they would see practical applications come to fruition.

This answer is available here : https://www.quora.com/What-is-the-relationship-between-reinforcement-learning-and-adversarial-learning-e-g-GAN

Great, thanks a lot for your detailed answer.

Dear Abhishek D. Patange ,

back of my information: it could be considered as two agents that each of them follow RL method in a competition with each other,

Yes I was thinking the same. Thanks for clarifying dear Amin :)