https://arxiv.org/pdf/1701.08718.pdf

https://arxiv.org/pdf/1605.06065.pdf

@Himadri Nath Saha

I am aware of Neural Turing Machines, their newer version, the Differentiable Neural Computers and the TARDIS model. But as far as I undrestand, they all employ external memories in array or matrix formats. While their controller modules are NNs optimized for tasks, their external memory modules themselves are not. Am I getting something wrong? What I am looking for is setups where the external memory module is a distinct NN optimized for its own particular task, i.e. memory. 

Check out some older work by Robert French 

Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference

http://leadserv.u-bourgogne.fr/rfrench/french/catfor94.pdf

AND

Interactive tandem networks and the sequential learning problem

https://www.researchgate.net/profile/Robert_French/publication/2423690_Interactive_Tandem_Networks_and_the_Sequential_Learning_Problem/links/02e7e519132cf3f03d000000.pdf 

Article Interactive Tandem Networks and the Sequential Learning Problem

Yup, that's it! ITNs do have a multi-NN architecture of the kind I was looking for. Thank you Daniel L. Silver. Now I'm curious whether Interactive Tandem Differentiable Neural Computers could be a thing...

I suspect the best way to look at this is to consider the memory network of this model to be a more complex hidden recurrent layer that influences the primary network in in such a way as to provide helpful inductive bias. 

The boundary between a complex layer and a network module in an intricate architecture eventually gets fuzzy. I think that a layer approach to memory would get us closer to Long Short-Term Memory Networks (LSTMs), wouldn't it? I like the network module approach in the sense that the module can have input, hidden and output layers itself that can be trained and evaluated somewhat autonomously. A network module could also potentially be interchangable between master networks. Also the GAN insight of many NNs effectively training each other and the benefits this layout affords can't actually be managed in the layer level, can they?

Yes, this sounds generally correct.  Currently, the focus of the deep recurrent networks is on learning predictive models from temporal sequences of input signals.  They continue to primarily do input-output mapping in the context of temporal aspects of the input signal. The move to Neural Turing Machines is a step in the right direction for learning arbitrary "algorithms" that consider complex internal and external context as well as temporal aspects.

@Joachim Pimiskern

My reasearch goal is not to simulate but to emulate human-type memory (be it long-term, short-term, working memory, muscle memory or whatnot) for computational applications. As it may be starting to get clear, I like the network-of-networks approach. Why do you think Hopfield Networks are a good choice for a long-term memory component?

I  think how memory is structured does play a role, even for humans. I' am all for the form-follows-function design adage, but when you are just beginning to explore the probability space of functionality, reverse-engineering from form seems a viable strategy.  A completely modularized long-term memory as passive storage with a working memory buffering content to perform operations is exactly what gets us the von Neumann bottleneck. Do you see a way around it retaining this layout?

I haven't actually contemplated on multiple working memories before. It's an intriguing concept. I can see why multiple personalities may not exactly be a desirable trait for a human. Is it necessarily an unwanted feature for an intelligent machine, though? 

David Andre and Astro Teller both did work with memory and mental models, as well as reinforcement learning, incorporated with genetic programming.   You can find the work from name and keywords.

@Joachim Pimiskern

Oh, Miller's Law! I feel that quantization protocols in this line of research vary so widely it almost obliterates any numerical sense for practical applications. Is there a physical limit to human parallel processing? There might as well be. Assuming there is, does it have to match the physical limit to all parallel processing? 

I've come across the Memristor concept. From what I gather, it's still theoretical and not successfully implemented as hardware. I like the 'memory-driven computing' spin on it. It's a more down-to-earth objective, repurposing already available tools instead of struggling to invent new ones. Still on the Processing/Memory dichotomy, though. Simply shifting focus. No NNs, either.  I like to think there's something in NNs that can get us past the dichotomy altogether. I can't quite put my finger on it yet. 

@ Ahmed Ibrahim

PSO sounds interesting. I'm not sure how it helps with Memory, though.