You need to know what reservoir is doing. It just creates a mapping to a high-dimensional space. It appears that for many problems a randomly generated reservoir (with specific characteristics though) is enough to map the inputs to a high-dimensional space in a way that it is linearly separable.

@Hooman: Is there any theory supporting the working of LSM?

I believe, by specific characterstics, you mean the probability of specific types of connection within the reservoir. How do you determine what combination will work? Is there a rule or we need to resort to hit and trial?

They are some very general rules. Other than that good characteristics are subject of open research AFAIK. Check reservoir recipes for some rules.

I have followed LSM of Maass group from the start around 2003, it is just a random recurrent SNN with some inhibitory and excitatory neurons. No clear idea behind it more than separation and approximation.

I think any other model can do better if there are realistic in it.

Recently has been some new development in training RNNs that might have change the landscape but a while ago RC was preferred for its easy low cost training. To have a better idea of the current state of the RNNs training methods you can check Geoffrey Hintones NN course at coursera and read his latest papers.

.

reading this scholarpedia entry might be useful

http://www.scholarpedia.org/article/Echo_state_network

(echo state networks and liquid state machines are -at least- very very closely related !)

.

The scholarpedia article is useful, however, it does fully answer the reason why these networks work. A similar concept is used in Extreme Learning Machines (ELM) and they have some theoretical support for their learning capacity of the networks. They are not recurrent though.

Thanks for your suggestions.

.

to me, ELM looks like some adaptation of the random projections technique into the feedforward neural networks community, as ESN/LSM look like some adaptation of the random projections technique into the recurrent neural networks community !

pointers about random projections, if you are interested :

https://sites.google.com/site/rpforml/

Bob Durrant papers : http://www.cs.bham.ac.uk/~durranrj/

S. Vempala's book : http://www.ams.org/bookstore?s3=vempala&quicksearch.x=0&quicksearch.y=0&pg1=&s1=&op1=AND&co1=AND&pg2=&s2=&op2=AND&co2=AND&pg3=ALLF&op3=AND&co3=AND&fn=100&d=BOOK&p=1&u=&r=0&l=100&f=S&subj=genint

and this paper : www.cc.gatech.edu/~vempala/papers/robust.ps

.

My understanding is that the important ability of a successful LSM/ESN is the separability of different states in the reservoir. Its not very important exactly what these states are, per say, or HOW the reservoir propagates from one to another with certain input, it's a question of whether (and for how long) different inputs produce reservoir states which are seperable by the read out neurons.

By having a high dimensionality, reservoir states are more easily seperable BUT, certain reservoir structures of the same spectral radius may not be equally good for a given learnign task. Ozturk et al. (2007) show how ESN's can be designed within specrtal radius limitations to make reservoir states more seperable than in the case of a "random" spectral distribution. It may be a first step to answering your question...

M. Ozturk, D. Xu and J. Principe - Neural Computation (2007)

In addition to the mapping to a higher dimension (also see rigotti..fusi 2010 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2967380/ for some theory behind randomly connected networks), the recurrents allow for information that comes in at one time point to be passed around for a while within the reservoir, so that computations can be done on combinations of inputs that are not necessarily simultaneous.

When training "regular" RNNs with stochastic gradient descent, you're sampling parameters from a large family of possible weight matrices comparing the transform applied by the mixture of those weights to your target. Typically these networks involve 10^1-10^2 units, and thus can model 10^4ish interactions.

With ResNets, you've got 10^4 units and 10^8 interactions. Instead of tuning one weight matrix carefully, you just create a much, much larger weight matrix and sub-sample linear mixtures of reservoir activity to comparing to your target. It's a bit like creating many, many "traditional" RNNs in parallel, and choosing the mixture of them that fits your output. In an extremely unlikely case you might imagine getting very lucky and creating somewhere in the ResNet exactly the RNN you'd have ended up with using SGD.

I think the intuition that tuning weights should give you an appropriate input output mapping is pretty surprising too! How do you know you have enough expressivity? The two approaches are a lot more similar than they get credit for, it's just whether you're sampling parameters serially or in parallel.

@Patrick: How is your answer related to Liquid State Machine? One still needs to train a ResNet!

If I remember well, LSM (which are closely related to ESN) synapses are also provided with STDP which means that not only the size and randomness play a role in the discrimination ability but also the network's ability to shift weights dynamically according to activity. (that added to the "supervised" piece of learning for readout neurons).