The statement about model-based and model-free doesn't mean anything. And the question, as given, is meaningless, since ``as well'' doesn't seem to relate two issues. In any case, the problem is, given the space of couplings of a learning model, to find a configuration that realizes the associations one is interested in, provided one has shown that such a configuration exists-not all associations can be so represented, ``learned'' by any given architecture. The way one can find the configuration is ``model-free', in the sense that it doesn't depend on the association to be learned, nor on the network's architecture; of course, the choice of parameters of the learning procedure will depend on the model, however. 

The statement about model-based and model-free doesn't mean anything. And the question, as given, is meaningless, since ``as well'' doesn't seem to relate two issues. In any case, the problem is, given the space of couplings of a learning model, to find a configuration that realizes the associations one is interested in, provided one has shown that such a configuration exists-not all associations can be so represented, ``learned'' by any given architecture. The way one can find the configuration is ``model-free', in the sense that it doesn't depend on the association to be learned, nor on the network's architecture; of course, the choice of parameters of the learning procedure will depend on the model, however. 

.

the RL terminology is very overloaded and may lead to many misunderstandings

Stam's answer above is quite right ; just to make it concrete, let's take an example : in a large state/action space, you might have to resort to some kind of approximation of the value function so as to implement Q-learning

therefore :

• you build a model of the value function
• but you are still in a model-free approach since you do not try recover the structure of the system (you do not care about transition probabilities, for instance, since you use Q-learning)

.

so, most often, in a large state/action space, you have to resort to some kind of modelling of the functions your RL approach relies on (the value function for the Q-learning example above), so you are building models for these functions but this has nothing to do with the model-free / model-based classification of the RL approaches (which lies in the reconstruction of the system structure or not)

.

now, reading the above, i'm not sure i made things clearer ... :((

.

I think we're getting two discussions mixed up here.

The model-free/model based discussion is not really part of the main question you ask about continuous state and action spaces. The terms model-free and model based have very specific meanings in RL and unfortunately the use of the word "model" makes it very confusing as people have assumptions about that word. Let's just deal with your main question: "Does anybody know what are the main approaches of reinforcement learning in continuous state and action spaces?"

The opposite of continuous state would be the tabular case. That is, for each unique state you can look up your value in a table - states are discrete. For continuous state we use function approximation. That is, we take in our data about the world, and pass it through some function which converts it to a feature vector. It is very common for us to use a simple linear function approximation. So for our value function we might learn a corresponding weight vector, w, which is the same length as our feature vector, x. The value of a state is then given by a dot product between w and x: v=w'x. This is covered in Rich Sutton's text, at least the second edition.

A common technique for featurizing scalar signals is a non-linear transform called tile coding (also sometimes called CMAC). Also covered in Rich Sutton's text. Additionally, Sutton provides a library for tile coding (http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/RLtoolkit/tiles.html) There is also a good explanation of tile coding in "Multi-Timescale Nexting in a Reinforcement Learning Robot", Modayil et al., 2014.

Now for dealing with continuous actions look at actor-critic algorithms, they can deal with both continuous state and continuous action. These allow you to separate value function estimation (critic) from your policy learner (actor). Actors can produce either discrete actions, or continuous ones. An example of an actor which can produce continuous actions is a Gaussian distribution. Rich Sutton discusses actor-critic in his text book (See ch 11 of the second version of his textbook, still in progress, but there is a link to the PDF at the bottom of https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html). You can see "Online human training of a myoelectric prosthesis..." (Pilarski et al. 2011) for an example of using actor-critic with a Gaussian distribution for continuous action.

That should at least give you what you need to look in the right direction.

Now for a quick clarification of "model-free" and "model-based". On one hand either can be said to use a "model", in the sense that they learn some mapping. Our learned weight vector, for function approximation, or our lookup table, in the tabular case, might be considered a model in that case. But this is NOT what is meant by the use of the word model. This can be a source of confusion about the two.

Model-free means that we are learning directly from data samples, there is no state transition model involved. Model-based means that we use a state transition model (either learned or given). Model-based methods are associated with planning, whereas model-free might be used in real time.

Article Online human training of a myoelectric prosthesis controller...

It may outside the question. I perceive that some supervised machine learning or hyper-heuristics ideas here. Perhaps it would be helpful to see how this type of reinforcement learning has been applied in these disciplines.

http://rlbook.busoniu.net/

http://burlap.cs.brown.edu/tutorials/scd/p1.html#intro