Hi Soumia Mehimeh !

First of all, the transition probabilities from one state to another usually depend on two things:

• the environment dynamics
• the agent's actions

The environment dynamics is simply the internal logic of the environment. For example, let's say you are playing poker (texas hold'em), and three cards are shown in the table. After the betting round, one new card will be shown, right? This means that the MDP will transit from one state where three cards are shown, to another where the fourth card will also be visible to all the players. This fourth card is independent from the actions of the players, and only depends on the environment dynamics

Also, between each part of the turn, players take turn to place some bet! This means that the state of the environment changes whenever a new bet is placed. This is where the environment state changes because of a player action: more money will be on the table, and in turn some players may be more tempted (or not) to keep playing (place the bet, or raise) or to just fold. In this case, the transition probabilities are somewhat more explicit with respect to other examples, like trading agents, in which the state of the environment might be a stock current price, which is influenced a ton of things, but also autonomous driving, walking humanoids robot, etc.

Now, in real life scenarios, you usually don't have access to these transition probabilities (since this is just not how the real world works). However, you can use Deep Neural Networks, among other approaches, to try to estimate the transition probabilities from one state to another, and therefore make intelligent decisions to try to maximize the reward.

This is the reason why in the most recent reinforcement learning papers, in the section in which they describe the Markov Decision Process, the state transition probabilities are usually unkown, and have to be "derived" from direct experience with the environment.

I hope I've been helpful!

From a control engineering perspective, any standard ordinary differential equation system or state-space model can be considered a special type of MDP which internal dynamics are deterministic and can be fully defined by a proper mathematical model. Although classical MDPs are often considered of stochastic and unknown nature, this assumption is not mandatory in the RL context, i.e., assuming some deterministic model is also fine and also often used in model-based RL (being closely related to model predictive control).

Since most of the environments are unknown and modelling these environments are difficult particularly determining the transition function. Therefore we can use model free RL like Q-learning and design reward function such that handles our problem goal.