I'm working on an time-discrete Dec-MDP where my states are defined by a network's topology at each moment t. Basically, the nodes are agents in 3D space and the edges are links between them (if they are close enough to communicate) so the topology is dynamic. But, the environment we're modeling with this Dec-MDP is ruled by deterministic differential equations: the agents move following preset trajectories and the actions taken by the agents (taking shots of some targets if they are in their trajectories, and communicating with other agents) doesn't affect the state of the environment (network's topology). The reward however depends on the task execution speed and quality, the latency of communication, the battery charge of the agents, and some other factors. I'm not sure if considering the topology of the network as the state of the environment makes sense, or if it's enough at least to describe the environment. Should I consider other factors?
Yes, and there are lots of papers that have considered these kinds of problems. As for continuous state (or observation in POMDP), the DL network (whether DNN, LSTM, or CNN) can handle this. As for the second part of your question, there is no necessity that the action should affect the state space. If you only train your agent to properly act in different states, it is still an RL problem. There are several papers in wireless communications where the state (such as channels) is independent of the executed action while the action affects the reward (for instance, the data rate). However, it is important to properly design your state vector and add different features. Sometimes, it can be beneficial to integrate the previous action into the state vector and feed it to your DNN. It is noted that, if you work on a single-agent problem, usually the lower discount values are better when the state evolves independent of the action.