To put it in a nutshell, a system can respond with different state and output dynamics, given a particular input and certain disturbances.
To design a controller, one first needs to select a system model that should hopefully represent most of the response (modelled dynamics). A controller is designed assuming that the selected system model does a good representation of the dynamics !
But if most of the response was accommodated within the system model representation, then at least some of the response (however small or insignificant !) must have gone unrepresented in the system model ?
It is this "left over" dynamics that goes by the term unmodelled dynamics.
The common points of interest for the designer are:
Was the selected system model "accurate enough" to represent most of the dynamics ?
Was the controller design "good enough" with the control task, though some of the dynamics were left unmodelled ?
How do we improve both to do an overall better control job ?
Those are the basics ! For the details, you may wish to look up the suggested references.
Model-based controllers need the presence of a perfect mathematical model for the controlled manipulator and in this way though to be highly complicated and computationally time consuming for non linear dynamic system. Non model based controllers does not need an essential data of the parameters of either the manipulator or the actuators and subsequently no mathematical model for the system
Usually, dominant part of dynamic behavior of a system is formulated as some mathematical equations to create the "model". This model would help us to analyse and simulate the system and also design an appropriate controller. While, some behavior of the dynamic system, which is not often dominant, has been left for some reasons. The main reason is that this part is so complicate to model as a mathematical formula. Some times, this part has been left intentionally to make the model more simple.
Anyway, you should find some constraints for the unmodeled part, in order to design the controller carefully. In this way, the controller has been designed for the worst case and the performance will be decreased.
In addition to the aforementioned good answers, I wish to add the following. As, mentioned by Alexander, both linear and nonlinear systems have unmodeled dynamics. This simply refers to the fidelity of the dynamic model. Normally, a dynamic model is written in terms of a finite number of coordinates or state variables. Since the actual system is of infinite dimension, then we actually use a truncated system (or reduced order system).
Let me give an example for a state variable controller or a modal controller. In modal control of a dynamic system, for instance, one defines his dynamic model in terms of a set of significant modes (often spans the low-ferquncy subsystem). This part of the system is the controlled part (or the modeled part). The leftover modes (normally is the high-frequncy subsystem) are uncontrolled (or the unmodeled part).
Important note: the modeled dynamics are observable and controllable, while the unmodeled dynamics are not observable by the model, yet they are affected by the controller. This causes a problem called control spillover from higher modes from the uncontrolled modes; which can be destabilizing to the system.
If you model a higher order nonlinear system with a finite order (say n) linear parametric model. Here for input (say sort of random signal), the linearised model may give an output almost near to the nonlinear system output (if a good model). So the residual between the two is the un-modeled dynamics.