All UAV are unstable. Imagine you have a quadrotor and you introduce for example some kind of input system to the rotors (a small force, the same for all). Then the qudrotor will start to gain some altitude and if you don't eliminate the input the quadrotor will never stop. In other words, unstable means that the system response tends to the infinity. Mathematically, if you verify the eigenvalues you will find positive and zero eigenvalues which logically means unstable.

However it s quite easy to stabilize quadrotor, by instance see

Jurado, F., Castillo-Toledo, B., & Di Generao, S. (n.d.). Stabilization of a Quadrotor via Takagi–Sugeno Fuzzy Control.

I have to disagree. The system whose response tends to infinity is called astatic. One can think about simple motor with incremental position measurement. However it is obviously stable.

Instability can be viewed as some undesirable behavior due to some kind of "wrong" input signal (e.g. to high value or lack of it). The simplest example is a inverted pendulum. When the control signal is removed the arm is falling down.

In case of quadrotor it is pretty the same: to low control signal causes falling and to high value will result gaining the altitude.

As I know an Astatic system is one that doesn't have any tendency to take a definite position or direction. I agree with the concept that inestabilty is an undesirable behavior due to a strange input signal, however, once the system presents this behavior, it tends to diverge, which in practical terms will be reflected in the system's malfunction, but in a teorical speaking, Mathematically, the system response will tend to infinity due to the divergence.

An astatic system is by definition unstable. A pendulum is a stable system. If disturbed, it will swing left and right until gravity returns it to its original position. But in the case of the quadrotor, tend to the infinity is just one example of inestability. The easy way is to see that without any control the quadrotor will never stabiize in any position and it the worst case it will destroy the system. Probably it is more important define stability in terms of the equilibrium points in the case yo are planing to control a linearized system of the quadrotor. We say that the equilibrium point zero is stable

if for any initial condition x (0) sufficiently near to zero, the time trajectory x ( t ) remains near to zero [2]. (More general definitions are given in terms of asymptotically or exponential stability).

[1] http://netwar.wordpress.com/2007/07/25/stable-and-unstable-systems/

[2] A Course in Robust Control Theory. Geir E. Dullerud and Fernando G. Paganini.

Stability is a complex topic, particularly for nonlinear systems. Inherent stability is the tendency of an aircraft to return to straight and level flight, when the controls are released by the pilot. A quadrotor will diverge from hover unless stabilizing control is applied. This can be proved by Lyapunov's first method by linearizing the dynamics around hover and inspecting the eigenvalues, most of which will lie at the origin. To be stable, the eigenvalues must all have negative real parts. Note that the stability has nothing to do with the actuation, although the stabilizability will.

From a general systems theory point of view, I have to disagree with parts of the last answer.

Given a nonlinear system dx_dt = f(x) with equilibrium x= 0, i.e. f(0) = 0.

If the all eigenvalues of the Jacobian linearization at x=0 have strictly negative real part then the nonlinear system is locally asymptotically stable. If there exits one eigenvalue with strictly positive real part the nonlinear system is unstable. If, however, there are eigenvalues with strictly negative real part and some on the imaginary axis, i.e. zero real part, it is not possible to determine stability of the nonlinear system via the Jacobian linearization.

In this case you need to use higher-order approximations based on the center manifold theorem. For details see H. Khalil: "Nonlinear systems", chapter 8.

Easy examples for the trickyness of linearization with eigenvalues on the imaginary axis are dx_dt = x^3 and dx_dt = -x^3. The former is unstable, while the latter is asymptotically stable.

Coming back to quadcopters:

If you linearize your quadcopter model and you find eigenvalues with strictly positive real parts, then you have verified its instability. If they have strictly negative and zero real parts, you need to do a different analysis.

Quadrotors systems have no equilibrium points to discuss their stability, i.e. there is no x to satisfy f (x)=0, where \dot {x}=f (x).

The quadrotor dynamic system structure is

dx/dt = f(x,u); y = h(x,u) where u is the control (input) and y the observed outputs. The equilibrium point variety is given by the set of points x and u such that f(x,u) = 0. This variety is not void of course.

The trim (equilibrium) is trivial for the quadrotor. The collective thrust is mg and both the differential thrusts (pitch and roll moments) and the torque (yaw moment) are zero. The resulting trim state is level hover.

If we consider the control input (u) to be non-zero, then the stability of the system will depend on our definition of stability. In terms of input-to-state stability, stability of a system defined by \dot{x}=f(x,u) usually requires the unforced system (u=0) to have a stable equilibrium point. Now the problem is that the unforced system of quadrotors has no equilibrium point. Nonetheless, I think we should consider the unforced system if we are discussing the "inherent" stability/unstability.

"Inherent stability" is a concept from aeronautics. To repeat - inherent stability is the tendency of an aircraft to return to trim (equilibrium) when the controls are released by the pilot. A heavier than air vehicle cannot be trimmed at constant altitude without power, i.e. an "unforced system" equilibrium point is impossible. An unpowered aircraft with a constant glide slope is the closest one could get, even then the elevator must be deflected or trim tabs set, i.e. there is an aerodynamic forcing from the controls. I can imagine that there are some stability concepts for other classes of systems (earth-bound structures ?) where unforced equilibrium is a necessary condition, but this is not the case for inherent stability of air vehicles.

Incidentally, the inherent stability of a quadrotor can be improved by angling the thrusts vectors to meet at a point above the centre of gravity. This essentially provides the aircraft with dihedral. However it also results in a loss of performance, so most quadrotor designs have parallel rotor axes. Note also that the rotating rotors improves the inherent stability due to their gyroscopic effects.