An "unstable periodic solution" is a solution that is both periodic and unstable. The terms "periodic" and "unstable" should be clear without ambiguity. In contrast, a "quasi-periodic solution" is a solution that is quasi-periodic, but not necessarily periodic. "A function ƒ(t) is called quasi-periodic if it can be represented in the form

f(t) = F(w_1* t, ..., w_m * t) where F" is a continuous function of period 2 pi in each of its variables. (from Friedman, 1967). You may also have a look at the appendix in Hale's book (Ordinary Diff Equations, 1980), but hopefully more (and better) references will be proposed by others.

Gunther's definition is basically correct but the conditions that $m \geq 2$ and that no frequency $w_i$ be a rational multiple of another $w_j$ are also necessary. Also, F is a continous function not F" (looks like a typo but may be mistaken for it's second derivative - while the condition will hold, it's not really necessary to put it this way).

The rationale behind these conditions makes the difference between periodic and quasiperiodic explicit and provides, in my opinion, a very simple geometric interpretation. Consider a trajectory constrained to the torus $T^2 \in R^3$, where the components can be defined $\theta = 2 \pi k t$ and $\phi = 2 \pi l t$ where $k$ and $l$ are real. That is, considered in isolation, the components trace a circle with regular periods, $1/k$ and $1/l$, respectively. If $l/k$ is rational (always true if $k$ and $l$ are integers) then the trajectory will eventually repeat itself. More specifically, if $l/k = q/p$ for coprime integers $p$ and $q$, the trajectory will wind itself $p$ times around the torus with respect to $\theta$ and $q$ times about the torus with respect to $\phi$, that is, since $p$ and $q$ are integers (i.e. complete revolutions), the trajectory is right back where it started. On the other hand, if $p$ and $q$ are not rational multiples of one another, the trajectory will never exactly retrace its steps. The former case, as it eventually repeats itself, is periodic. The latter case, as it can be cast as a generalization of a periodic function but does not ever repeat itself, is quasi-periodic. Note also that this two dimensional case is the simplest case in which this behavior can occur (hence the $m \geq 2$ condition).

A periodic solution that grows is an unstable periodic solution. For example, a negatively damped linear oscillator will have an unstable periodic solution of the form $x(t) = \exp(\eta t) \exp(i \omega t + \phi)$. As $\eta > 1$, the solution is an unstable periodic function as it has an increscent exponential envelope. Note that, as in the quasi-periodic case described above, the trajectory never repeats. However, as the solution is clearly decomposable into a secular growth and a strictly periodic component, this is not considered a quasi periodic solution (also $m < 2$, violating one of the conditions above).