It models traveling with fixed velocity (in size and direction) for a unit period of time along great circles on earth (i.e as the bird flies). Equivalently, it models traveling in a fixed direction, for a certain period of time along a great circles. The exponential map is of great use in Riemannian geometry because it relates in a bijective way a neighborhood of a point to tangent data in this point. For example it gives a map of the earth centred in Amsterdam that describes Bangalore as flying (with 1000 km/hour) for 6 hours in south eastern direction, London as flying 15 minutes south west west, Rome one hour south south east, New York 6 hours west etc. This description breaks down for New Zealand because it is antipodal to Amsterdam: Wellington is both 20 hours south east and 20 hours north west.

Just a remark concerning the motivation and the terminology: If your manifold happens to be a Lie-group, or more specifically, a matrix group M such that its elements A can be represented in the form exp(Bt), then the Lie-algebra, that is the tangent space of M at the identity I, is the vector space of such matrices B. The "exponential map at I" in the sense of differential geometry is, in this case, the matrix exponential map. Just to consider the simple case: O(2)=S^1: In this case the tangent space of O(2) at I is the set of skew-symmetric matrices \begin{pmatrix} 0 & -s \\ s &0 \end{pmatrix}, which can be identified with the imaginary numbers \{ ix | x\in\R\}. The matrix exponential

T_I O(2) \to O(2) can be written as ix \mapsto e^{ix}, which is exactly the map that corresponds to "following the circle with speed x for time 1."

A particularly simple example, a special case of the Lie group, has been used "forever": the usual real or complex exponential function from the field to the nonzero elements. For the matrix Lie groups the usual exponential function series or differential equation is satisfied and is the Riemannian metric exponential for the nicest invariant metrics.

As Professor Bishop pointed out the name is a relic from matrix exponentiation used in Lie group theory but let me point out some basic facts from Riemannian geometry.

The exponential map defines a special coordinate system near a point (called normal coordinates) and if you expand the metric tensor in those coordinates the first order terms (Christoffel symbols) vanish and the second order terms give you the curvature tensor on the nose. Besides, the second variation formula for geodesics (Jacobi fields) controls the derivative of the exponential map and that can be used to prove comparison theorems in Riemannian geometry. In fact, understanding the way geodesics are "focusing" or "expanding" is also important in General Relativity.