If you are looking for rigid object tracking with minimal prior information about the object, I would recommend the winner of the VOT 2014 challenge:

Danelljan, Martin, et al. "Accurate scale estimation for robust visual tracking." Proceedings of the British Machine Vision Conference BMVC. 2014.

DLib has an implementation of it:

http://blog.dlib.net/2015/02/dlib-1813-released.html

For tracking highly non-rigid objects (such as faces), cascade regressors are the state-of-the-art currently:

Xiong & De la Torre, "Supervised Descent Method and Its Applications to Face Alignment", CVPR 2013

Kazemi & Sullivan, "One Millisecond Face Alignment with an Ensemble of Regression Trees", CVPR 2014

The later one has an implementation as well:

http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html

One issue you will need to consider is how the kinematic (i.e. position & velocity) information is weighted/balanced with the feature information (i.e. size, shape & colour). Anyway, much of the algorithm/implementation details will depend on the nature of your particular problem (e.g. what objects are you trying to track and what does the background look like?).   

For deformable objects, cascade based regression are the state-of-the-art currently (in face alignment):

The original paper is: Dollar et al, Cascaded Pose Regression, CVPR 2010.

Then it is applied successfully for face alignment with some modifications:

Cao et al, "Face Alignment by Explicit Shape Regression", CVPR 2012.

Xiong & De la Torre, "Supervised Descent Method and Its Applications to Face Alignment", CVPR 2013

Kazemi & Sullivan, "One Millisecond Face Alignment with an Ensemble of Regression Trees", CVPR 2014

However, the tracker proposed by these methods just simply detecting the frontal face by face detectors and aligning the face, so the performance depends on the face detectors.

As in any Computer Vision application, the answer is context dependent. If you are not interested in measuring the deformation itself, but just in tracking the object. The main point is to build a detection/segmentation algorithm which may detect the object regardless its shape change. This can be as easy achieved as using a color based segmentation or background subtraction algorithm. It is actually a problem of detecting an invariant feature of the object which is not present in background or in non interest objects of the scene.

Then, tracking the object can be performed by a bayessian algorithm based only on the detected position of the object (just checking the consistency of the trajectory according to a dynamical or displacement model). One of the most useful algorithm to this end (is implemented in Matlab and OpenCv) is the Kalman filtering.

idTracker http://www.idtracker.es/

Digital image correlation technology. Look for Michael Sutton at al in the Internet