Did you acquire the data under controlled conditions? I.e. are the actor and the background easily discriminated on colour?

Dear Kennedy, 

Thank you very much for your attention. Unfortunately, we did not perform any screen matting like control for this data acquisition since the data is taken using Kinect. At the start of our data acquisition settings, we wanted to rely on the 3D skeleton data generated by the Kinect, but we found the data has many errors that might be caused by fast movement. On the other hand, the video data seems to be OK.   

We can set up the data acquisition again using controlled conditions so that the actor and the background are easily discriminated. In such case, could you suggest the best way to segmented the video frames and how to extract good features for different action classification? 

If your actor is wearing red and the background is blue, then you could just make a colour histogram for each frame. Use it to find the mean x, and covariance matrix P, of the actor and background. These will be a 3x1 vector and a 3x3 matrix for an RGB image. Then in a given image assign each pixel to either the foreground (i.e. the actor) or the background. Use a distance measure d to make this assignment, where d = (x-y)'*inv(P)*(x-y), where y is the RGB intensity vector of each pixel. I,e to which distribution is a given pixel closest to. You may then need to apply a filter to remove speckles or to identify the biggest blob (the actor).

I haven't done this before, but this would be the first method I would try.         

Just a suggestion to try Sir , since is not my area :

- LVQ

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=1115147

- Intrinsic distance

http://link.springer.com/chapter/10.1007/978-3-540-77255-2_57

- Motion History Gradients

cgit.nutn.edu.tw:8080/cgit/PaperDL/kyc_091020140619.pdf

Good luck sir :)

Dear Kennedy and Widiyanto,

Thank you for your answers. We are now looking into it.