I would like to reduce sequences of about 50 RGB frames of 640x480 pixels to get a representation of the data I could feed into an deep neural network. The goal is to recognize a performed activity/gesture.
I have seen many examples for individual images with static gestures but I struggle to find practical examples using whole sequences with a dynamic gesture.
I have worked through the tutorial here* and they use the MNIST dataset to train the network. So their input are images of the size 28x28 pixel. I would like to use my data as input but I don't really know how to reduce and how much reduction is enough/necessary.
What I did until now is remove the background and then perform edge detection using the openCV Canny edges algorithm** which works fine but still leaves me with a lot of data.
I tried using image flow to generate something like a heatmap but I am not very happy with the results. I read about DTW or Space Time Shapes, but have not yet found a way to apply the theory.
So, do you have any hints, tips or links to papers, tutorials, presentations or whatever to help me reducing the video sequences without loosing to much data? I would prefer practical examples.
Thank you!
* http://www.deeplearning.net/tutorial/DBN.html#dbn
** http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_imgproc/py_canny/py_canny.html#canny
Seems like an interesting problem, but I don't think that a deep neural net is what you really want to use. However, here are some bibliography links I have on the subject of Spatial-Temporal analysis for Gesture Interpretations:
http://faculty.ucmerced.edu/mhyang/gist.html
You can also use some of these datasets to train:
http://homepages.inf.ed.ac.uk/rbf/CVonline/Imagedbase.htm#gesture
Another thing you might want to look into is the area of Content-Based Video Retrieval, that deals with the same problem of reducing image sets into features that could be indexed and later be retrieved. While not a lot of cohesive literature around, here is a paper I like to use as introduction:
http://arxiv.org/ftp/arxiv/papers/1205/1205.1641.pdf
Hopefully this helps!
Seems like an interesting problem, but I don't think that a deep neural net is what you really want to use. However, here are some bibliography links I have on the subject of Spatial-Temporal analysis for Gesture Interpretations:
http://faculty.ucmerced.edu/mhyang/gist.html
You can also use some of these datasets to train:
http://homepages.inf.ed.ac.uk/rbf/CVonline/Imagedbase.htm#gesture
Another thing you might want to look into is the area of Content-Based Video Retrieval, that deals with the same problem of reducing image sets into features that could be indexed and later be retrieved. While not a lot of cohesive literature around, here is a paper I like to use as introduction:
http://arxiv.org/ftp/arxiv/papers/1205/1205.1641.pdf
Hopefully this helps!
it is a very interesting problem. I think that the problem here is how to find the good attributes that descrive the videos.
Sabeur, that is exactly the problem.
Juan, thank you. Can you tell me why you think using a deep neural net is not a good approach?
While I personally don't like the fact that many people use them as a 'black box' method to apply to images and it just 'works', for some, not everybody. I also think that for a set of moving images that needs to be labeled as a certain event/activity, setting up a set of layers to do this right should be tricky and might add extra complexity to your network. However, it is an interesting approach and I wold be interested in seeing how well it works for you. Good luck!
On the deep net approach:
In many classification tasks, you will not see much (or any) improvement using deep nets over other learning algorithms (e.g. SVM). Deep nets are most suitable for large datasets and general problems where you do not have good hand-designed features for that particular task. Good examples are deep convolutional networks for image classification and phoneme recognition.
If you want to extract hand-designed features from videos and classify such feature vectors, pick any reasonable learning method that you are familiar with, and spend time on the features.
You should give more details about your data if you want better advice. What are the gestures? What is in the videos? How much effort are you prepared to invest? ...
Regards.
Thank you all for participating in the discussion.
Michal, the gestures are from italian sign language and the most active limbs are arms and hands. The shots of the different actors vary in background, clothing, skin color and lighting with occasionally partly occluded body parts. The data is taken from the "chalearn 2014 - Looking at people" dataset for track 3*. I am working with a set of 7750 gestures.
I have cut out the hands in each frame and merged them to only get a patch of 50x100 pixel on which I use HOG for feature generation. I guess that still counts as no hand-extracted features. And that is also part of the idea to use as little effort as possible for the feature extraction.
My problem lies in the next step of combining the single frames of a sequence to feed them into a simple Deep Believe Net. I want to throw something in and see how it turns out, then tweak the input data a little and see what changes in the result.
What do you mean by "effort prepared to invest"? Time, work,...?
*http://gesture.chalearn.org/mmdata
You can follow my human activity recognition paper and its implementation on Github.
Article Activity Recognition using Temporal Optical Flow Convolution...
https://github.com/Aminullah6264/Activity_Rec_ML-LSTM
- Badges
- Science topic