I am thinking about a project where I use data from a kinect sensor to recognize gestures. I will get a training set of 14.000 labeled scenes from a vocabulary of 20 gestures. I intend to use the multimodal inputs (RGB, RGB-D and Skeleton) on a Deep Belief Network.
The idea is to implement 3 phases: Unsupervised Feature Learning with Restricted Boltzmann Machines to initialize supervised feedforward neural networks with backpropagation and then testing.
I am not yet sure about the preprocessing of the data. But I guess I will have to do a lot to compress the input. Any ideas here are highly welcome, too.
Does that sound like it makes any sense at all? Will I need a supercomputer or something? How can I calculate the computing time? Are there any ground rules?
The idea is inspired mainly by the paper "Multimodal Deep Learning" by Jiquan Ngiam, Aditya Khosla, Andrew Y. Ng and others. I am pretty new to all this and would love to hear opinions or advice. Anything is welcome. If you need some more input, I am happy to provide if possible.
Here is another question I have about the project:
https://www.researchgate.net/post/Deep_learning_implementation_in_Java
Another interesting approach could be to use stacked autoassociators. In its basic form, it is a MLP with a compression hidden layer, which is trained to reproduce the inputs on the output layer. With linear neuron it performs a PCA. With more than one hidden layer with non linear neurons, it performs a "non-linear PCA". I have tested this approach for Face Detection. It was trained to project input image on the set of face images: face images were reproduced identically, and non face images were reproduced as the closest face image.
see:
https://www.researchgate.net/publication/3193210_A_Fast_and_Accurate_Face_Detector_Based_on_Neural_Networks
This model was also used for hand gesture detection in an hand gesture recognition system.
see:
https://www.researchgate.net/publication/3845484_Hand_gesture_recognition_using_input-output_hidden_Markov_models
This model is also used for unsupervised learning representation by stacking the hidden layers. The denoising autoencoders are trained incrementally to reproduce the preceding hidden layer with an additionnal white noise.
see:
http://dl.acm.org/citation.cfm?doid=1390156.1390294
Article A Fast and Accurate Face Detector Based on Neural Networks
Conference Paper Hand gesture recognition using input-output hidden Markov models
Another interesting approach could be to use stacked autoassociators. In its basic form, it is a MLP with a compression hidden layer, which is trained to reproduce the inputs on the output layer. With linear neuron it performs a PCA. With more than one hidden layer with non linear neurons, it performs a "non-linear PCA". I have tested this approach for Face Detection. It was trained to project input image on the set of face images: face images were reproduced identically, and non face images were reproduced as the closest face image.
see:
https://www.researchgate.net/publication/3193210_A_Fast_and_Accurate_Face_Detector_Based_on_Neural_Networks
This model was also used for hand gesture detection in an hand gesture recognition system.
see:
https://www.researchgate.net/publication/3845484_Hand_gesture_recognition_using_input-output_hidden_Markov_models
This model is also used for unsupervised learning representation by stacking the hidden layers. The denoising autoencoders are trained incrementally to reproduce the preceding hidden layer with an additionnal white noise.
see:
http://dl.acm.org/citation.cfm?doid=1390156.1390294
Article A Fast and Accurate Face Detector Based on Neural Networks
Conference Paper Hand gesture recognition using input-output hidden Markov models
Potentially, you could utilize theano implementation of deep learning which utilizes GPU computing power....
http://deeplearning.net/software/theano/
Maybe these resources help you to find what you want:
http://neuralnetworksanddeeplearning.com/
http://markus.com/deep-learning-101/
http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf
https://www.coursera.org/course/neuralnets
Regarding the compression of the input, the work from Geoffrey Hinton and co-workers regarding Semantic Hashing might provide a possible solution. They work with extremely high-dimensional word-count vectors for classifying the contend of a texts. Similar to the idea you mentioned, they use several layers of Restricted Boltzmann Machines to compress the input and learn then hash codes for the individual texts. The learning method is set up such that texts similar in contend produce hash codes that have small hamming distance. Hinton also used a very similar approach for image recognition.
Regarding the software, I would also strongly recommend the Theano framework. Most other software models your neural networks as graphs of objects that ist created at runtime. In contrast, Theano generates C-code that represents your network and custom-compiles it for your computer architecture. Thereby, the compiler can already do a whole number of performance optimizations. In my experience, Theano is one of (or even the) fastest software around. As already mentioned, it supports GPU computing. However, also its conventional multi-threaded implementation is extremely efficient.
If a scene means a sequence of frames for a single gesture, 14000 does not seem to be a big number for a small scale experiment with Deep learning. You can easily learn how to use the code on Geoff Hinton's page , or Ruslan S. page and start building up from there. Before you go multimodal. I suggest to invest some time on researching how to extract features that reflect the temporal structure of your data and just start with the skeleton data. I recommend to check Graham
Taylor's dissertation "Composable, distributed-state models for high-dimensional time-series"
Perhaps this other approximation ("Learning Feature Representations with K-means"-Adam Coates and Andrew Y. Ng ) could be useful for reducing computing time:
http://link.springer.com/chapter/10.1007/978-3-642-35289-8_30
As already indicated by several authors the approach you proposed seems valid. In terms of algorithms, both the Deep Belief Networks from Toronto as well as the cascaded sparse filters approach from Stanford (http://fastml.com/deep-learning-made-easy/) have been known to give good results in a large category of problems (with some parameters tweeking).
One difference is that the DBN is a full fledged classifier (including the supervised part) while the Stanford method learns the features that can be then used in a broad range of classifiers (in the link above Random Forest is used for the supervised part). I would guess that DBN is more computationaly heavy (but can produce great results wth relatively small effort) while the other apprach is lighter and can be applied in a hybrid (non purely Deep Learning) setup if you do not want to completely give up on feature engineering.
In the later case for activities having temporal properties I would also consider the Fisher Kernel (and derivation of Fisher Vectors) which is basically a kernel of a generative model. It uses second order statistics that have been found very usefull for preserving certain properties of the problem that many features can not easily capture.
Thank you all for the answers so far. I really appreciate it. The response rate and quality tells me, it's a good project and I will try to realize it.
There were many new aspects and ideas I wasn't yet aware of. I find the subject very interesting and will be digging more into it. You probably might see more questions in the future. Thanks!
- Badges
- Science topic