Hi, in this paper, we listed a set of datasets that can help you,

http://dsn.sagepub.com/content/12/8/1550147716665520.full

These datasets are on activity recognition. I want hand gesture data set that contains RGB and Depth videos.

The following lists contain, among other things, dynamic hand gesture data sets:

http://www0.cs.ucl.ac.uk/staff/M.Firman/index.html

http://liris.cnrs.fr/voir/wiki/doku.php?id=datasets

 Hi Bindu

Try t check the following link

http://www.idiap.ch/resource/gestures/

Good luck

try this 

http://www.iis.ee.ic.ac.uk/icvl/ges_db.htm

this dataset it better matching your request

http://cvrr.ucsd.edu/vivachallenge/index.php/hands/

No, I have not

so which one of datasets you chose at the end ?

I have used SKIG dataset and DHG2016 dynamic hand gesture dataset. There is one more LISA dataset.

http://www-rech.telecom-lille.fr/DHGdataset/

https://lshao.staff.shef.ac.uk/data/SheffieldKinectGesture.htm