I have a datasets which is comprised of skeleton XML files and RGB videos, I want to apply classification, my question is how we can use both data features as whole input?
Is the term you used "skelton" equivalent to "topological skeleton"? If it is and you want to create a classification you will have to move into volumetric color space. For this use a color cube or a color cone and identify a subset volume of color you want to associate. Within your subset color volume, have it tied to your XML topological skeleton to extract your classification. (US color = colour)