Hello Ilham,

Problem 1:

This problem is very similar to the calibration of cameras in stereo vision problems. What I see as being the main concern here is that the two images are not easily matched by references points, which essentially rules out automatic calibration techniques. In this case, I would suggest defining the transformation matrix between cameras by hand, and using the geometry of the system. A useful resource may be: http://www.cse.iitm.ac.in/~vplab/courses/CV_DIP/PDF/GEOM+PROJ+STEREO.pdf

There are also online calculators that may be useful:

https://nerian.com/support/resources/calculator/

Problem 2:

I think the easiest solution here will be down-sampling the higher resolution image, or up-sampling the lower resolution image with interpolation between pixel points. Now, a better solution would be to use the transformation matrix from before to project the segmentation results of one image into the other. Assuming you are taking the segmentation from lower resolution and projecting into the higher resolution, you will have a sparse segmentation result. From here you can use the RGB information between pixels, or interpolation between pixels, to refine the resolution of the segmentation.

I hope this helps.

Hello Ilham,

Problem 1:

This problem is very similar to the calibration of cameras in stereo vision problems. What I see as being the main concern here is that the two images are not easily matched by references points, which essentially rules out automatic calibration techniques. In this case, I would suggest defining the transformation matrix between cameras by hand, and using the geometry of the system. A useful resource may be: http://www.cse.iitm.ac.in/~vplab/courses/CV_DIP/PDF/GEOM+PROJ+STEREO.pdf

There are also online calculators that may be useful:

https://nerian.com/support/resources/calculator/

Problem 2:

I think the easiest solution here will be down-sampling the higher resolution image, or up-sampling the lower resolution image with interpolation between pixel points. Now, a better solution would be to use the transformation matrix from before to project the segmentation results of one image into the other. Assuming you are taking the segmentation from lower resolution and projecting into the higher resolution, you will have a sparse segmentation result. From here you can use the RGB information between pixels, or interpolation between pixels, to refine the resolution of the segmentation.

I hope this helps.

For the second problem, I'd advise against down-sampling, as it means that you lose information (unless if your application is really time-critical.

Up-sampling and super-resolution techniques from multi-modal cameras have been applied to this type of problem before. Have a look here: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.645.7987&rep=rep1&type=pdf

Thank you very much, Mr. Garrett Winkelmaier and Mr. Geert De Cubber , for your answer and the time you have given me.

To be clearer about the 2 problems I encountered I will note the following things :

1) We suppose that we have a database already acquired so we do not have the information concerning the camera (Fx,Fy,..).

2) For the 1 problem it is a problem concerning the alignment of the two images depth and color and here is an example of this problem : color, depth, the third one is the color image segmented using the depth image, and you can see that there is a segmentation problem at the top since the two images are not aligned

3) For the 2 problem it is concerning the resolution of the 2 images for example: for the color image I have a size of 640*480 and for the depth image I have a size of : 320*240 so segmentation is possible if I can find a method to make them with the same size but keeping the location of the information to allow segmentation.

Your depth image does not seem to include the information above the hand, so the depth data cannot be used to help with the segmentation, so I think it's quite normal that you arrive at this solution.

Thank you, Mr. Geert De Cubber , for your answer.

I hope there is a way to solve both problems and if you have an idea I will be grateful.

Hello Ilham,

I am new to CV so, not an expert but have an idea. You are using segmented depth image as a mask to segment the color image. So,

* First, you can create 2 masks from this depth-mask; first one is eroded "some amount" so that no unwanted information included which will be the lower-bound and the other one dilated "some amount" so that all desired information included that will be the upper-bound.

* Next, use lower-bound to segment color image and mark them as one component and find connected components of this segmented part within range of previously defined upper-bound.

I know it will be much more costy but may do the trick I suppose.

* You can also try an edge detection on color image segmented by upper-bound mask which may be faster...

Good luck on your project.

Just as an aside here, your colour image *looks* like it is interlaced - those little fringe/comb patterns round the edges of objects. If the length of them increases when objects move quickly and if they vanish when objects are static, then this is interlacing. What this means is that even (horizontal) lines come from one timestamp and odd lines come from a slightly different timestamp. So instead of, say, 30 frames per second, you actually have 60 *fields* per second.

It might be worth converting the frames to (half height) fields to make spatial synching easier. Otherwise, you'll be trying to match the smooth depth-camera edges with the fringed (comb-effected) colour image edges. You could apply a de-interlacing filter which might make things easier.

Thank you very much for your answer Mr. Ali Osman Atik and Mrs. Pamela Johnston.

Mr. Ali Osman Atik I'll try the walk you gave me,

For your method Mrs Pamela Johnston , I didn't quite understand the process I'd be grateful if you could clear things up for me a little bit.

Wikipedia https://en.wikipedia.org/wiki/Interlaced_video probably gives a decent enough view of interlacing. Basically, instead of 30 frames every second (say, other frame rates are available), you capture 60 fields every second. Each field is half the height of a frame and the interlaced frames contain data from two different time stamps. You then interleave the fields and let the human brain do the compensating.

For your problem, I assume you need to sync both spatially and temporally. With interlaced video, the spatial alignment becomes tricky due to the combing effects. The temporal alignment becomes tricky because you have data from two different timestamps in the same frame: which is the "correct" one? And the combing artifacts will make segmentation tricky, too - which is the "correct" edge?

I think the easiest way of avoiding these additional "tricky" parts is to deinterlace the video so that you have 60 fields per second rather than 30 frames per second. You can manually deinterlace just by sampling all the even horizontal lines for one field and then all the odd lines for the next. Your fields will be half the height of the frames, but you'll trade spatial resolution for temporal resolution and lose the combing artifacts.

Ordinarily, I'd say "don't downscale, you'll lose information" like Geert De Cubber but in this case, you're not losing it, you're trading it.