I guess that they will not fuse, because it is not as simple as inverting a matrix and you can not even invert all matrices. However, both disciplines should work together closely like all others working with models and simulations (=computer graphics) and those using experiments to gain new knowledge (=pattern recognition / computer vision). You find similar tandems e.g. in physics (experimental and theoretic).

For the last question exists an uncanny valley, which means that highly realistic imaging tends to make users feel uncomfortably. You can see this in the design of robots: the more human they look the spookier they are. Therefore robots are designed to look more like characters from comic books, which appear rather cute.

There is some work on establishing a "graphics Turing test", trying to find out how much computational power is needed to make an image indistinguishable from reality. See

http://arxiv.org/abs/cs/0603132v1

http://arxiv.org/abs/0801.1500v1

which conclude that the need is about 1036.8 TFlops peak and 518.4 TFlops sustained. That is a bit of processing power, although we are likely to get there eventually.

There is also plenty of work of calculating the BRDF and texture maps from natural scenes. It looks pretty decent, but I don't know how well it scales.

I suspect the real problem is indeed making a robust 3D object mapping that doesn't get surprised and "flips" when new data arrives. Especially barely seen objects are likely underconstrained, so if you move so they are gradually revealed there is a good chance they will misbehave as the model gets updated. Nothing too different from our own perceptual system, but likely more detectable when done by a machine.

With the advances in computer graphics during the last decades, this question certainly comes into focus of computer vision people more and more. There were and are projects going on using computer generated (rendered) images and video streams to test and benchmark computer vision algorithms, see e.g. http://www.vis.uni-stuttgart.de/index.php?id=sabs

I think the last question is worth being discussed in greater detail, since the term "indistinguishable" has different meanings depending on the step of image processing and whether it is answered by a machine or human. Computer graphics can generate images that humans judge indistinguishable from a photograph of the scene, although a difference image on pixel level would show significant differences in parts of the image, that the human eye or the human brain simply ignores. Work on this question has been published by Daly "The Visible Differences Predictor".

I agree with Hubert that both disciplines should work closer together (and i see this process picking up speed in research) because with recent advances in physically correct rendering CG algorithms and the massively increasing GPU processing power, CG generated images and video streams will become a worthful input to computer vision.

Interesting philosophical discussions Sam and Hubert... We do need these fundamental questions and have to reflect on them both from a mathematical and an engineerikng perspective - I guess we all know the 'macho' joke about the beautiful princess and the mathematician as well as the engineer trying to approach her having the distance every step of the way ^_^ Q; what is close enough.

I am fascinated by Hubert's matrix-inversion analogy. By the same token, video as well as speech/audio codecs carry out a similar action: first compress the (video/audiop) scene and then try to resynthesize it, so that they 'look or sound' very similar, but never perfect... And our community designs transcoders amongst standards too, whilst minimizing impairments.

Best wishes

Lajos

Interesting discussion, indeed. Early vision tasks can be massively parallel, and performing them on a GPU is a natural fit. The problem with the current technology, is a fairly long bus: one has to push the captured images to a GPU, process them there, and pull them back to the CPU world for further processing. In a better world, one would perform all IP/CV tasks on a GPU-like device, where the captured images get straight from the sensor(s), which calls for implementing such specialized hardware and some SDKs (e.g. by extending OpenCL or CUDA). To make parallels with the bio-world: the vision part would correspond to seeing, while the graphics part would roughly correspond to dreaming. If attempted today, such complex devices may be quite expensive, but technologically possible. The real question is one of the motivation: before attempting any implementation, think what you are going to do with the results.

There is a fundamental aspect to your question that also needs to be asked. Is there a real demand for this? If there is then it ultimately is only an issue of engineering. In the early 80's Apple recognised the demand for a GUI melding text and graphics. As oneof the first to use its technology to do some of the first computer-based graphic design, I as a businessman did not need further convincing of its potential nor did any of my colleagues and competitors. Now we don't think about that any more but the demand in the marketplace and Apple recognising it and responding to it is probably what saved Apple from being overrun by Bill Gates. Ultimately, if there is a will there will be a way.

There is a more specific area where I think graphics and computer vision will merge. A rapidly expanding research field in robot vision is the simultaneous pose estimation and reconstruction of a scene from a video. By tracking points in the video you can calculate both the camera poses and the 3D position of the points using methods such as SLAM or PTAM. Rather than tracking points, you can do a dense reconstruction for all pixels or in a given volume - mostly referred to as DTAM. My colleague Andrew Davison at Imperial is a leader in this field.

Generally, DTAM methods make very simple assumptions about the surface properties - assuming just Lambertian reflection you can just expect the projection of the same 3D point to have the same colour in all video views. However, if you incorporate other properties, such as specular reflection, you need to have an idea what the lighting of the scene is. Some researchers are looking into this. In general you can say that the more accurate the graphical model the more accurate the reconstruction will be.

An area that interests me is registration/tracking/pose estimation for surgical scenes, where specular highlights are a significant issue for methods like PTAM/DTAM. You may already have a model of the scene you are looking at from preoperative images. In other fields such as camera-based navigation you may also have increasingly accurate predefined models of the scene. In this case a possible way to solve the problem is to match renderings of the model to each of the video frames. Again the more accurate the rendering the better the pose estimation will be.

It's unlikely that these problems will require the very high end of realistic rendering, but it is an area that links graphics and computer vision. Is this perhaps the kind of thing you were thinking of, Sam?

Small point to Hubert, by the way. Indeed there is an uncanny valley, which can be an issue. It is, however, generally accepted to be a valley. If you get realistic enough you can push through the valley and things become comfortable again. So while it should be taken into account, it shouldn't be taken as a reason not to pursue a particular line of research.

It is absolutely true, that it would be great if somebody builds a bridge over this valley.

Another point came to my mind yesterday evening: there exists a community between CG and CV, they work on visualization techniques for science, knowledge, big data, etc. As far as I know their work, they have a bias towards computer graphics and use image processing techniques from CV to visualize patterns, which may not be seen otherwise by the human eye.

I don't think a complete fusion is possible. Only "dense colored 3D structure reconstruction" is the inverse of CG, but CV deals with several other problems that do not require structure reconstruction: crowd counting, face recognition, OCR, behavior analysis, gesture recognition, classification of animals (e.g., fish), obstacle detection, quality measuring and assurance... In a general sense, CV deals with "extracting meaningful information from images". While approaching some of those problems from the reconstructed structure may seem appealing, there are far more efficient techniques.

Off the cuff, I think the title of your question is much more interesting than the reality of the subtext description. We will clearly get to the point where we can do a seamless/simultaneous "scan to render" process, but that's fundamentally boring. We already know how to do this, we just need to pedal faster.

Where I think the fertile ground for thinking lies, is in the area of thinking of CV as /potentially/ being implementable, with astronomically faster hardware than today's, as a stochastic forward projection of hypothetical geometry, through hypothetical rendering pipelines, and comparison of the resultant images with the "real" image.

Depending on how one defines one's "real image" (do we get to use cameras that return depth as well as color? sample reflectance with different light sources, etc?), this turns into a problem where one can think about the system as providing probabilistic rankings of possible scene descriptions and rendering pipelines, that would have produced images closely approximating the CV input. If it's not already being thought about, I would guess that there are some interesting opportunities in that, for theoretical analyses of how one usefully samples the space of possible scene descriptions and focuses analysis on regions of that space, and on rendering pipelines that maximize the likelihood of producing the observed input.

I beleive to some big extent, yes. As a CG profesional I have found the need to use CV techniques more frequently. Particualrly in fields such as mixed reality where both CG and CV come together naturally. Additionally, CV techniques are useful to produce CG models. On the other hand, we are finding computational geometry (very common in CG) useful for CV

Definitely many similarities. A fundamental difference exists however in that CG is essentially the physics of light and image formation and CV is the "physics" of the human mind and image understanding. Reasoning that one is the inverse of the other is a fundamental error--in my opinion. The front end of CV can definitely benefit from the mathematics of CG but forcing the fusion of CG and CV can lead to frustration. Just my two cents.

The future does not simply unfold; it is something we create through vision, effort, funding priorities, and so on, which makes prognostication difficult with respect to your question.

It is likely that there will be no one point at which capture to render becomes indistinguishable from a video stream but rather many points depending on the complexity of the scene. Office scenes are less complex than scenes of the natural world; human artifacts are typically less complex than human beings. The point is likely to come sooner for scenes that are more compressible using model-based compression. With enough groundwork and assumptions, for very simple scenes, we're already there.

At present, the uncanny valley in computer graphics is mainly an issue for scenes that contain human or anthropomorphic characters. In part because of the uncanny valley, the human effort required to create scenes that are indistinguishable from reality is much greater for these kinds of scenes. The computer processing power required is also greater. However, this is not an insurmountable problem.

There is also the issue that observers can become sensitized or desensitized to variation from photorealism in computer animation. Computer graphics animators are better at spotting a fake (i.e., computer-generated human being) than ordinary people; however, when they do, their negative emotional response may be less, because they have habituated to computer-generated characters.

Article Too real for comfort? Uncanny responses to computer generated faces

Article Individual differences predict sensitivity to the uncanny valley