Image processing software, like all software, needs to be both verified and validated. Synthetic images are very useful during the software development process to verify that both simple aspects such as file handling, as well complex image processing algorithms are working properly. They can be used to test individual discrete functions as well as complex chains of functions to verify that code segments are working properly. They can also be used in "tolerancing" software such as testing to determine when discrimination algorithms begin to become less accurate, due to insufficient gray levels or subtle color differences. Other tolerancing examples include object size measurement testing, texture measurement testing, noise susceptibility, etc. This type of testing is especially important for medical image processing whether it is for pathology or clinical procedures.

Ground truth image sets, including calibration images, are useful for validation of software - testing the overall ability of the software to provide realistic results with real images, under real-life conditions. Ground truth images can also be modified and used to create quasi-synthetic images for testing susceptibility of algorithms to things like motion blur, image defocus, misregistration and noise.

The pros and cons of the kinds of image data you use, really boil down to using these image sets appropriately to both verify and validate your processing algorithms.

Synthetic images are great for testing (debuggin) your algorithms. Example: If you create a synthetic image with one egde, and your algorithms should detect edges, then it obviouly should detect the one edge in your synthetic image. This helps to integrate and to test your code automatically. However, if you need to publish something, in nearly all cases people (reviewers) are crazy to see "real" images from "known" datasets. Sometimes the evaluation on external datasets takes much more time than the actual algorithm design and implementation. However, since publishing is required for PhDs, his has to be done somehow. :/

I was also looking for synthetic data when debugging my depth maps algorithms. I used Blender to do so. I first created very simple scenes, such as a chessboard, cubes in a room. I further wanted to acquire images of a more complex realistic scene. I then used sample scenes of the Blender Open Projects, which were amazing. You just need to be patient when rendering.

I agree with Salim, depending on what you are evaluating, the kind of images to use will be different. I do not know datasets only with synthetic images, but there are some with real scenarios where synthetic elements are integrated. As an example, you can see the image dataset developed for the algorithm competition in the Video Surveillance and Sensor Networks (VSSN) conference in 2006.

Computer vision applied to real images reveals the features of real world, and this is the very point of computer vision. Computer vision applied to synthetic images will reveal the features of image generation algorithm and comprehension of its developer. What is the point then? Synthetics may be useful for debugging and simulating some rare situations. But using them to develop real-world application is just barren.

I feel that the question should have been narrowed down a bit, making clear what is meant by the term "synthetic images". According to how I understood your question:

Pictures of numbers or letters, e.g., are by their nature artificial images; pictures of landscapes yield completely different statistical pixel properties. The distinct statistical properties (Stoop et al, Phys. Rev. E 67, 061918 (2003)) have a significant influence on the results obtained by almost any pattern recognition algorithm - if this is where you are heading for.

Natural images can be approximated as much as desired by artificially generated images (see also fractal or jpeg 2000 compression algorithms).

What therefore matters-seen at least from this angle- are the statistical properties of pictures, rather than their origin.

Synthetic Images and capture images by some sensor are completely different. one can't correlate both for implantation through a certain algorithm. There features and somehow description are totally different. E.g. my photograph and my sketch are different .

I think the general point is the goal is to work with real world images. When your algorithms fall apart you can use synthetic ones (aka digital phantoms, artificial scenes, image structures etc, defined noise levels etc) to figure out what is going on and develop work arounds. "Synthetic" images are incredibly useful for debugging applications but in the end one always needs to demonstrate on real world data.

Images are always real. Artificial is a name to design the technological origin of the photons and all photons that hits our retina producing the primary isomerization of rhodopsin are real. And it have mass within an entangled system we know everywhere as the dual universe. I don't know "public databases for synthetic images" but it is possible to shortening by simulation with apropriate metamaterials the "distance" between images produced by technologies and our vision.

.

Realistic CG Stereo Image Dataset with Ground Truth Disparity Maps

Sara Martull et al

convincingly argues for the use of "synthetic" datasets for problems where annotation is difficult :

Stereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, research in some branches of the field has been constrained due to the few number of stereo datasets with ground truth disparity maps available. Having available a large dataset of stereo images with ground truth disparity maps would boost the research on new stereo matching methods, for example, methods based on machine learning. In this work we develop a large stereo dataset with ground truth disparity maps using highly realistic computer graphic techniques. We also apply some of the most common stereo matching techniques to our dataset to confirm that our highly realistic CG stereo images remain as challenging as real-world stereo images. This dataset will also be of great use for camera tracking algorithms, because we provide the exact camera position and rotation in every frame.

http://ypcex.naist.jp/trakmark/workshop12/2012wsproc/pdfs/Martull.pdf‎

(link to the dataset at the end of the paper)

.

Both are important. With synthetic images you can have an objective ground truth that you can use for a quantitative evaluation of your algorithm, and a quantitative comparison against other methods. Same thing with synthetically-generated cases, for instance, taking a real MRI image and applying a known transformation to it, to obtain an image registration test case where the true transformation is known.

Real images with manually labeled ground truth can also provide a quantitative evaluation, but you must take into account that different experts might label an image slightly differently, so there's an inherent uncertainty in the ground truth. Unfortunately, this uncertainty is rarely quantified because it takes so much work to do it.

Finally, real images without ground truth can also be useful for qualitative assessments, especially when you have a test case where most algorithms clearly fail and you are designing one specifically for those cases.

Perhaps

www.trakmark.net

is of interest to you. It contains real and synthetic sequences for vision based tracking / registration evaluation.

Medical images simulation is the right way for understanding!

Image processing software, like all software, needs to be both verified and validated. Synthetic images are very useful during the software development process to verify that both simple aspects such as file handling, as well complex image processing algorithms are working properly. They can be used to test individual discrete functions as well as complex chains of functions to verify that code segments are working properly. They can also be used in "tolerancing" software such as testing to determine when discrimination algorithms begin to become less accurate, due to insufficient gray levels or subtle color differences. Other tolerancing examples include object size measurement testing, texture measurement testing, noise susceptibility, etc. This type of testing is especially important for medical image processing whether it is for pathology or clinical procedures.

Ground truth image sets, including calibration images, are useful for validation of software - testing the overall ability of the software to provide realistic results with real images, under real-life conditions. Ground truth images can also be modified and used to create quasi-synthetic images for testing susceptibility of algorithms to things like motion blur, image defocus, misregistration and noise.

The pros and cons of the kinds of image data you use, really boil down to using these image sets appropriately to both verify and validate your processing algorithms.

Some recent approach target to produce realistic data based with synthetic data.

Noise reaction can be simulated and analysed. Make me think to: Siggraph2013: http://www.cs.utah.edu/~bergerm/recon_bench/

I mostly agree with N. Mackinnon, although my vision is more scientific than bussiness-oriented. It's my belief that the use of synthetic images should be restricted to exemplifying the output by given algorithms. This is so because these images can be designed (or modified) with that specific pourpose (i.e. to empower or diminish the perceived performance of some method or family of methods). It's common to reproduce in synthetic images those situations for which state-of-art methods fail, or those for which new proposals excel. But these are examples, not conclusive tests.

The use of synthetic images can never render in proper conclusions about the performance of the methods. Or at least, any conclusion you present will be shaded by the doubt of whether it was the dataset you prepare what leaded you to such conclusion. If you want to go beyond the mere example, synthetic imagery cannot be used. You must go for real image datasets, either problem-specific or general-pourpose ones. As a personal recommendation, I would go for the second type unless the problem-specific dataset has wide acceptance (and understanding) in the community you are addressing your paper to.

Beware of synthetic imagery; know its limitations and its usefulness.

It is an excellent source for INITIAL development of image processing algorithms.

But don't do extensive development of algorithms using such imagery.

There may be critically important differences between real and simulated imagery - especially in the microstructure of the images.

Medical imaging brings up an excellent point. While the results of many vision algorithms are presented (eg in PowerPoint) as a before and after (Real photo + labels), the truth is that most algorithms are not working directly from the original pixels to the final label in one step. Rather, the real is used to create edges and blobs and superpixels and depths. So using synthetic images in an INTERMEDIATE step allows you to perfect a more theoretically correct, provable, algorithm. Then the job becomes working on the other algorithm whose job it is to make the "real" images look like the synthetic ones. And the better it does, the better the results will be. And if it cannot be done well, then you need a different intermediate representation. Either way, There is a strong argument to be made for doing learning algorithms on synthetic images where the correctness can be exactly verified under all conditions.

I'd say it's similar to the field of programming where there are Unit tests, and then their are beta releases so "real people" can test it. You could say unit test are pointless, the real value is testing by users, because that's the final evaluation metric. But in reality, unit test provide a testable probable way toward algorithmic correctness. If the user still finds errors, then it's not that they can do things that can't be done with unit tests, it just means we need new unit tests that take into account more possibilities.

So in summary, I think from an algorithmic perspective, the goal is to have a well-understood algorithm that performs exactly as planned on synthetic (perfectly labeled) data, and we use "real world" data only to test if our model is complete. From the programmers perspective, the goal is program that meets requirements and passes all tests. From the Marketer's perspective the goal is for it to work as the user's expect, regardless of if it passes any/all unit tests. So are you the marketer or the programmer?

Its been over two decades since my own personal explorations in performance modeling of vision systems where my mentor Prof. Haralick advocated systematic characterization of performance of vision systems. Synthetic data can be viewed as samples from a probabilistic generative model. The key question is how close the probabilistic generative model to reality and how much the model approximation error affects your intended goals. For details of references on topic you can see:

http://fias.uni-frankfurt.de/neuro/ramesh/ .

Many model use synthetic image for case of study in artificial vision :

Using Visual Attention Models and Saliency Maps to

improve Gaze Tracking in Interactive 3D Applications

-->http://nouarti.free.fr/pdf/hillaire%203dui.pdf

and mécanism of the processes

http://www.imagwiki.nibib.nih.gov/mediawiki/images/e/e6/GIPSA-lab_Visual_Perception.pdf

But the whole of synthetic data against true images for the evaluation algorithm is really useful to study the simple shape of objects or proto-objects.in visual attention in neuroscience and artificial intelligence for autonomous robotic.

Synthétic data allow to modelize simple visual scene or simplie objects to see how human gaze works and to see how the computianals algorithms works in identical conditions.

Especially in the model (L.Itti : Itti, L. in http://scholar.google.com/citations?user=xhUvqK8AAAAJ&hl=fr and Koch : http://www.klab.caltech.edu/ ) and this site :

http://ilab.usc.edu/bu/

this lab study bottom up and top down mecanism of visual attention and evaluation algorthm to modélize human gaze in neuroscience.

These researchers works on the saillancy map analysis in

video.or image for O. Lemeur (http://people.irisa.fr/Olivier.Le_Meur/) and low and high frequency detection in visual scene for simple object and proto object.

Because these models introduces economy of the complete image analysis by reproducing mecanism of the human gaze and visual attention in the visual scene. Here simplified image (synthetic) for perform évaluation test in model allows to compare validity of the artificial gaze vs human gaze (eyes tracker analysis data in same visual scene).

Sorry for my bad English, I hope that you will understand me.

Images may be somewhat different, but I work with textual data and the main problem in this area is that when people collect data from large machine-based corpora there is a danger of confirming their own prejudices, rather than reflecting the real world. This is particularly so when automatic or semi-automatic labeling is used.

I don't know if you are interested in fingerprint images, but for sure this is a nice work: SFINGE (Synthetic FIngerprint GEnerator) distributed for free by the University of Bologna. It is much more convenient than trying to acquire thousands of people!

http://biolab.csr.unibo.it/research.asp?organize=Activities&select=&selObj=12&pathSubj=111%7C%7C12&Req=&

I would agree with most of the comments of this post. Synthetic data are a good way to evaluate algorithms and to have a better understanding on their behavior. Then we should be able to apply them in real situations. As an example, for the evaluation of background subtraction algorithms, we have chosen to propose both synthetic and real data in BMC (Background Models Challenge): http://bmc.univ-bpclermont.fr .

As I am more interested in texture, I usually work with Brodatz and Vistex databases. From some images of texture, I created my own dataset of artificial textured images. It is very interesting for testing algorithms, because I know where are the true boundaries.

Artificial images are really essential for evaluation, real images also : with real images, sometimes you find an unusual configuration that could lure an algorithm....

The conclusion (for me) is that you need to use both real and artifical images, if you want to develop and test image computing methods.

FYI:

VascuSynth: http://vascusynth.cs.sfu.ca/

VascuSynth is short for Vascular Synthesizer: An algorithm and accompanying software for synthesizing images of vascular or other tubular, tree-like structures

DeformIt: http://www.cs.sfu.ca/~hamarneh/software/DeformIt

Simulate novel images with ground truth segmentations from a single image-segmentation pair, now with support for scalar, vector and tensor-valued images.

HairSim: http://www.cs.sfu.ca/~hamarneh/software/hairsim/

Given a hair-free dermoscopic (skin) uimage, HairSim provides you a hair-occluded image accompanied with its binary hair-mask

Answer to question 1: On that list you are referring to nbr 70: MPI Sintel http://sintel.is.tue.mpg.de/ is a synthetic movie with depth map as ground truth, its purpose is to be used for evaluation of optical flow methods. But maybe it can be used for other purposes as well.

This is a hot topic in biometrics. The synthetic generation of samples make possible experiments with thousand of subjects (same benchmark data for all researchers) without any legal concern (avoiding also the time consuming acquisition procedures). You can find biometric synthetic generation for several traits: fingerprint, face, handwritting signature, iris, palm,...

Some examples:

R. Cappelli, “Synthetic Fingerprint Generation”, in D. Maltoni, D.Maio, A. K. Jain, and S. Prabhakar, Eds., Handbook of Fingerprint Recognition, pp. 203-232, Springer, 2003.

N. Poh, S. Marcel, S. Bengio, “Improving face authentication using virtual samples”, Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 233-236, Apr. 2003.

J. Zuo, N. A. Schmid, X. Chen, “On generation and analysis of synthetic iris images”, IEEE Trans. on Information Forensics and Security, vol. 2, pp. 77-90, Mar. 2007.

T. Dutoit, An Introduction to Text-to-Speech Synthesis, Kluwer Academic Publishers, 2001.

C. Rabasse, M. Guest, C. Fairhurst, “A New Method for the Synthesis of Signature Data With Natural Variability”, IEEE Trans. on System, Man, and Cybernetics – Part B, vol. 38, no. 3, Jun. 2008

J. Galbally, R. Plamondon, J. Fierrez and J. Ortega-Garcia, "Synthetic on-line signature generation. Part I: Methodology and algorithms", Pattern Recognition, Vol. 45, pp. 2610-2621, Jul. 2012.

J. Galbally, J. Fierrez, J. Ortega-García, R. Plamondon, “Synthetic on-line signature generation. Part II: Experimental validation”, Pattern Recognition, vol. 45, pp. 2622-2632, Jul. 2012.

Also the GPDS group from Las Palmas de Gran Canaria (Spain) shares a very long off-line and on-line synthetic signature database. You can download them freely without any legal restriction. If you were interested, I suggest coming across the following paper to contact with the corresponding author:

M. A. Ferrer, M. Diaz-Cabrera, A. Morales, "Static Signature Synthesis: A Neuromotor Inspired Approach for Biometrics", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.37, n.3, pp. 667-680, March 2015. doi: 10.1109/TPAMI.2014.2343981

check this paper "Comparative study of background subtraction algorithms" they used synthetic and semi synthetic video to evaluate different background subtraction methods

Synthetic datasets are very interesting for testing IP algorithms. With synthetic images, you perfectly know the image components, their location and attributes ... So it is perfect for testing. I used Brodatz textures and Vistex databases to tune my algorithms by making artificial images with parts picked within these databases.

Of course you must keep in mind that some issues may not appear on artificial images. This is why it is important to test also on a set of real images with manually labeled ground truth. The ground truth must have been done properly, not by experts who draw regions that are obviously not visible on the image. It can happen sometimes when experts know the ground and remember things that are not present in the images !