I have three image segmentation techniques that I have developed.
I wish to compare the performance of the segmentation techniques using measures such as BDE(boundary displacement error), GCE(global consistency error) etc.
In the previous literature I've seen, sample size was either 10, 20, or 30... the mode being 20 images.
However, I'm not certain why these values were chosen.
What criteria is usually used to set sample size in this scenario.
It's fairly depending on the segmentation method which you use. generally in deep learning methods the more data will lead to better results. you can use augmentation techniques to increase the dataset size.
Dear Flavia,
It is not possible to evaluate all segmentation methods similarly. One common method of evaluation is to use ground truth data another method is the one you mentioned GCE using reference image (hand segmented image). The number and size of ground truth data depends on the complexity of the segmented image. The same applies to GCE the reference image should contain more truth data as the image and method gets complex. There is not limit to the number as long as you feel confident of the results (i.e. adding more truth data does not improve accuracy).
There is no easy answer because the answer depends on:
1. the application in which the segmentation methods are to be used,
2. the requirements of the application,
3. the types of errors that you wish to optimize for,
4. the relative costs of those errors to the application,
5. the breadth of applications in which the algorithms are to be used, and perhaps most importantly,
6. the reliability that you require for the test (governs test-sample size).
So for instance, item 1 addresses the criticality of the application. For more-critical applications, more extensive testing is usually required.
Items 2, 3 and 4 reflect the accuracy requirements of the application. These requirements must spell out a metric that allows one to evaluate the various errors. For instance, in a segmentation algorithm, you might have segments that are too small but the right shape. Is that better or worse than having segments that are distorted shapes but have the right area? What about a method that works flawlessly with uniform illumination but fails miserably with illumination that has a strong gradient? What about non-closed segments? Shifted segments? Segments that have failed at 'T' junctions of edges? Segments with noisy edges? Segmentation that fails as the noise level increases? Etc. You have a multidimensional space of errors and you are trying to reduce that to a single scalar. Mathematically, this is an impossible task. We only solve it by assigning an arbitrary weighting to the types of errors, and those arbitrary weights must be selected to reflect some aspect of the relative importance of the errors to the application.
Item 5 speaks to the degree to which the task is constrained. The size of the evaluation test set must be such that it spans and indeed exceeds the range of errors for all error dimensions over the space of all images that might be experienced by the application. Thus, applications that are tightly constrained and for which few types of error are at issue will require smaller test sets than applications that are more general and with many types of error, all else being equal.
Item 6 is hugely important. The size of your test set must be sufficiently large that the results of the test are reliable at the level that you desire. This speaks to the statistical significance of the test. Given a measure of the signoficance that you desire, and knowing the number of degrees or freedom, it should be possible to design a test that has sufficient statistical power to deliver the desired reliability of the test. Note that this reliability is not the reliability of the segmentation method; that is a whole problem in and of itself.
So the answer? It depends on:
1. what you are trying to do,
2. how well you need to do it, and
3. how confident you want to be in the evaluation of the various segmentation methods.
Peter
Katumba Noah
Is there a correlation between method used and sample size for evaluation purposes?
- Badges
- Science topic
- Similar topics
- Computer Science
- Images