To which organ (e.g, lungs, liver etc) in CT image does your segmentation of ROI is confined?

Review of best "Interactive Segmentation Tools" that can be tried for preparation of ground truth.

The tools are ranked as per ease of use, features, accuracy etc

1) For 2D images human guided scribble based interactive segmentation tool.

Links:

http://web.archive.org/web/20110825021314/http://kspace.cdvp.dcu.ie/public/interactive-segmentation/

http://www.youtube.com/watch?v=efUh0zcpsXc

2) 3D Interactive Segmentation Tool FIJI

Links:

http://132.187.25.13/ij3d/?page=IntSeg_3D&category=Extensions

http://fiji.sc (or) http://rsbweb.nih.gov/ij/

http://www.youtube.com/watch?v=IR_1n24qCaU

3) 3D-2D Interactive Segmentation Tool TurtleSeg

Links:

http://www.turtleseg.org/

http://www.youtube.com/watch?v=TBEoJxj-EdQ

4) 3D-2D Interactive Segmentation Tool FIJI

Links:

http://132.187.25.13/home/?category=Download&page=SegmentationEditor

http://fiji.sc (or) http://rsbweb.nih.gov/ij/

http://www.youtube.com/watch?v=tC-b-BFq6jk

5) For 3D-2D images MITK Interactive segmentation tool

Links:

http://www.mitk.org/Interactive_segmentation

http://www.youtube.com/watch?v=hEZsQsTxMlo

6) GIMP - GNU Image Manipulation Program (Open source alternative to Adobe Photoshop)

Links:

http://docs.gimp.org/en/gimp-tools-selection.html

http://www.gimp.org/

http://www.youtube.com/watch?v=p0VbK6MjEv8

7) 2D CO3 for Ultra-fast and Accurate Interactive Segmentation

Links:

http://www.stat.ucla.edu/~ybzhao/research/co3/

https://www.youtube.com/watch?v=TdQAW6dEAYo

8) 3D NIH-CIDI Open Source Automatic Lung Segmentation Tool

Links:

http://www.nitrc.org/projects/nihlungseg/

http://www.youtube.com/watch?v=VD3GQ0O7weE

To which organ (e.g, lungs, liver etc) in CT image does your segmentation of ROI is confined?

Review of best "Interactive Segmentation Tools" that can be tried for preparation of ground truth.

The tools are ranked as per ease of use, features, accuracy etc

1) For 2D images human guided scribble based interactive segmentation tool.

Links:

http://web.archive.org/web/20110825021314/http://kspace.cdvp.dcu.ie/public/interactive-segmentation/

http://www.youtube.com/watch?v=efUh0zcpsXc

2) 3D Interactive Segmentation Tool FIJI

Links:

http://132.187.25.13/ij3d/?page=IntSeg_3D&category=Extensions

http://fiji.sc (or) http://rsbweb.nih.gov/ij/

http://www.youtube.com/watch?v=IR_1n24qCaU

3) 3D-2D Interactive Segmentation Tool TurtleSeg

Links:

http://www.turtleseg.org/

http://www.youtube.com/watch?v=TBEoJxj-EdQ

4) 3D-2D Interactive Segmentation Tool FIJI

Links:

http://132.187.25.13/home/?category=Download&page=SegmentationEditor

http://fiji.sc (or) http://rsbweb.nih.gov/ij/

http://www.youtube.com/watch?v=tC-b-BFq6jk

5) For 3D-2D images MITK Interactive segmentation tool

Links:

http://www.mitk.org/Interactive_segmentation

http://www.youtube.com/watch?v=hEZsQsTxMlo

6) GIMP - GNU Image Manipulation Program (Open source alternative to Adobe Photoshop)

Links:

http://docs.gimp.org/en/gimp-tools-selection.html

http://www.gimp.org/

http://www.youtube.com/watch?v=p0VbK6MjEv8

7) 2D CO3 for Ultra-fast and Accurate Interactive Segmentation

Links:

http://www.stat.ucla.edu/~ybzhao/research/co3/

https://www.youtube.com/watch?v=TdQAW6dEAYo

8) 3D NIH-CIDI Open Source Automatic Lung Segmentation Tool

Links:

http://www.nitrc.org/projects/nihlungseg/

http://www.youtube.com/watch?v=VD3GQ0O7weE

Yep, this is a big problem.

The best you can do is to get the ground truth from several experts. They need to delineate the ROI manually; preferably several times, so you get an idea about intra- and inter-observer variation.

If your segmentation is well within the above variability, it can be considered 'good'.

I would NOT offer the experts a semi-automated delineation tool because that introduces a strong bias!

Both answers are very useful, but you have to keep in mind that even very carefully performed experts' segmentations will never be an actual ground truth - just a reference, the use of which in the way described by Lambert is generally accepted. Sometimes ground truth can be obtained from excised pieces after surgery or from a different imaging modality. Some authors generate synthetic images with known geometry and properties reproducing as closely as possible true imaging results. Such a geometry can be considered as ground truth. However, one can always argue that such a synthetic procedure cannot reproduce all the imaging pitfalls. The last solution is to acquire real images of physical phantoms, the geometry of which is known. Once again, such a geometry considered as ground truth is valuable if the material used to build the phantom closely reproduces real tissues.

There is an article about evaluating segmentations at http://infoscience.epfl.ch/record/125734/files/TowardsFullyAutomaticImageSegmentationEvaluation.pdf.

The training on recognizing "typical" abnormalities plus the background noise removal help to crop without human input. I worked with wavelets to detect unknown changes of signal behavior, but I don't know if the program you use already has this feature incorporated. Maybe the wavelet analysis could help adjust the recognition threshold and change resolution as the target is approached.

Zijp wrote:

"If your segmentation is well within the above variability, it can be considered 'good'."

You may use the Friedman test for this, is implemented in SPSS

>> Is there any way to make this quicker?

Reduce the quality of the image, increasing spacing and reducing slices. Consider the previous as lossy compression. Try to comprise the image to a point just before it is notable. Don't forget to annotate your experts opinion on how comprised the image can be before compression is notable.

>>Is there any other way to evaluate the proposed segmentation method?

The given answers are extremely complete. I would like to add that sometimes you may evaluate your segmentation by data obtained from process posterior to segmentation rather than the segmentation itself, for example:

In metallographic microscopic they separe two components in the image by a segmentation process: perlite and ferrite. Later they determine ( by simple pixel counting) ferrite percent for a given sample of steel to wich this percent is known. If the percent matches whithin a given range, the segmentation is good (and no one ever look at the segmentation )

You may also gather an expert's committee to evaluate your segmentation, let's say from 1/10. Try to gather as much experts as you can.

Anyhow, if you get lost, always remember:

1. Keep calm and search for a solution :D

2. To get an article accepted on a magazine, investigate in literature what they did. Are they using phantoms? Are they using experts, if so, how many experts? I you repeat the process in the literature you may expect that your validation process to be accepted. Don't forget to use/reference/compare to the process of earlier articles in the magazine you want to publish! It will boost your chances to get the article publised!

Since nobody else has mentioned it, CT images are (to some degree) self-segmenting in that ranges of Hounsfield units have been associated with different tissue types & organs (for example Blood: +30 to +45, Muscle: +10 to +40, Grey matter: +37 to +45). Before everyone jumps down my throat, I realize that the story is not that simple & the Hounsfield ranges change with the intensity of the x-ray (and probably with several others variables). But... I felt compelled to add this to the discussion.

Unluckily, these ranges overlap, and don't forget the partial volume effect

Quote Dan: "for example Blood: +30 to +45, Muscle: +10 to +40, Grey matter: +37 to +45"

Exactly!! That is the problem: the Hounsfield units of several tissues overlap; soft-tissue contrast is poor. That's why segmentation purely relying on pixel-values is difficult.

(Skin, lung and bone are easily done, although bone of old people is still challenging)

@Muhammad: what organ are you segmenting; the prostate by any chance?

@Marcelino: Are you sure the Friedman test is appropriate? If a segmentation of the prostate is perfect everywhere except that a chunk of the rectum is incorporated, such a segmentation cannot be used 'as is' for radiotherapy purposes. I mean: a small bad part can be concealed by a big good part, thus passing the test.

I think one needs to calculate the standard deviation of the expert delineations FOR EVERY PART (VERTEX?) of the organ, and then see where the segmentation performs less than desirable.

LOL!

I knew there were problems with using HU for segmentation but simply copied a few ranges from the wikipedia page as an example and didn't appreciate that they overlapped. I guess things are worse than I thought.....

Sorry about that.

No problem Dan, but you should have copied lung, fat, muscle and bone from Wikipedia :-) !

Hi Muhammad,

Could you specify a bit more your question?

1. What kind of anatomical structures are you targeting? Organs, tumors?...

2. What kind of feature do you intend to extract from your segmentation? Contours, volume, maximal diameter, average intensity, texture?....

These last mentioned aspects are key to define a validation protocol.

For instance you would find good quality phantom for lung but less quality for liver or other organ. To be noted that public image are available for synthetic lung (NCI database)

If you are interested in segmenting lesion, and depending on your endpoints, you would repeate your segmentations than process STAPLE algorithm (check NIST publications).

I hope this helps

Thank for the all suggestions. Actually i try to segment the aorta from the CT image.

if you have IVUS images, it is a golden standard for CT cardiovascular image segmentation.

Muhammad....have u reached a way to create ground truth for aorta segmentation ?

No, not yet