Could you a bit more precise, please. Do you get a kind of "3D" file of the cytology images like with a confocal microscope? Or is it classical 2D picture?

If it is a kind of 3D file or a sequence of 2D pictures, you could use ImageJ. The Fiji version (http://fiji.sc/Fiji) include already all the add-on of ImageJ.

This article may contains information you need to solve your problem:

http://imagejdocu.tudor.lu/lib/exe/fetch.php?media=tutorial:working:workshop3d.pdf

Best regards

Geoffrey

I  misunderstanding  what you need, But for segmentation it is better to calculate threshold then you go to segmentation, Also you can HPF mask applied on imfilter Matlab function.

Regards

Could you post some example images?

I am working with 2D images and Matlab

There are different methods for touching nuclei seperation in literature. Before segmentation find out the seed points accurately. Euclidian distance map with multy scale laplacian of gaussian, radial symmetry transform,single path voting with shifted gaussian and concavity point detection are some of the technique seen in the literature. I also need matlab code for these. if any one has the code,pls help