The Fully-Convolutional Net (FCN) implementation is the most popular architecture for semantic segmentation . FCNs only have convolutional and pooling layers which give them the ability to make predictions on arbitrary-sized inputs. Another option is R-CNN (Regions with CNN feature). It performs the semantic segmentation based on the object detection results. R-CNN first utilizes selective search to extract a large quantity of object proposals and then computes CNN features for each of them. Finally, it classifies each region using the class-specific linear SVMs. Compared with traditional CNN structures which are mainly intended for image classification, R-CNN can address more complicated tasks, such as object detection and image segmentation, and it even becomes one important basis for both fields.

Yes, I believe if you use Convolutional Layers at 1st step as in MASK-RCNN for segmentation it will add a property of size invariance to CNN.

Yes, you can use CNNs for large images. You can use MATLAB to resize images and then use CNNs for the resized images.

Resizing with matlab is an interpolation operation which can cause loss information. Using convolution will be a kind of lossless resizing

To 1.:

Definitely possible, but maybe not feasible.

If you want to use an already trained CNN, then the input size will most likely be fixed, unless the CNN specifically provides that flexibility (I do not know if such implementations exist though).

If you want to build and train the CNN yourself, then you can just specify the input dimensions. But CAUTION: In this case you will require a LOT of data and the computational effort for such large input shapes is probably more than any standard machine can handle.

Usually, you might want to retrain an existing CNN. The feature extraction part is the convolutional part of the network. The weights of this part are fixed during re-training since features in images are assumed to be somehow universal. You only train the part that is doing classifications based on those features. Therefore, I do not think that adding new convolutional and pooling layers will help for downscaling, since it will alter the feature extraction part that you don't want to change.

To 2. and 3.:

Maybe a nested approach can work, where you first identify large structures (which can use a downscaled image) and then within those larger structures you do another detection of smaller structures. Maybe you can also train a CNN to predict where the gap should be located depending on the larger structures, then, if this works, zoom in and predict the size of the gap.

As always, think about the complexity of the problem too. Is it really hard for a human to spot that gap or can you still easily see it even once you change the resolution?

Cheers,

Julius

To 1.

Absolutely yes, you can inference a much larger image, even the network was trained with small images.

But there are some has to be checked before you start the experiments, like the size of features are similar? Is the input size of network flexible?

To 3.

You can imagine that the feature extraction of CNN is like the sliding window process, as long as the spatial resolution are similar, you can probably apply the same feature extraction process on a larger image.

Hsin-Hung Tseng

I think I get the idea of what you are saying, but I can imagine that programmatically this is really hard. I agree that shape and colour or other features in images are persistent through a range of resolutions, but if the features of interest cannot be resolved in the training image resolution you cannot do reasonable predictions right? Do you know of any implementations?

Thanks for your answers! I just trained a U-Net for this segmentation task!

My problem is that the two classes are "too" unbalanced. The white pixels are just about 0.1% (or less) of the whole image. The Dice Coeff loss function seems to not be working since it predicts always black pixels.

• Is there any specialized loss function for such unbalanced data? I can not find anything that works.
• Is the U-Net arcitecture suitable for such segmentation tasks?

I have tried to train with the following setup:

Epochs: 50 Batch size: 4 Learning rate: 1e-05 Training size: 451 Validation size: 23 Checkpoints: True Device: cuda Images scaling: 0.25

and also with batch size of 1 and a learning rate of 10^4. I would appreciate some help.

Cheers

Training size: 451 Validation size: 23 ...seems like a very small data set

"The Dice Coeff loss function seems to not be working since it predicts always black pixels." How about using a negative of your images?

There are several options for the loss function in case of a strongly imbalanced dataset like the one of yours. As a very first approach, you may like to try the Focal Loss (Article Focal Loss for Dense Object Detection

Actually, there are a number of other options that you may explore using the paper Preprint Segmentation Loss Odyssey

"How about using a negative of your images?"

This approach will not address the imbalance issue of the problem: a neural network with BCE or a smooth approximation of Dice will just learn to predict ones (instead of zeros in case of not-negative images).

Actually, in the case you describe in the first place, I would suggest reformulating the problem. CNNs aren`t really good at segmenting really thin objects. Perhaps, there may be an alternative formulation. Is there any chance to segment the rounded shapes you mention instead of the thin intermediate? Or maybe there is a way to warp (transform) the image the way that would widen the gap you are trying to segment?

 The Fully-Convolutional Net (FCN) implementation is the most popular architecture for semantic segmentation . FCNs only have convolutional and pooling layers which give them the ability to make predictions on arbitrary-sized inputs. Another option is R-CNN (Regions with CNN feature). It performs the semantic segmentation based on the object detection results. R-CNN first utilizes selective search to extract a large quantity of object proposals and then computes CNN features for each of them. Finally, it classifies each region using the class-specific linear SVMs. Compared with traditional CNN structures which are mainly intended for image classification, R-CNN can address more complicated tasks, such as object detection and image segmentation, and it even becomes one important basis for both fields.

Nikolay Raychev Thank you for pointing us to FCNs, because this answers my questions to Hsin-Hung Tseng above.