If you want that CNN filters "move" along the third dimension of the input volumes, you should use 3D CNN.

Suppose the single volume related to a patient have dimensions:

[n_rows, n_cols, n_depth, n_channels]

(where n_depth is, for example, the number of slices related to an MRI volume and n_channels are the number of channels for that volume, for example 1 for grayscale volumes or 3 for RGB colored volumes), you should create a volume with dimension:

[batch_dim, n_rows, n_cols, n_depth, n_channels]

where batch_dim is the number of patients you are using in the learning process (for mini-batch gradient descent).

To each volume [i, :, :, :, :], the volume related to i-th patient, you can associate a label.

Often the creation of the previous 5D-volume is transparent to the developer and you should only define the 4 dimensions and batch_dim is defined only when you set the optimizer.

PyTorch and Tensorflow have conv3d layers that you can use as "typical" 2D CNN. Here some references:

https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv3D

https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html

https://keras.io/api/layers/convolution_layers/convolution3d/#conv3d-layer

An example of implementation is:

https://github.com/kcct-fujimotolab/3DCNN

However if you don't want that the filters "move" along the third dimension, you can also use the "simple" 2D CNN where you can use the channel dimension as depth dimension (obviously you can treat only grayscale images in this way).

Hello,

In short:

You stack the images into a [WxHxCxN] matrix (tensor) where

W = width

H = height

C = the number of slices of the image

N = the number of images.

And use con3d instead of conv2d.

We often think of images as 2-dimensional objects intuitively, however, in CNN's, images have 3 dimensions: height width, and the number of channels.

While black and white images have only one channel, color images have three channels, one for each color Red Green Blue, and MRI images have as many channels as the depth of the 3-dimensional representation, so in that sense, there is no difference between X-ray image and an MRI. However, since the MRI is representing a 3-dimensional object, it also holds features that span across the different layers (channels) and not confined to single layers, therefore you should use a 3d filter that captures 3d filters and has stride in 3 directions, 3d pools and so on, instead of the usual 2d filters.

Good luck

Good question. As far as I know, there are totally two kinds of networks which can handle such cases. 1) 3D CNN - all convolutions and pools are 3D. 2) 2.5 CNN. It is actually a 2D based network that treats each slice in the 3D volume as a single channel. Obviously, 3D CNN can capture additional inter-slice correlation information. However, from what I have experienced, 2.5D CNN works pretty well.

3D networks sometimes perform badly. I guess this arises from the conflicting between iso-tropical kernels in the 3D convolutions and the inconsistent through-plane and in-plane resolutions (take a look at the paper: https://ieeexplore.ieee.org/abstract/document/9169868). Takeways: Try 2.5D network firstly when your dataset is small...

Have a look at https://analyticsindiamag.com/brain-tumor-prediction-through-mri-images-using-cnn-in-keras/

https://www.pyimagesearch.com/2018/12/03/deep-learning-and-medical-image-analysis-with-keras/