Imrus Salehin

Applying dropout to kernels in a CNN means that, during training, some of the kernels are randomly turned off, effectively reducing the number of parameters in the model. This can help prevent overfitting by reducing the complexity of the model and making it less likely to memorize the training data.

Applying dropout to feature maps in a CNN means that, during training, some of the activations in the feature maps are randomly set to zero. This can help prevent overfitting by reducing the ability of the model to learn high-frequency details in the training data.

In general, applying dropout to feature maps is more effective in improving the generalization performance of a CNN, as it is more computationally efficient and less prone to overfitting than applying dropout to kernels. However, the best approach will depend on the specific problem and the architecture of the CNN, and may require experimentation to determine the optimal configuration.

Generally, dropout is applied to fully connected layers to prevent overfitting.

https://arxiv.org/pdf/1904.03392.pdf

https://www.sciencedirect.com/science/article/abs/pii/S0893608018301096

http://mipal.snu.ac.kr/images/archive/1/16/20190516013446%21Dropout_ACCV2016.pdf

Vrushali Lanjewar

Thanks For your recommended paper. And I know about this paper. The second recommended paper was published by my lab!

Janak Trivedi

Thanks for your valuable answer. But in the normal case, we drop the neuron with some weight. in the feature map which is the output of the kernel. feature map has no weight. Kernel has some weight if you consider normal neural network neurons. So, is it a good approach to use a feature map for dropout?

Imrus Salehin

Yes, you are correct. In the traditional dropout technique, the idea is to randomly drop out (i.e., set to zero) some of the activations in the hidden layer with a certain probability (e.g., 0.5) during each training iteration. This has the effect of reducing the representational capacity of the network and preventing overfitting, as the network cannot rely on any single activation to make predictions.

Applying dropout to the feature map in a Convolutional Neural Network (CNN) is equivalent to dropping out some of the activations in the hidden layer, as you mentioned. This can help prevent overfitting, since the network will not be able to rely on any single feature to make its predictions.

So, to answer your question, it is a valid approach to use the feature map for dropout in a CNN. However, as I mentioned earlier, the best choice of regularization technique depends on the specific problem and architecture, and it may be necessary to experiment with different methods to find the best solution.

I don't believe that dropout in front of another convolutional layer makes sense. Dropout does not really ignore the feature in the feature map, but it sets it to 0, which is a valid number. This can introduce patterns in the feature map that do not exist without dropout. When applying another convolution on top, the kernels need to learn to deal with random patterns that they never see during deployment.

Dropout on top of the *last* convolutional layer is surely useful, but not in between convolutional layers. This is identical to dropout after linearization in front of the first fully-connected layer, which is how dropout was designed to work.

However, I have to admit that even experts in deep learning (and I am giving a lecture on that topic) have no real insights on how these beasts learn their jobs, so I agree with Imrus Salehin that the best is to try it out.

Manuel Günther Thanks, Professor! For your valuable hypotheses!