This is, admittedly, a bit of a fuzzy question related to neural networks. However, I have engaged in recent discussion on the matter of whether the ‘loss landscape’ (sometime referred to as a ‘loss surface’) is altered when dropout is used. There seems to be a divide among the researchers polled and I am curious to hear your thoughts.
Hi.
I'd rather say that a loss surface is altered when dropout is used. Let us understand why it happens. A loss function L(W) depends on a set of weights. Assume that we use some weights and don't use a zero dropout value. Then a corresponding loss surface can be projected. Let us increase a dropout value with the same weights. Will it affect the output of L(W)? Yes, it will, because signal flow through layers will be changed; the output value of the model y_pred will take another value for the same weights. Thus, L(W) output will be different.
In order to prove this assumption, I'd like to recommend you to see this stunning video that demonstrates the dependency of the form of a loss surface on a dropout hyperparameter: https://www.youtube.com/watch?v=2PqTW_p1fIs.
Best regards,
Artem Kramov
https://arxiv.org/abs/1712.09913
You can check this paper. Very relevant and interesting read.
You can look up dropout in a neural network for some tips.
https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf
Hi,
This is an interesting question. Regardless of the dropout one might avail similar loss landscape unless the parameter space has shifted necessarily enough to provide deviated performance for same input space.
Also, you might find it interesting to investigate the knowledge propagation for your trained parameter space with and without dropout to see exactly how much of dropout can push the loss landscape.
Finally, you can also argue about the layer-wise dropout for your scenario. Say, you might reach very same landscape with significant dropout within initial layers, but you will reach very different set of points if you push the the final layers to hit void.
Again, what is the specific case if you add dropout for residual connections? Perhaps, you can share your amazing thoughts on this.
In essence, dropout will lead you to different landscape based on som rudimentary 'if-else' cases.
Have a good day ahead.
- Badges
- Science topic