Transfer learning seems to be widely used for the convolutional layers of neural networks architectures. But for fully connected layers I haven't seen It used.
Can you point to me research where fully connected layers use transfer learning? And what are the reason fully connected layers are not used like convolutional layers for this case?
Transfer learning is not limited to CNN based neural networks. Howard and ruder 2018 applied transfer learning to RNN based networks, also Vaswani et al., 2017 introduced Transformer architecture. It is encoder-decoder architecture, where each part contains several layers, each layer includes self-attention in addition to feed forward layer and other components. Transformer is widely used to transfer learning in NLP tasks. CNN-based architecture are preferred in computer vision tasks, Transformers are achieving SOTA in NLP tasks.
Shortly, you can transfer learning of any deep Neural network (DNN) , but to have benefits from this transfer learning, your original DNN should be trained using very large generic dataset, and your downstream Dataset shares main characteristics with the original generic Dataset.
For example, when you train your DNN to generate English text (Language Modeling), you can use transfer learning to perform named entities recognition of English texts. However, you can not transfer learning to classify Spanish texts for example.
Have a look at Preprint An Efficient Transfer Learning Technique by Using Final Full...
https://cs231n.github.io/transfer-learning/
https://machinelearningmastery.com/how-to-use-transfer-learning-when-developing-convolutional-neural-network-models/
- Badges
- Science topic
- Similar topics
- Biological Science
- Bioecology
- Systems Ecology
- Landscape Ecology
- Connectivity