E.g., Transfer the weights of deep neural network from ImageNet to Natural Language Processing task (e.g., sentiment analysis):
http://ruder.io/transfer-learning/
https://machinelearningmastery.com/transfer-learning-for-deep-learning/
https://arxiv.org/pdf/1708.00107
https://arxiv.org/pdf/1803.01164
Hey there,
I think transfer learning works fine when the domains source (the ones where a network have trained) and target (the new domain) are similar. For example, to use a pre-trained network on ImageNet and finetuning it to face verification. On the other hand, when the domains are different, i.e, from ImageNet to Natural Language I think it is better training a deep network from scratch (in other words, without using fine-tuning). In particular, in the case where the domains are very different, I think the network convergence is a problem.
I have working on transfer learning approaches, you can check our work Conference Paper Face Verification: Strategies for Employing Deep Models
, where we propose a successful transfer learning technique applied to face verification.
Yes. Just try to use the same weights of the pretrained models. Also, don't forget that you must not reinitialize the biases of the network. Good luck.
Hey there,
I think transfer learning works fine when the domains source (the ones where a network have trained) and target (the new domain) are similar. For example, to use a pre-trained network on ImageNet and finetuning it to face verification. On the other hand, when the domains are different, i.e, from ImageNet to Natural Language I think it is better training a deep network from scratch (in other words, without using fine-tuning). In particular, in the case where the domains are very different, I think the network convergence is a problem.
I have working on transfer learning approaches, you can check our work Conference Paper Face Verification: Strategies for Employing Deep Models
, where we propose a successful transfer learning technique applied to face verification.
Dear Mohamed,
I agree with Artur in the sense that, in the example that Mohamed has provided about a Transfer Learning from an Image Processing problem to an NLP problem, Transfer Learning would not be really useful as the structure of the data is very different and thus, the low-level features captured in the first layers of the Neural Network (NN) may differ considerably. So I would say that Transfer Learning makes sense when transferring across domains that have the same input data structure (e.g. image processing problem A to image processing problem B, or Speech Recognition problem A to Speech Recognition problem B), and where low-level features could be helpful for learning the new problem.
Just for providing a little more insight, I would say that one of the main objectives of Transfer Learning is to reuse the low-level features that have been learned from one problem, let say problem A, to other problem B. For example, in the computer vision field, where I come from, low-level features, such as edges, curves, contours, etc, that can be learned for example by using ImageNet, may be really helpful to solve other computer vision problems (e.g. object recognition). In addition, another important motivation for using Transfer Learning is the amount of available data. For example, if you have a large amount of data for solving problem A (e.g. ImageNet) and a relatively small dataset for solving problem B, this motivates Transfer Learning techniques (e.g. fine-tuning).
In this direction, another important variable is the amount of data that is available for solving the task to which you want to transfer to. If you have a relatively small dataset, maybe the most interesting procedure is to take out the last layer of the NN, initialize randomly the last layer's weights, and re-train only the weights of the last layer (fine-tuning). If you have a relatively large dataset maybe you can re-train all the parameters in the network taking as starting point the knowledge learned from problem A (pre-training).
Hope that this helps.
Carlos.
- Badges
- Science topic