I hope that be helpful.

In my experience that transfer learning is very powerful in classification the objects, so when you to adapted the pre-trained model this technique is named fine-tuning.

So, I suggest to use the very architectures are VGG and ResNet to update to your problems.

I hope that be Clair for you.@ Fazla Rabbi Mashrur

Transfer Learning and Fine-tuning are used interchangeably and are defined as the process of training a neural network on new data but initialising it with pre-trained weights obtained from training it on a different, mostly much larger dataset, for a new task which is somewhat related to the data and task the network was previously trained on. In transfer learning, usually the last few layers of the network is replaced by new layers and initialised with random weights, the unchanged layers can either be frozen ie made un-trainable or be kept trainable.

Learning from scratch simply means initialising the weights of a NN to random values and starting the training on the main dataset and task.

https://www.facebook.com/1523997277911742/posts/2524496717861788/

Have a look at this in Quora : https://www.quora.com/What-is-the-difference-between-transfer-learning-and-fine-tuning

What is the difference between Transfer Learning vs Fine Tuning vs. Learning from scratch?

Transfer Learning is just as you said, a method when you freeze some layers, and do additional training to the rest of layers to adjust for your purposes and goals. Learning from scratch is a building completely a new model or mixing parts of others, and do training from the first layer. Fine tuning of the network is the process when you adjust parameters such as a learning rate, the number of epochs, the optimizer, regularization parameters etc. to achieve best possible results.

Hope this is useful for you.

You can take a look on this link is an example of training model from scratch.

Conference Paper Wheat Ear Detection in RGB and Thermal Images Using Deep Neu...

Best regards,

Thank you Željana Grbović Muhammad Ali Harsh Jalan

Hi Fazla Rabbi Mashrur, nice question. All three terms are so related to transfer learning. By definition, we do not train a model from scratch in transfer learning. Rather, we finetune some parameters of specific layers of a pre-trained model. I think you got a very clear explanation from Željana Grbović.

I will try to explain with an example of detecting COVID-19 using X-Rays. Since the dataset of COVID-19 X-Rays is currently limited and only 300-400 images of positive patients are available, we first train our deep learning model on a much larger dataset of X-Ray Images for example a dataset having 100,000 X-Ray images of 10 different lung related diseases. And then we use the weights from this model as a Base in our COVID-19 model so that our CNN model doesn’t have to learn basic features like edge detection from the limited dataset. This is known as transfer learning. While learning from scratch would include directly training the model on the limited dataset (300-400 COVID-19 Images). And Fine tuning is to be used in both the cases in a sort of hit and trial method using an intuition which we get over time to set the values of hyperparameters like learning rate.

The example I used above is basically for Convolutional Neural Networks but the same can be used in other types of NN like RNN, etc.

best regards

Thank you very much Md Meftahul Ferdaus and Harsh Panwar

Check this paper

https://www.mdpi.com/2076-3417/10/13/4523

It is about TL

Thanks a lot Laith sabah Alzubaidi

Hello Fazla, take a look at Figure 4 of the following paper,

Article How Deeply to Fine-Tune a Convolutional Neural Network: A Ca...

Transfer learning is when you ask a physicist to solve math problems.

Fine-tuning is when you tell that physicist to solve math problems for the next year.

Learning from scratch is when you're a high school student who goes into college as a math major and proceed until you finish your PhD in math and solve math problems for the rest of your life.

Thanks a lot Ibrahem Kandel and Sean Yi

Transfer Learning, as the name suggests, you are trying to reuse an already trained model on a new but similar problem. The already trained model can be adopted and directly applied to your test data.

Fine-Tuning: As the model is applied on a new problem, so you need to adopt it to suit your problem, hence you need to fine-tune it. It may involve fine tuning some parts of the network, like you said freeze the layers, and only update the required layers for the problem that you are going to solve. Usually, in this cases, we use the top layers of the network. For example, if you use a network that is trained on a large data set consisting of 1000 classes, and you want to use this model for classification on 10 classes, for sure you can fine-tune the network to adopt to only 10 classes.

Learning from scratch, means you will define architecture, build a model,and then using your training data you will train the it and then apply on your test data.

learning from scratch is refereed to when you build a deep network such as CNN by using massive number of images for training. However, Transfer Learning is refereed to when such a deep network was already built for you and ready to use for your application. As a result, using Transfer learning is much faster than building a system from the scratch for that particular application.

Fine turning is an improvement or diviation of a neural network functionality by small manipulations, without significant changes of architechure, data or learning process. Beside other techniques, TL can be used for FT.

So, I understand FT as a more general term. Other example of FT is additional learning of a trained neural network with newly discovered false positive samples only and with using a very low learning rate. In contrary, we can include new bad samples into the whole training data set and perform learning from scratch.

Fine-tuning is the small improvement made to NN by adjusting hyperparameters like learning rate, method of parameter initialization, etc. Transfer learning, on the other hand, is the technique in which a model is already trained on some other dataset (say ImageNet), we freeze certain layers of this trained NN model and train the classifier on top of that with the new dataset (on which we want to conduct classification). TL can be seen as the method of achieving good results on new datasets without using much computation power for training from scratch.

Training from scratch means that we create a new model (or some existing mode but not trained on any data) and train that on some dataset. this approach utilizes a lot of computational power.

typically if we want to train a NN model on some new dataset (say COVID-19), and we do not have much computational power at hand, TL method is the best-suited method in this case.

Transfer learning is when a model developed for one task is reused to work on a second task.

In Transfer Learning or Domain Adaptation we train the model with a dataset and after we train the same model with another dataset that has a different distribution of classes, or even with other classes than in the training dataset).

In my experience learning from scratch leads to better results, but it is much costly than the others especially regarding time and resources consumption.

Using Transfer Learning you should freeze some layers, mainly the pre-trained ones and only train in the added ones, and decrease the learning rate to adjust the weights without mixing their meaning for the network. If you speed up the learning rate you normally face yourself with poor results due to the big steps in the gradient descent optimisation. This can lead to a state where the neural network cannot find the global minimum but only a local one.

Using a pre-trained model in a similar task, usually have great results when we use transfer learning. However, if you do not have enough data in the new dataset or even your hyperparameters are not the best ones, you can get unsatisfactory results. Machine learning always depends on its dataset and network's parameters.

So, we need to evaluate the trade-off between the resources and time consumption with the accuracy we desire, to choose the best approach.

Thanks a lot Fatemeh Abdolali Mohammad Khalid Pandit Andrey Dashkov Abdolreza Mohseni Akhtar Jamil

I hope that be helpful.

In my experience that transfer learning is very powerful in classification the objects, so when you to adapted the pre-trained model this technique is named fine-tuning.

So, I suggest to use the very architectures are VGG and ResNet to update to your problems.

I hope that be Clair for you.@ Fazla Rabbi Mashrur

Thanks a lot Barodi Anass

Fazla Rabbi Mashrur In simple terms I could say that

Transfer Learning:

Transferring or using the learnt parameters (weights, bias) of a pre-trained network to a new task. Usually in the new task, we keep the network's layers and the learned parameters of the pre-trained network unchanged and we modify the last few layers (e.g. Fully connected layer, Classification layer) which depends upon the application.

Fine tuning

Fine tuning is like optimization. We optimize the network to achieve the optimal results. May be we can change the number of layers used, no of filters, learning rate and we have many parameters of the model to optimize.

Learning from scratch

Learning from scratch means building a network like AlexNet, GoogleNet etc. of your own. Every pre-trained network like AlexNet has some fixed or standard number of layers. If you build a network from scratch, you should decide the no of layers, no of filters etc. You should build a model that produces the maximum result for your application. Like trail and error method you should keep on working with different layers, increasing or decreasing the no of layers, filters, deciding about which normalization technique to use and deciding which activation function to use etc.

You could read my survey paper, where i have discussed about different deep learning architectures and techniques that have been used for Medical domain applications.

Article Survey on Recent Works in Computed Tomography based Computer...