My question is "can we use pre-trained models like InceptionV3 on medical image datasets?" Healthcare data has two major characteristics when using deep learning: 1) often small sample size, 2) different data content from the pre-trained model (e.g., ImageNet).
Recently research discoveries may say that ImageNet pre-trained model has very limited help for some tasks. ImageNet pre-trained model is mainly trained using natural images. Because of the big difference between natural images and medical images (e.g., CT/MRI), we have to fine-tune our networks, which is known as transfer learning. Transfer learning may have a very limited effect when we switch the data content from one type to another. Hence, transfer learning in this case may be no better than training from scratch, as the networks learn very different high-level features in the two tasks. Some statements could be found in 'Rethinking ImageNet Pre-training, Kaiming He'. Certainly, we know if we have enough data, training from scratch is a feasible approach. However, I am standing in the healthcare field. Additionally, we trust that transfer learning could boost the coverage time, but how about the performance?
I here ask all experts' help to explain to how to solve the current challenges in the healthcare field: 1) small sample size 2) different data content from pre-trained model.
It's true that, training the networks with some initial weights perform better than random weights (some papers proved that applicably of transfer learning in medical image classification, i.e mammography images). If it possible to initiate your network with some trained weights, so do it!
However, It's really depend on the your research problem, and also possibility of application of transfer learning on your data, as most pre-trained networks are 2D architectures, and medical images are 3D. For transfer learning in medical images, you may use weights from Cat&Dog images, however at least your not initiate your network weights from random noises!
About sample size, you may use two approaches:
First, you can use data augmentation techniques. You can use transformation with different sub-transformation including zooming, translation, rotation,shearing and etc. Beside that, you should be aware about data augmentation techniques in medical images, you should augment images with simulating different situation in imaging which depend on imaging modality.
Second, use patching strategy. In patching you may convert one patient to more than 100 sample!
Hope this be helpful.
Isaac
Yes you can, however, you could get a good accuracy when using images that LOOK similar. For example one of my student applying learning from cornia (eye) images to cell mitotic spindle and getting good result because cornia nerves seems to look similar to spindle microtubule structure.
Dear Z. Shi ,
Look rhe link, maybe useful.
http://infolab.stanford.edu/~echang/HTC_OM_Final.pdf
https://arxiv.org/ftp/arxiv/papers/1708/1708.03111.pdf
Preprint Med3D: Transfer Learning for 3D Medical Image Analysis
Regards,
Shafagat
Transfer learning works pretty good in medical images. Most published deep learning models for healthcare data analysis are pretrained on ImageNet, Cifar10, etc. Pretraining most times does not necessarily need to be done on dataset of similar domain but just to give a model a general context about objects. This has been proven to fasten convergence of deep models than training from scratch.
Yes, it is true that healthcare research is limited by small dataset, but many research labs have collected quite large dataset. A good example is on video dataset which at varying fps, thousands of images could be extracted.
However annotations of extracted images could be tedious. Most publicly available dataset are quite small because only a small size groundtruths could be annotated but, you can leverage on large unannotated samples to pretrained your models on some self-supervised tasks. In this case, thereafter you can finetune your model on the annotated samples .
Visit this website: http://camma.u-strasbg.fr/datasets for large medical and healthcare datasets.
Merci beaucoup!
Z. Shi
Thanks!
For patching, you can extract random patch from image (2D or 3D). You can move (with a stride) and crop each image with a standard kernel (i.e 16*16 for 2D and 16*16*16 for 3D).With patching you increase the number of data per classes.
Training process would be as follow (which mentioned in papers):
" each image is divided into smaller patches and each patch is classified with a “patch-wise” classifier network and assigned to a label. To classify at the whole image level, the patch-wise network is followed by another classifier that receives output labels from the first network as input and generates label scores. These techniques achieve high accuracy with high confidence on image patches, however, they fail to capture global attributes of the image: Once all image patches are labelled, the spatial information is ignored and any possible feature that is shared between patches is lost "
and also can be combined with data augmentation techniques:
"This would save the image information to a great extent and reduce the computation complexity effectively. These works can not only obtain the fixed-size but also augment the image samples. In addition, we would flip the image horizontality or verticality to further amplify the image datasets. At test time, the network makes a prediction to each patche and averaging of the predictions made by the softmax layer if the patches belongs to the same image. The influence of image augmentation on the accuracy and running time will be discussed in the following experiments."
You should have a group of images for the use of training your model. Then test the rest of your database on your model. The good example may be help you is blood vessel segmentation by supervised algorithm.
We attempted this problem with my advisor Preprint Transfer Learning based Detection of Diabetic Retinopathy fr...
Transfer learning has been recommended in research works as one of the solutions to the insufficiency of training data in healthcare. While transfer learning helps your training process in terms of reducing size of required training dataset and saving training time, you have to also study and look at pre-processing techniques that can be applicable to your specific area of healthcare images first.
You should also implement heat-maps in order to confirm that the convolution part (assuming it is a CNN) of the pre-trained model you are importing is extracting the right features of your training images. If the convolution part is extracting or focusing on the wrong sections of your images, it may be a sign that you need to work with a different pre-trained model.
https://arxiv.org/abs/1905.07203
Hi Z. Shi ,
Yes it is possible. By using pre-trained models on ImageNet for example as initial weights for your training on medical images. Despite the difference between ImageNet and medical images, there are generic representation that might be beneficial to be learnt from a large-scale dataset like ImageNet. I used pretrained models on ImageNet as an encoder for my segmentation network [1] and classification networks as well, and it made the training convergence faster (at least).
[1] https://arxiv.org/pdf/1901.09197.pdf
[2] https://ieeexplore.ieee.org/document/8641815
Regards,
Ahmed
Transfer learning is very important due to the limited datasets in the medical and healthcare domain, otherwise, the convergence of weights won't be possible. In my opinion, these following steps will help you to get good accuracy on your dataset.
1. Heavy data-augmentation (horizontal flip, vertical flip, and random rotations)
2. Don't freeze any of the CNN layers.
3. Experiment with slow learning rates (3e-3 or even lower) and a large number of epochs for the better convergence to your dataset.
Hello , Yes it is possible, I worked on such problem on my paper "An image-based segmentation recommender using crowdsourcing and transfer learning for skin lesion extraction", you can find more details on it and if you want I can send you the full paper.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size