I have trained the vision transformer several times and noticed that it takes so much time to train the network while working with images, especially when we increase the number of transformer blocks. I am wondering if there is a way to optimize the computation in the transformer while obtaining the most efficient information from the image? In other words is there any approach or idea that helps to optimize the transformer?
Hi,
I suggest you see the links on the topic.
https://viso.ai/deep-learning/vision-transformer-vit/
https://towardsdatascience.com/using-transformers-for-computer-vision-6f764c5a078b
https://pub.towardsai.net/overview-of-vision-transformers-is-all-you-need-88727438ff8d
https://theaisummer.com/transformers-computer-vision/
https://analyticsindiamag.com/hands-on-guide-to-using-vision-transformer-for-image-classification/
https://tianxintao.github.io/data/ViT_for_Visual_RL.pdf
Best regards
There are several things that can be done. Some ideas from my side:
1. Using self-supervised pre-training techniques such as contrastive learning (Preprint Understanding Contrastive Learning Requires Incorporating In...
) can improve performance.2. Increasing the number of transformer layers can improve performance, but it would also increase the number of parameters and computation complexity (and therefore time).
3. Incorporating attention mechanisms specific to vision tasks, such as non-local attention (https://openaccess.thecvf.com/content_cvpr_2018/papers/Wang_Non-Local_Neural_Networks_CVPR_2018_paper.pdf), can also improve performance.
- Badges
- Science topic