I used BERT in a few papers and I was going to use it in a future project. I needed to understand its drawbacks in a real ongoing project.
I read somewhere that its biggest disadvantage is that it is very compute-intensive at inference time, meaning that if you want to use it in production at scale, it can become costly.
Its pros and cons: https://www.quora.com/What-are-advantages-and-disadvantages-between-pre-trained-and-trained-models (read other discussions on the left of this open link)
The main drawbacks of using BERT and other big neural language models is the computational resources needed to train/fine-tune and make inferences. However, more recent research have proposed different methods to overcome this issue. Knowledge distillation, quantization, pruning and other techniques [1] make BERT models reasonable for production environments. Recently Google announced its production usage of BERT-based models on English searches: https://searchengineland.com/google-bert-used-on-almost-every-english-query-342193
[1] Preprint A Primer in BERTology: What we know about how BERT works
- Badges
- Science topic